Selecting tests with -k doesn't work as expected

[pytestbot]

Originally reported by: Brianna Laugher (BitBucket: pfctdayelise, GitHub: pfctdayelise)

---

Using platform linux2 -- Python 2.7.5 -- pytest-2.3.5

I suspect this is related to the changes in d16cc2d which went into 2.3.4. Basically I find it a lot harder now to select tests, especially parametrised ones.

Here are some examples:

#!shell

$py.test tests/system/combined/test_marineWindWarnings.py --collectonly
=============== test session starts ===============
platform linux2 -- Python 2.7.5 -- pytest-2.3.5
plugins: xdist, cov
collected 6 items 
<Module 'tests/system/combined/test_marineWindWarnings.py'>
  <Function 'test_windSpeedsTreatedConsistently[25.4-<None>-25]'>
  <Function 'test_windSpeedsTreatedConsistently[25.5-CW1.W:STR-30]'>
  <Function 'test_windSpeedsTreatedConsistently[33.4-CW1.W:STR-30]'>
  <Function 'test_windSpeedsTreatedConsistently[33.5-CW1.W:GALE-35]'>
  <Function 'test_windSpeedsTreatedConsistently[63.4-CW1.W:STO-60]'>
  <Function 'test_windSpeedsTreatedConsistently[63.5-CW1.W:HURR-65]'>

• If I run -k 'None', I get no tests selected (rather than 1).
• If I run -k '<None>', I get a SyntaxError!
• If I run -k '33.4', I get 6 tests run! No idea why. Same with '25.4'. And even '25'. I have no idea how it is handling numbers.
• If I run -k 'test_windSpeedsTreatedConsistently[33.4-CW1.W:STR-30]' (which is what I used to do) I get another SyntaxError that shows that my string has been changed to test_windSpeedsTreatedConsistently[33.4not CW1.W:STRnot 30].

It seems to me at the moment there is no way I can select only this first test.

To be honest, I really prefer the old behaviour of -k. Being able to run collectonly and then copy a test name into -k was useful.

Related, I can't find an explanation in the documentation about which characters/words are "special" and not treated as a flat string. Am I writing a regexp? I can see "-" is converted to "not"; what about other punctuation?
The usage docs say

#!python

py.test -k string     # only run tests whose names contain a string

which is basically not true right now.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/357

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

Most people like the new -k and -m syntax ASFAIK but it is more limited with matching against param values, indeed, as you analyze. Not sure atm how to remedy it -- we could fall back to the old behaviour if the expression is a syntax error or introduce a specific parameter match option. Like --testmatch p 'part_of_name_or_param' which would do a string match. Or -t for short.

[pytestbot]

Original comment by Brianna Laugher (BitBucket: pfctdayelise, GitHub: pfctdayelise):

---

I would be happy if there was a separate option for the old behaviour, or maybe I just need to build a plugin/extension to add it back.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

fix usage docs for "-k" (addresses issue357)

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

happy to review a PR which add an option to match tests via a regex on their id.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

actually have a better idea (maybe): if the -k expression contains no spaces, revert back to the old behaviour of a simple string match and "-TEXT" as negating. This should have avoided your surprises, right?

[pytestbot]

Original comment by Brianna Laugher (BitBucket: pfctdayelise, GitHub: pfctdayelise):

---

Hmm, I don't really like the idea of having it behave differently depending on what you specify. That would just be more confusing for the user. "-TEXT" will appear in all names of parametrized tests (with at least 2 parameters). There is often spaces in my test names too as the parametrization might contain a string with spaces.

I think a clearly separate option is better than overloading -k with multiple behaviours. If no one beats me to it I will look at adding it as a plugin or patch.

BTW I still don't really understand why searching for numbers like '33.4' or '25' would match all my tests?

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

hum, your "behaves differently depending on what you specify" criticism makes sense. On the other hand i am not sure it would be a practical problem. I am fine with a separate option and guess it should go into core.

As to why 33.4 matches: internally the -k expression is evaluated as a python expression. 33.4 evalues to True so independently of test names it matches. Apart form parametrization people cannot have float numbers appear as part of the test name which explains why it hasn't come up yet. We could also think about making the option match parametrization specifically, maybe. Note also that we are considering long-term to have test ids not be generated based on the values of the parameters because it disallows parametrization like [value, value], i.e. repeating tests. However, more thoughts on this topic are warranted.

[pytestbot]

Original comment by BitBucket: jenisys, GitHub: jenisys:

---

Related to change-request/enhancement #387

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

fix issue357 - special case "-k" expressions to allow for
filtering with simple strings that are not valid python expressions.
Examples: "-k 1.3" matches all tests parametrized with 1.3.
"-k None" filters all tests that have "None" in their name
and conversely "-k 'not None'".
Previously these examples would raise syntax errors.

Also add a note to the docs about what is allowed.