Proposal: Enable separate configs per run/discovery/debug with testing

[eleanorjboyd]

The following outlines a proposal for how to handle testing args.

Submitted issues related to this problem:

• Add separate python.testing.pytestArgs settings for regular, debug and discovery runs #21255
• Argument filtering for pytest fails to handle custom arguments #11736
• pytest arg parsing bug #21176
• Support different options for discovering and running tests with pytest #15148
• Read test debug configuration from settings not just launch.json #18778
• Support setting environmental variables for the test runner #17171
• design of UI as someone selects args: Test discovery flow is unusual/weird #17094
• launch.json "args" don't get passed to pytest unittests #12997
• Add "Run Tests" to launch.json #11932
• setup testing configs will be changed: Unittest rewrite - configure tests quick pick #21116
• people can use plugins on only a single on of the actions: Test Discovery fails on large codebases #21922
• As part of this overhaul we will stop parsing args:
  • Python unittest runner doesn't respect arguments to pass to unittest runner #20425
  • Python.testing.pytestArgs does not work with option -m #21798
  • Switch to using arg map for testing args #22076

Current design:
Currently, to pass any command line args for test run/discovery via the extension there is a single setting per test type (one for unittest, one for pytest). The two settings are python.testing.pytestArgs: [] and python.testing.unittestArgs: [] which take a string array of command line args. The args are then processed by the extension to remove some not applicable to the current action (removing run args during discovery and discovery args during run) but this is buggy and hard to maintain. Additionally to debug a debug config can be created in the launch.json.

Issues with current design:

1. With only a single place to put args for each test type, a user cannot distinguish between which command line args they want to include during test run, discovery, coverage, or debug.
2. Since pytest does not allow some args to be included during with the arg --collect-only (used to distinguish run from discovery by the extension) these args are parsed out by the extension before sending the request. This means custom args are removed from test args and generally it creates a buggy experience that is hard to maintain with the breadth of args users would like to pass into their tests.
3. There is no straightforward solution to providing environment variables while running and discovering tests.

Proposed new design:
Create new custom launch configs that can be setup for test run and discovery. Also, increase compatibility with debug launch configs to make sure these work in tandem. The launch config would be set up something like this:

    {
      "name": "Python: pytest run",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "PYTHONPATH": "${workspaceFolder}"
      }
    },
    {
      "name": "Python: pytest discover",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "VAR": "true"
      }
    }

The two particular fields "args" and "env" would provide functionality useful to users. The rest would be default values.

The new launch config types that would be added would be:

1. Python: pytest run
2. Python: pytest discover
3. Python: unittest run
4. Python: unittest discover
   The names of each one are up for discussion, please provide feedback below!

Next a way to set up these launch configs would be introduced to the extension. The initial steps would be:

add configuration -> python -> python config menu
Two possible options for how to display the testing config options would be:
A: to have another submenu called something like "testing" which, when opened, lists the 4 config options above
B: have the 4 config options sit in the current python submenu.

(I think option A is better for clarification even though it requires one more menu clickthrough).

additionally the button "configure tests" found on the test explorer would be changed so it routed the user to a menu of the launch config options for testing (similar to option A).

Concerns with proposed design:

1. The use of launch configs may be harder for new / beginner users.
2. The discoverability of launch configs is harder.
3. The transition from a setting to using a config could be complicated/confusing for current users.

Possible ways to address above concerns

1. help beginner users
   a. Increase documentation and provide specific examples of launch configs.
   b. provide default values for all fields in the launch config (this is already planned so the launch config should be runnable upon creation without edits meaning no args or env vars need to be provided)
2. discoverability
   a. There is already a larger discussion happening on how we can improve this, switching to configs for testing provides even more reason to tackle this problem
   b. switching the button "configure tests" to the new config flow is very discoverable since that is already the workflow
3. transition
   a. We can have a deprecation notification regarding the setting and show it to all those still using the setting for test args
   b. It should be simple to take someone's test args in their settings and turn that into a launch config so we could offer a button that handles this for the user to switch them over (drawback is we would be setting discovery and run to the same args but the user could see this then go change it)

Notes on Debugging:
Less investigation has been done into test debugging as we already offer a similar launch config. Further analysis should be done to see what is missing in this flow and how to improve it to align with test configs once a design is decided on.

Action Items:
We are looking for feedback on this proposal so please comment below any thoughts. Thanks!

[roblourens]

The "name" of a launch config is for the name that the user picks, it shouldn't be used to change the behavior of the launch config. Also what does "discovery" involve? We shouldn't use launch configs for anything that isn't basically running/debugging a program.

[roblourens]

Another suggestion, I don't know how much complexity is behind the "python-pytest" debug adapter, but if the point is just to invoke the python DA with some args, another option would be to add a config snippet to help users figure out how to write their pytest launch configs. An example of this is the snippets that we have for js-debug to help users write launch configs that start an npm script.

[eleanorjboyd]

Yes, I forgot to update the "name" of the launch config to represent that as a user input and instead focus on using "type" to change the behavior.

discovery involves running pytest discovery. It is equivalent to running "python -m pytest --collect-only" from the command line. So it is running python code if that is the concern.

I will look into the js-debug snippet! Thanks

[eleanorjboyd]

adding #18778 to the related issues as this discusses limitations with current setup.

[roblourens]

discovery involves running pytest discovery

Yes- and what is pytest discovery? 😄

and instead focus on using "type" to change the behavior.

I would say this also isn't what I expect- "type" should indicate which debug adapter is being used, and this should just be "pytest" or even "python" or something like that, and then there should be some other argument that changes the behavior.

[roblourens]

In that issue, it sounds like it's discussing limitations with launch configs, not settings? I don't understand what Karthik means when he says

We currently only read this from the launch.json. We could expand the scope of where we read it from, but that has some things that we have not decided yet.

[eleanorjboyd]

in #18778? I tagged it since resolving this args issue would then remove the setting entirely so it would simplify this process since the launch config would be the only place to reference. I mis-attributed it to "limitations" it is more like this issue would provide clarity to the issue brought up in #18778.

[eleanorjboyd]

Yes- and what is pytest discovery? 😄

During discovery pytest goes through and discovers all files that fit the file format for test files and finds tests within those files (both test classes and just functions). It also should read markers to determine which tests to return. So it isn't executing any of the users test code.

I would say this also isn't what I expect- "type" should indicate which debug adapter is being used, and this should just be "pytest" or even "python" or something like that, and then there should be some other argument that changes the behavior.

gotcha yeah that makes sense

[roblourens]

During discovery pytest goes through and discovers all files that fit the file format for test files and finds tests within those files (both test classes and just functions)

This doesn't sound like a good match for a debug config then, if I'm not going to be setting breakpoints and debugging my code. It sounds more like a task perhaps? Or just something that the extension does, configured by some settings.

[eleanorjboyd]

@brettcannon, @karthiknadig, @luabud : thoughts on @roblourens comment?

[luabud]

The thing is that to configure the debugger to debug pytest tests, one needs to create a launch.json file. Given that, I think it makes sense to migrate how users can configure how to run pytest tests in launch.json too.
But the whole issue is that users would like to pass different configurations for pytest discovery, which needs to be done prior run/debug. If the config for running and debugging pytest tests are in launch.json, isn't it a bit weird to have the configuration for discovery elsewhere?

[luabud]

The other thing to call out is .env files -- using launch.json allow users to have different pytest configs with different environment variables per run, which is harder to do with settings

[brettcannon]

I guess my question is is launch.json is just for debugging why isn't it named debug.json? 😉 I personally view launch.json as the place to configuring launching your code which is traditionally for the debugger. As well, I views tasks.json as a way to run something in the terminal that we don't have extension support for and to parse the output (which this isn't doing; it's directly tying into things).

The other thing is that while you might not use the debugger every time, you may want to use the debugger when running the tests, and so it isn't a disconnect, I think, to use launch.json this way. And since discovery and running are tied together, I think putting one somewhere else just because you typically don't debug with it seems like we're putting purity over practicality. I.e., what Luciana said. 😁

Do we know how other testing extensions handle configuration for debugging compared to running? And we can bring this up in the Pacific stand-up if necessary.

[roblourens]

I still don't really understand what discovery is or what the user flow is for this scenario. I haven't used a testing framework that has discovery as a separate step. What is the output of discovery?

So if I want to debug my tests, I'm going to run a "discovery" launch config, wait for that to finish, then run the "run tests" launch config?

[roblourens]

different configurations for pytest discovery, which needs to be done prior run/debug

Sounds like a prelaunch task?

[brettcannon]

I still don't really understand what discovery is or what the user flow is for this scenario. I haven't used a testing framework that has discovery as a separate step. What is the output of discovery?

What tests are known to pytest. We need to do this to get the list of tests to display in the test explorer. Otherwise we would have to execute the tests in order to know which tests exist. How does js-debug know what tests to list in the test explorer?

So if I want to debug my tests, I'm going to run a "discovery" launch config, wait for that to finish, then run the "run tests" launch config?

You won't directly, but yes because we need to know about the test in order for you to be able to run it, debugger or not. Now if you're asking about when you run your tests after we know the test exists, then the discovery step is skipped as we already have a way at that point to tell pytest "run this specific test you told us about previously".

Sounds like a prelaunch task?

If by "prelaunch" you mean "before running tests", then yes. But if that's a push for using tasks then I still think that disassociation is going to confuse people. To a Python developer, test discovery is just a flag/command to the test runner, not a separate task (it essentially a dry run of tests where you are saying, "what tests can you find?", and skipping the actual execution of the tests).

[roblourens]

How does js-debug know what tests to list in the test explorer?

There's no connection between js-debug and the test explorer. Our self-host test provider that we use for the vscode repo https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-selfhost-test-provider creates a filewatcher for our test files, and parses them when they are added or changed. It's automatic there, I don't know about other test provider extensions. I wouldn't expect to run a launch config to populate the test explorer, that feels backwards.

I didn't realize that this is about the test explorer. So is the expected user flow like this?

1. Run the "discovery" launch config
2. Tests show up in the test explorer
3. Debug a test

And does 3 happen via the Test Explorer UI or the proposed Python: pytest run launch config?

[brettcannon]

parses them when they are added or changed.

pytest and unittest handle the test discovery, so we need to have those tools tell us what they find.

1. Run the "discovery" launch config

We run the discovery launch config; there's no manual step here beyond any custom configuration from the user.

This issue is just about configuration; the user doesn't perform any extra action for testing because of this and this is only for special cases where users need to specify extra details.

[roblourens]

We run the discovery launch config

Launch configs are usually for users, they don't get run automatically in the background by an extension. When you do that, the debug toolbar will show up and the statusbar will turn orange and that will be spooky if it happens by itself.

[brettcannon]

• Are these settings an alternative to labelling the configs as pytest/unittest or are they doing something else?

Something else; they already exist as settings for the extension to control which testing framework you are using and thus which one you want us to integrate with.

[eleanorjboyd]

Some feedback reading through:

• If your configs map to TestRunProfiles, you don't need to have settings for the default configs, as these are handled/persisted by the editor. That also solves some things like "where to set which config to run?"

Yes this looks perfect!

• Can pytestEnabled/unittestEnabled be covered by the presence of the associated run config? / are there cases where I could have one but not the other?

I think this is possible that the configs could have an associated pytest/unittest with each and it runs the given one. So if you select "run profile A", somewhere in the config for "run profile A" you defined it runs with "pytest" so then anytime you attempt "run profile A" it selects "pytest". @brettcannon or @luabud is this too hard for users to do or to discover? I am unsure of how often people want to switch between "unittest" and "pytest" and therefore how important this consideration is.

• I would try to minimize how much the user needs to cross-reference between settings.json/launch.json, especially since I don't think you can do much with intellisense as an extension there

Yeah I think this is a very good point. If a user presses the "debug" button on the test explorer with a given set of args / env vars what is the difference then them create a config in their launch.json with the same args/env vars and running it there? Wondering if we just allow the option to use launch.json to continue as it is and we focus on just expanding the test explorer.

• Is it the most common case that a user will use the same configuration to both run and debug? If so, are there design decisions that can be made to simplify that path?

Yes I think so. I think it would be good since you could set a different default per run / debug if we maybe just offer every run config as an option on the debug config list. Like if the person creates a config "profile 1" they will see that in both the "run" and "debug" dropdowns? Or do we think people will find this cluttered / not enough control? I guess we could also always add an arg that is like "show for: debug, run" and the default is both but could be changed.

[eleanorjboyd]

The dropdown looks great! Sorry I missed this earlier!!! (also idk why it isn't letting me "reply" like above to your comment, the formatting was getting messed up)

For others here is how it could look with the dropdown:

[eleanorjboyd]

replies to @roblourens comments:

2.1: If it's in a launch.json, it should be debugging by default 2.2: I don't understand what you mean 2.3/2.4: I know you said

Debug by default makes sense, that solves 2.3 as well. 2.4 was just about duplicating settings so I just more wanted to repeat that question there

but I agree with Connor and if I was doing this I would probably leave out this part of referencing settings.json configs for simplicity. I wonder how complex the configurations typically are? But if this is really important, I think the pattern is fine.

I think its common to have a few environment variables and args but maybe @luabud can speak more to this question.

"purpose" doesn't quite mean the right thing to me. I would suggest "extends", implying that it will use that settings.json config as a base, but you can override properties from the settings.json config by setting them in the launch config as well.

yes this sounds fine to use "extends" if we do want them to build off each other.

2.6: There is a builtin command "Run without debugging", it starts your launch config with the "noDebug" property added dynamically, I would suggest building on that.

this is great ya, no need then for 2.6

[brettcannon]

I think this is possible that the configs could have an associated pytest/unittest with each and it runs the given one. So if you select "run profile A", somewhere in the config for "run profile A" you defined it runs with "pytest" so then anytime you attempt "run profile A" it selects "pytest". @brettcannon or @luabud is this too hard for users to do or to discover?

I would expect users to never swap between them once they have chosen their test runner.

For Rob and Connor's edification: pytest can run pytest and unittest testing code (i.e., be a test runner for anyone), but unittest can only run unittest testing code, hence why users even have to specify anything (and this sparks an idea I'm going to go talk to Eleanor about 😁).

[eleanorjboyd]

Following this round of comments, here is my new proposal on the design.

settings.json design

{
      "name": "run config: unique name", // user defined name
      "args": ["-v"],
      "env": {
	"PYTHONPATH": "${workspaceFolder}",
        "DJANGO_TESTING": "true",
	"DJANGO_SETUP_SCRIPT_PATH": "/path/to/file",
      },
       "envFile": "path/to/file",
       "testRunProfileKind": ["discovery", "run", "debug"] // all run kind will also display under debug, but not vice versa 
},
"python.testing.framework": ["unittest","pytest"] // this is NOT required, the default will be used if not included

therefore the simplest config if you do all the most basic setup steps with pytest for example would be:

{
      "name": "pytest discovery config", 
      "args": [], // this might even be optional too- open to suggestions on if this is a required part of the json.
       "testRunProfileKind": ["discovery"]
},

default values

• default value for testing library (unittest or pytest) will be pytest if it is installed in the environment and unittest if not with this value being configurable via a single setting "python.testing.framework".
• the default config will now be defined by the dropdown menu on the Test Explorer tab for run and debug like so:
  

launch.json design
Launch.json configuration for debugging testing will stay exactly as it is now with the current functionality and design. Users will NOT be able to reference their settings.json test configurations from their config.json file. They will need to duplicate / rewrite any configurations they want to use as a debug config there. This will cause duplication of work but I believe will be the simplest for the MVP and I predict users will tend to stick to a single way of use that will make sense to them. @brettcannon, @luabud, @karthiknadig let me know your thoughts on this as I most unsure about this decision and not sure it impact down the road.

Possible UI design:
Will use TestRunProfiles and their related UX. I will update this issue shortly with the flow of how users will set up testing once we switch. Additionally @connor4312 would it be possible to get a dropdown the same as on the "run" and "debug" icons on the reload icon to allow us to configure different profiles of kind "discovery" that could appear here? With the same option to set a default?

[brettcannon]

If we can't unify in a single file then I am afraid duplication is just unavoidable. We can operate under the assumption that if you're debugging a test it's a single test and thus going to be a bit specific rather than generic to your whole test suite.

One question about "python.testing.framework" is whether that should be global or should we make a per-config setting (thinking of the monorepo case)?

[cameronbrill]

hi @eleanorjboyd! I love this proposal, it would fix an issue my org has with our test suite (we use pytest-xdist to parallelize test runs, but this breaks when we try debugging tests in VS Code). Is there a timeline for when this will be supported?

[brettcannon]

Is there a timeline for when this will be supported?

Eleanor is busy w/ non-test stuff this month, but the plan is to pick it back up in April.

[alenzo-arch]

+1 for this. The collection thing has been really frustrating. Not just that it doesn't work with some args, but also that if you have coverage enabled like so, it overwrites the coverage report. Not critical but it is annoying if you have it setup to discover on save.

The debugging portion of this is a deal breaker. It's working again now, but my issue #22820 is an example of how the current solution is brittle.

"python.testing.pytestArgs": [
        "--cov",
        "--cov-report=html",
        "-vvv",