Debug setup simplification (user testing)

[isidorn]

1. Review feedback from the first round of user feedback
2. Based on feedback try to simplify debug setup
3. Run another round of user feedback to get feedback on potential changes / explorations

In general try to simplify setting up debug as much as possible.
Also related #285

[stevencl]

Based on what we learned from the studies, I believe that we have the following set of problems that we need to address

1. Discovering debug functionality. This wasn't a major issue, but it was a stumbling block.
2. Choosing a debug environment. The main problem here is that this is an unexpected requirement, which causes users to dismiss the prompt that appears as soon as they click 'Start debugging' sometimes without even reading the prompt.
3. Choosing a startup file/program. Once users got past the debug environment issues, they were stuck on specifying the appropriate value for the program attribute.

Suggested solutions.
For addressing debug viewlet and commands discoverability:

1. Do nothing about the initial lack of discoverability. The problem we observed is likely the experience that people have the first time that they use the product. We didn't observe anything that would suggest that this is a persistent problem.
2. A global debug menu would advertise functionality. It could expose actions such as 'Start debugging', 'Select debug environment', 'Select startup file/program'. 'Stop debugging. The first action could be disabled until the environment and startup file have been selected. Other actions could be exposed such as 'Show debug console', instead of placing them in the View menu. However, one downside to this solution would be that it adds yet another top level menu, increasing the UI surface area and potentially adding distracting UI.
3. Instead of a debug menu, consider placing a Start Debugging action in the breakpoint context menu. This avoids adding more clutter to the UI but is less discoverable than a global menu. It also doesn't draw people to the debug viewlet.

I believe that option 1 is currently the best approach, with the addition of perhaps adding the start debugging action to the breakpoint context menu.

Choosing a debug environment

1. Choose this automatically based on file extension of the current active file in the editor. For example, running the same scenario we used in the study inside VS2017, I was able to start debugging a .js file without having to specify the debug environment or the startup file. This would only work for supported environments.
2. When the user sets a breakpoint in a file we show UI that prompts them to specify the debug settings. We prefill the settings based on the file that the user set the breakpoint in. VS2017 shows a floating toolbar when setting a breakpoint and when hovering over the breakpoint. We could do something similar?

We might consider both options. We'd need to consider what the long term effect of this is with respect to users learning about the launch.json file.

One way we could address the lack of visibility for the launch.json file would be to write a message in the debug console saying that we are debugging this file based on the settings inside the launch.json. We could have a link in the message that allows the user to change these settings.

Choosing a startup file/program

1. Similar to above, if we could set this automatically based on the current active editor, that might be a good start.

I think overall what we want to do is to add minimal UI that points people towards launch.json so that they learn about how to use it over the long term and in more complex scenarios. If we can automate a lot of the getting started process but at the same time point people towards how to set this up themselves, then we avoid the need for more permanent UI which could ultimately become distracting.

Adding @bgashler1 for his thoughts too.

[isidorn]

@stevencl thanks for the nice writeup! I agree that these problems exist

Addressing debug viewlet and commands discoverability

I agree that doing nothing for now is best here. It is always easy to add more stuff (menu items) but a problem comes when we become cluttered and need to remove something.
Another two reasons why I am not concerned here:

• F5 is a generally well known shortcut to start debugging
• Viewlets are always visible on the left, and as soon as the user gets in touch with them he will find debugging

Choosing a debug environment

For this we currently have a technical issue that vscode is not aware of what environment debugs what language. Namely, VSCode does not know that our node debugger is interested in .js files.

I have an idea on completely leaving this step out. Namely we would generate an empty launch.json which would immediatly go into snippet mode and the user would choose if he wants to add an attach or launch configuration for node / c#.

Choosing a startup file/program

Yes, we could set this by default to be the active editor. And if the user does not like that he can change it in the launch.json

I would like to produce a vscode build that does not have 'choose a debug environment' step and has active editor as the default program. After we have a build it would be great to collect another round of feedback.

@weinand fyi

[bpasero]

Maybe also interesting is to explore what @chrmarti is doing with a welcome page on first startup for new users. Since he is able to embed the editor into the welcome page, it would imho be cool if you had a little debug toolbar in the welcome page to start debugging and play around with the debugger in a simple sample.

[chrmarti]

@stevencl Haven't seen this mentioned here: The user testing included the instruction to open a folder. That itself is a hurdle in addition to the ones listed above. (Related: #285)

[stevencl]

Good point @chrmarti. We left it out of the study because we know it's a problem but we should make sure we address it.

[isidorn]

Ok - just had a better idea for

Debug setup simplification

We will remove extension development and node2 as the environment options. It will be possible to add both of these using the add configuration button in launch.json.
This puts us in a spot that brand new users only have the choice of node, so we would automatically pick it and not even show a envionrment picker dialog. Once the user installs some extension we will show the dialog - but then he is no longer a complete begginer and will not be scared of the dialog.
As @chrmarti suggested in that case we also show a "More..." at the end so people can discover other debuggers

[isidorn]

The insiders build tomorrow will contain the changes mentioned above

1. No choose environment dialog for out of the box experience
2. ${file} is the default program for node (the opened file will be chosen for debug)

@stevencl it would be cool if we could run another round of 5 participants.
In the future it would also be interesting to see some studies with experienced vscode users - not only complete beginners

fyi @roblourens @weinand I released new versions of your adapters to the marketplace 1.9.2 (since for now I removed the initialConfigurations from extension development and node2 - these adapters can still nicely add snippets, and in the future when we replace node with node2 node2 can again provide initialConfigurations)

[stevencl]

Sounds good, once the build is ready I'll kick off another round of studies.

[isidorn]

After a discussion in the standup we agreed that the "select environment" step is always needed, even when there is only node debug to choose from as we now show the 'More...' option. And not showing anything would worsen the experience a lot for PHP users as an example.
We can get rid of this step once we have more infrastructure and we are able to tie a project to a language - we currently are not there yet.

Even though we are still showing this step it would be interesting to run these studies as we have simplified it by showing less options and the ${file} should also help.

[stevencl]

Hmm, the select environment step was completely unexpected for the participants in the last study we did. Ultimately, if we absolutely must ask the user for the debug environment, I think we need a better way of asking for this, something that explains why we are asking for this information. Without that motivation we run the risk of people simply ignoring the prompt (as happened in the study).

[isidorn]

After digesting the feedback we got from users here are the updates we have done

1. Enabled debugging in no-folder workspace (currently only supported by node but soon to be picked up by other debuggers)
2. Enabled debugging in folder workspaces that do not have a launch.json - this made debugging much easier for participants
3. (available in tomorrow's insiders) Only asking the user to pick the environment if we can not deduce ourselves. Namely if the user has a javascript file open we will automatically pick the NodeJS environment making it easier to create an initial launch.json

I believe these items simplify the initial start debugging experience a lot. We should still invest into how to make users not scared of launch.json and how to make them feel more comfortable in editing a launch.json. This is mostly seen for users that have no experince in JSON. However this should be captured in a seperate issue and this might be a general concern as our settings.json is also a json file.

Closing this issue as we have done all that was planned.
Thanks again @stevencl for running the studies and feel free to open seperate feature requests if I failed to capture some of the useful feedback which we got.