Investigate adding an exit survey for Native Editor

[jmew]

Given criteria on whether someone has churned the native editor, prompt for an exit survey to better understand their churn reasons

E.g.
-uninstall python ext after using Native editor for "X" # of days
-uninstall VS Code after using Native editor for "X" # of days
-used the native editor more than "X" number of days but then havent used the editor for like "Y" number of days but still uses the Python extension regularly

Numbers would need to be fine tuned

[DonJayamanne]

Once extension or VS Code has been uninstalled, there's nothing we can do. That rules out 1 & 2.
The third one is possible (i.e. use native editor more than x days...).

[DonJayamanne]

I'd consider adding a survey when a notebook is used. This gives us the opportunity to get first impression feedback:

• After running 100 cells in total
• 30minutes after opening a notebook & no cell was run - possible this user didn't get jupyter to start. If not, then what else are they using this for (feedback)
• After opening an nb 5 times
• Etc...

Oh yes, I'd only consider this for a small number, e.g. 5-10% of our users.

[jmew]

@DonJayamanne I agree, i think the entrance survey is good too to get first impressions, i think we should do that as well. That + the exit survey would be pretty good knowledge

[jmew]

Can we change when the current DS Survey Monkey survey is triggered? Right now it is >500 DS commands, but I would like to change it to Don's criteria to get a better representation of initial feedback.

File containing the survey logic can be found here: https://github.com/microsoft/vscode-python/blob/4883cff5c8a16d69f81a40e82694ccccc1f1acba/src/client/datascience/dataScienceSurveyBanner.ts#L85

Prefer if this could be done for the next release

[rchiodo]

@jmew Just to be clear, you want this?
Survey when:

• After running 100 cells
• After having a notebook open for more than 30 minutes and not running a cell
• After opening 5 notebooks

Anything else?

[DonJayamanne]

After having a notebook open for more than 30 minutes and not running a cell

I'd ensure we do this across VSC sessions, but in same workspace folder (i.e. user might re-start VS Code in an attempt to get things working).

@jmew if I'm not mistaken, we're trying to identify why the user didn't run a cell and its possible things didn't work out hence we'd like some feedback.

If this is the case, then we'd want to capture this telemetry ONLY if a user attempted to run a cell and running a cell did NOT work.
Else, I wouldn't want to throw a popup with a survey asking a user how nb experience is when all they're doing is previewing a notebook! Also, this will take away other users from the pool of users who didn't manage to run a cell successfully.

[jmew]

@DonJayamanne good point, when this issue was first created, we weren't like 100% sure on what other major things could be causing the churn, but now that we're pretty confident that its the getting started process, we should switch up the logic to check only when the user runs a cell but it didnt run for whatever reason.

Also, ensure that the popup only appears once per user as to not annoy them.

Let me know as well when you need the survey link @rchiodo

[IanMatthewHuff]

Validated. It did take more than 100 cell executions (132 I think?) but I was hammering on ctrl+enter so I probably just flooded it.

[IanMatthewHuff]

Also did 6 notebook opens to show it.