App-Less Onboarding Experience #632
Comments
|
Sharing the spec here for visibility as well: https://docs.google.com/document/d/1d3AsJeAB3AuUwAUTvvMzwKVQeVniX1Q06LPCBna5BbU/edit#heading=h.hu0rd8wz6rcd |
toolmantim
added
the
design
toolmantim
changed the title
|
Draft designs for these are ready for feedback in Figma! New sign in:
New welcome chat experience w/ optional app download:
Notes:
|
|
I've updated the Figma with all of the states I can see we will need to handle:
We're also going to need to handle Cody Enterprise embeddings states as well in this update, so I'll work on those next to ensure we've got a view across everything going into the eng work. |
|
@dominiccooney @chenkc805 here's a simplified design, focusing more on the simplified App-less sign-in, and less on creating a great embeddings experience:
Design notes:
|
|
Engineering update:
|
|
Post sign-up survey redirect WIP at sourcegraph/sourcegraph#56360 , this just needs automated tests. It's hard to test GitHub, GitLab auth with sourcegraph.test but this works with email sign in. Unlike GitHub, GitLab sign-in, email sign-up has the verification step; if users aren't signed up the email flow may need some more work. |
|
As per this Slack thread: https://sourcegraph.slack.com/archives/C05PC7AKFQV/p1694105019978109 @dominiccooney we're going to make 2 changes to the VScode signup flow in light of the security incident:
|
This adds a simplified login interface for easier onboarding. Part of #632 Caveats: - No automated tests yet. - No experiment selection or caching yet, noted with TODOs. - No exposure logging yet, noted with TODOs. ## Test plan Check nothing broke: ``` pnpm test:unittest && pnpm test:e2e && pnpm test:integration ``` Will add automated tests for this new UI in a follow-up change. https://www.loom.com/share/3660f9da9e7f45b8b6fb4032ac91360f?sid=3db70fde-fe8e-4c3a-a36e-bf14d47406d4 Manual test: 1. Go to User Settings (JSON) 2. Set a key `cody.todo-internal-onboarding-qa` to any value. 3. Run the extension. Sign out if necessary new sign in UI should appear. For each of GitHub, GitLab, Google: 1. Open a browser and sign out of sourcegraph.com and the provider 2. Click the sign in button in the extension 3. Sign in to the provider 4. Complete the post-sign up survey, if necessary. 5. Open the VScode URL and you should be logged in. 6. Log out. The new login experience should appear. Repeat, but be signed *in* to the identity provider already. Repeat all of the above with different default system browsers, operating systems, etc. Click the Sourcegraph Enterprise sign-in link and confirm those sign in options continue to work.
|
This is all currently pretty specific to VS Code. Is that an intentional message, that is, we only want to adjust this flow for VS Code, or is this client only an example, and we want all clients to use this flow? I think our intention is the latter, and I'd, at the minimum, include adjusting the flow for JetBrains IDEs in this project. Even if we decide to include JetBrains, it might not be worth it to tweak all our existing docs, issues, mocks, etc. with this in mind, but going forward, I suggest explicitly describing which clients we want to include and that VS Code is just one example. |
|
I want us to test this first in VSCode before shipping to all clients. I'm fairly confident it will do well so we should build this in an extensible way for the other clients, but we'll test with VSCode first and then if it does well for VSCode, we'll ship to all other clients |
@chenkc805 , @vdavid After we validate this is VScode, here's a doc outlining what other clients need to do. The way VScode is implementing this leans heavily on dotcom and the existing token provisioning method. Other clients should be able to adopt it without problem! 🎉 @olafurpg had a similar question and has thought through the approach for JetBrains/IntelliJ. The doc is exactly what we discussed and adds more technical detail. |
|
Doc with logging design. Thanks @chenkc805 for feedback. Will implement in a follow up to #1036 To do:
|
Users are randomly assigned to get the classic onboarding experience
('control') or a simplified onboarding experience without app setup
('treatment'.) This allocation is hard-coded in the extension; if we
want a different rollout we need to update the extension.
Once we render the login component, we consider the user has been
exposed to a specific arm of the trial and we cache this fact in local
storage.
Currently fixes the rollout at 0.5 so people running main will get
exposed to the experiment. We can bump `SIMPLIFIED_ARM_ALLOCATION` back
to zero if we want to hold back the experiment.
Testers can set a boolean property, `testing.simplified-onboarding`, to
force the extension to display the treatment (true) or control (false)
arm of the trial.
Part of #632
## Test plan
Automated test:
```
pnpm test:unit # new tests for experiment arm assigment, caching
pnpm test:e2e && pnpm test:integration # hard-coded to use the classic onboarding
# e2e tests for the new login flow TBD
```
Manual test:
1. Cody, ... menu, Sign out, sign out.
2. Control ("Other Sign in Options..." etc.) login should appear.
3. Open the VScode Output pane and change the dropdown to Cody by
Sourcegraph. This log message should appear:
```
logEvent (telemetry disabled): CodyVSCodeExtension:experiment:simplifiedOnboarding:exposed {"arm":"control","excludeFromExperiment":false}
```
4. Verify that login, etc. works.
5. Preferences: Open User Settings JSON. Add a key
`"testing.simplified-onboarding": true`. Reload VScode.
6. New login experience should appear.
7. VScode Output pane, this log message should appear:
```
simplified-onboarding: not logging exposure for testing override selection
```
8. Verify login works.
|
QA notes here: https://docs.google.com/spreadsheets/d/1bzrQT_QGYr0xstMSVj6U8DWbI5wbERFZKk41W7E-m2U/edit#gid=0 ✅ Overall seems good. It's hard to test new SG user flows with new GitHub accounts I just created, due to the security incident, but the one I was able to do (Google SSO) worked correctly (saw the "for work/personal" flow and then after that got pushed back to the IDE). So I think it's safe to assume Github and Gitlab probably work as well. ✅ You got the sign off from me! @toolmantim can you give this a look from the design POV? |
|
@chenkc805 , regarding
Yes, we use the tokens for accessing the LLMs but the state is stored locally and not associated with accounts. Good feedback for design that this is confusing. |
|
@chenkc805 @dominiccooney looks good to me too! 🚀 |
|
@dominiccooney can you link all PRs related to this work here? I'm trying to organize them into this issue so this issue is the master issue and all your PRs are sub-tasks of this issue. That way we can keep track of this body of work in 1 place. |
Adds popups to install app, set up embeddings, etc. when using the simplified onboarding flow. Automatically switches to app for embeddings, when app has embeddings available. Polishes vscode storybook so codicons work and there are more relevant style variables. Part of #632.  ## Caveats - The app prompts are still displayed on Windows, but there's no Windows app to download. - [These prompts](https://www.figma.com/file/INGthyKM1cYki5H02JPDpW/VS-Code-Beta-2-%3C%3E-App-Onboarding?type=design&node-id=53-374&mode=design&t=ivzuiEzqtVsJtkD8-0) for Enterprise are not implemented. Enterprise uses the legacy widget. - There's little/no logging for interactions with this UI. ## Test plan ``` pnpm -C vscode storybook ``` Open http://localhost:6007 and examine the App-less Onboarding storybooks. Manual tests: 0. Uninstall Cody App. Remove test VScode state: `rm -rf /tmp/vscode-cody-extension-dev-host` 1. Run VScode with "Launch VScode Extension (Desktop; separate instance)" (equivalently, `--user-data-dir=/tmp/vscode-cody-extension-dev-host`) so you have no App token cached. 2. Open user settings JSON and add `"testing.simplified-onboarding": true` and restart VScode. 3. Create a new git repository with `git init /tmp/bar; cd /tmp/bar; git remote add origin git@host.example:bar/bar.git` . Open `/tmp/bar` in VScode. 4. Sign in with simplified onboarding. 5. The context underneath the chat box should be red. Click to open the notification. Check the link and Install App buttons work. 6. Install App. Click the Reload button. The prompt should change to Embeddings Not Found/Open App. 7. Click Open App, add embeddings for the repo. Wait for embeddings to complete. Back in VScode, click the Reload button. The indicator should change to a checked database icon. Or, if App presents a "back to VScode" button, click it and the indicator should change automatically. Or simulate that click by going `open vscode://sourcegraph.cody-ai/app-done' at the console. 8. Close app and wait 20 seconds. The indicator should go red/show a cross. 9. Open app and wait 20 seconds. The indicator should be checked. 10. Close app. Wait for the indicator to go read. Immediately open app, pop up the toast, and hit reload. The indicator should be checked immediately. 11. Clone something indexed by dotcom, for example, https://github.com/preactjs/preact . Open in VScode and open a file in the repository. The database icon should be checked (that is, using dotcom embeddings.)
|
@dominiccooney when can we start QA'ing the post-sign in experience with the App? CC: @MaedahBatool as we'll probably need to update some docs around this |
…in (#1210) LocalAppDetector stops monitoring app once the user has signed in. Simplified Onboarding needs to pick up the app token, even if the user has logged into dotcom, so that we can use app for local embeddings. This makes simplified onboarding's LocalAppWatcher ping the LocalAppDetector to retrieve the app token, if it detects a change in the app.json file, even after the user has logged in. Part of #632. ## Test plan 0. Open Cody App, add a repo, and add embeddings for a private local repository which has a remote. 1. Quit Cody App if it is running. Open Keychain and delete any vscodesourcegraph.cody-ai tokens. Delete ~/Library/Application Support/com.sourcegraph.cody/app.json . This means VScode will not have a cached app token. (Some uncertainty around where VScode stores secrets across versions. You can set a breakpoint in the extension LocalAppWatcher and run `secretStorage.deleteToken(LOCAL_APP_URL.href)` to clear the token. Alternatively, logging into App and signing out should also clear that token.) 2. Run VScode, edit User Settings JSON and add `"testing.simplified-onboarding": true`. Restart VScode. 3. Log into dotcom using GitHub, etc. if necessary. 4. Open the repository from step 0, open a file from the repository, click on the chat input status indicator. You should see a pop-up to open app. 5. Open app and wait up to 20 seconds. Chat input status indicator should flip to having a checkmark indicating VScode has detected app is running, picked up the token from app.json, and found app has embeddings for the repository. 6. Close app and wait up to 20 seconds. The chat input status indicator should flip to red/open app popup.
|
@dominiccooney can we close this? |
|
@chenkc805, @toolmantim and I plan some follow-up design improvements this week, then we can close this. |
Dotcom/App users can always click on the chat input status indicator and see the state of embeddings in a popup. Caveats/limitations: - There's no progress indicator while app is indexing. You need to wait and hit the "Reload" button. - Only for dotcom/App users however this will be fixed in a follow up PR. - Various existing quirks of local embeddings remain for now. For example, your git repository must have a remote with a slash in the name. Part of #632 ## Test plan This does not add automated tests yet. That is in a follow-up change, I need to update the existing tests to use simplified onboarding first. UI check: `pnpm storybook` and examine the App-less Onboarding stories. Manual tests: *A directory that's not a git repository:* ``` mkdir /tmp/project cd /tmp/project echo hello > main.c code . ``` Open `main.c`, open Cody, click on the chat input status indicator:  *A git repository that does not have a remote:* ``` git init . ``` Cmd-Shift-P, Reload Window (we do not watch for repos changing status in this way.) Chat status indicator should be as above. *A git repository with a remote:* Ensure App is not running. ``` git remote add origin git@host.example:cool/project.git ``` Cmd-Shift-P, Reload Window (we do not watch for repos changing status in this way.) Chat status indicator:  *A git repository that is indexed with app:* ``` git add main.c git commit -am 'first commit' ``` Install app if it is not installed. Click Open App and go through the repository indexing flow. If there are existing repositories click Add a repository. You can force repo indexing by right clicking on App UI, Inspect Element, `window.location = '/api/console'` in the Console, and run this GraphQL mutation: ``` mutation addrepo { scheduleRepositoriesForEmbedding(repoNames: ["host.example:cool/project"], force: true) { alwaysNil } } ``` Note there is no *.git* on the repo name. Check that there is a job with state "COMPLETED": ``` query arewethereyet { repository(name: "host.example:cool/project") { name id embeddingJobs(first: 2) { nodes { state revision { oid } startedAt stats { filesSkipped filesScheduled filesEmbedded } } } } } ``` In VScode, open popup and click Reload, status indicator:  *git repository indexed by dotcom:* ``` cd /tmp git clone git@github.com:facebook/react.git --depth=1 ``` In VScode, File, Open Folder, `/tmp/react` and trust the authors. Chat input status:  Repeat the above looking at different themes. --------- Co-authored-by: Tim Lucas <t@toolmantim.com>
|
Onboarding is now app-less and we've done about as much clean up as we can without removing app entirely. I plan to track the work for completely app-less in #1443 so closing this issue now. |
To reduce friction in getting started with Cody, we want to make downloading Cody App an optional step that happens much later in the onboarding process. This will mean:
PRD: https://docs.google.com/document/d/1d3AsJeAB3AuUwAUTvvMzwKVQeVniX1Q06LPCBna5BbU/edit#heading=h.hu0rd8wz6rcd
Overall flow:
https://docs.google.com/drawings/d/1AboEUVSGurFaQqgZww-sNAwTj33GlbH7fpGnS6A1D-o/edit
Design wise, we'll need:
Tasks
The text was updated successfully, but these errors were encountered: