Test: Offline/disconnected BYOK

[dmitrivMS]

Ref: #246551
Ref: #275591
Ref: #313461
Ref: #272843
Ref: microsoft/vscode-internalbacklog#7044

• Linux @zhichli
• macOS @vritant24
• Windows @hediet

Complexity: 4

Authors: @dmitrivMS

Create Issue

---

This TPI covers scenarios of using AI features via BYOK models with or without GitHub sign-in/network.

BYOK models are configured via Manage Language Models command (via UI or in JSON). You can use OpenRouter or any other BYOK model for online scenarios and Ollama or any local model for offline scenario (without network).

The following group of features (GitHub-only features) can only be used via GitHub account - they are not available via BYOK with or without utility models:

• Semantic search
• NES/editor completions
• Any feature that relies on embeddings

There is a number of AI features which are enabled by utility models (Utility model features) - these are available either via GitHub account or by directly setting chat.utilityModel and/or chat.utilityModelSmall settings to a BYOK model. When not signed in and those settings are not set, these features will not work. The list is long, but they include session title, progress message in chat, commit message generation and many others.

For all scenarios below are sample workspace with some files is recommended, but not required.

Scenario 1: GitHub AI still works
Start VS Code with fresh user profile (--transient).
Make sure you see all the usual sign-in prompts for AI (in Welcome dialog, title bar, status bar, gear, chat watermark).
Sign in to GitHub account, make sure all prompts disappear and you observe the usual AI behavior (verify features from all categories above - GitHub only, Utility model and chat (try tools/terminal commands).
Your goal is to ensure all features work as normal.
Sign out of GitHub account, verify you are back to seeing prompts to sign in to use AI.
You can now sign back in and proceed to scenario 2.

Scenario 2: BYOK with GitHub works
After signing into GitHub account add a valid BYOK model.
Use all categories of AI features, and use chat with BYOK model - make sure everything works as expected.
Things to be aware of: you should not see any prompts to sign-in after adding BYOK model, you should not see notification in the chat about utility models (since you are signed into GitHub).
After testing all scenarios, sign out of GitHub account but don't remove the BYOK model.
You should not see AI sign-in prompts (compare to scenario 1) except for 2 locations: 1) in gear menu 2) in status bar "frog" icon.
Try using one of those items to sign-in, make sure you are back to signed in state and things work as before.
Now sign out again, switch to chat and select BYOK model - you should see a notification about setting utility models.
Make sure clicking the notification takes you to the settings UI but don't set utility models yet.
Proceed to scenario 3.

Scenario 3: BYOK without GitHub works
Use the chat with BYOK model to perform various tasks (edit code, use terminal), make sure things work as expected.
Note that you will have limited functionality due to missing utility models (e.g. session title will not be updated etc).
Close and reopen VS Code, verify you are back to the same state - i.e. no sign-in prompts.
Use chat with BYOK model, make sure it's responding.
Reload VS Code window, verify chat is responding.
Open a new VS Code window and use chat with BYOK model there, make sure things work normally.
Open settings and set chat.utilityModel and chat.utilityModelSmall to a BYOK model.
Verify chat and other functionality (such as title update, commit message generation, etc) now works with utility models initialized
Reload VS Code and verify same functionality still works and you don't see notification in the chat about setting BYOK models.
Now remove all BYOK models from Manage Language Models UI/JSON.
Verify that you are back to initial state of VS Code where sign-in prompts appear inviting you to sign in to use AI.

Scenario 4: BYOK from scratch
Start VS Code in a new user profile (--transient) and without ever signing into a GitHub account add BYOK model via Manage Language Models command (it should be available).
Then proceed testing through the steps in scenario 3.

Scenario 5: Offline BYOK
Install a local BYOK model (.e.g. Ollama), install VS Code but don't open it.
Disconnect your computer from network.
Start your model and open VS Code, add the local BYOK model via Manage Language Models UI/JSON and proceed testing through the steps in scenario 3 (obviously you want to perform operations that don't require network).
Verify things work as expected (i.e. code is edited, terminal works etc), verify utility models operate as expected once set (title update, commit message generation etc).

[hediet]

In scenario 1, after signing out, I see this (focus border even though it is not focused):

After reloading it is gone:

Also, after removing model, it stays selected (though I think it is fine to leave that unfixed):

I used gemma 4 on a desktop computer and it worked okay! (nothing I'd use for real work though)

[vritant24]

For scenario 2, After signing out, I still see this:

I also saw this when I was signed in, which seems incorrect considering the same account in regular insiders doesn't see this (it's my microsoft linked account)