From a3f826a53564fdfd9ea06a2c67bd176b377defab Mon Sep 17 00:00:00 2001 From: xinyi-gong Date: Fri, 8 May 2026 14:04:29 +0800 Subject: [PATCH 1/2] add test plans for endgame 0.17.0 --- .../test-plans/add-context/add-context.md | 120 ++++++ .../test-plans/model-picker/model-picker.md | 354 ++++++++++++++++++ .../test-plans/nes/nes.md | 311 +++++++++++++++ 3 files changed, 785 insertions(+) create mode 100644 com.microsoft.copilot.eclipse.swtbot.test/test-plans/add-context/add-context.md create mode 100644 com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md create mode 100644 com.microsoft.copilot.eclipse.swtbot.test/test-plans/nes/nes.md diff --git a/com.microsoft.copilot.eclipse.swtbot.test/test-plans/add-context/add-context.md b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/add-context/add-context.md new file mode 100644 index 00000000..05d214ca --- /dev/null +++ b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/add-context/add-context.md @@ -0,0 +1,120 @@ +# Chat: Add Context (Attach Files) + +## Overview + +Tests the **Add Context** button in the Chat view, which lets users attach +workspace files to a chat prompt. Attached files provide additional context to +Copilot so it can give more relevant answers. The button is an icon-only flat +button (tooltip: **Add Context...**) in the action bar's file reference area. + +After attaching, files appear as **chips** below the input area showing the +file name and a close (×) button. Users can click a chip to open the file, or +click × to remove it. + +Entry points exercised: +- **Ctrl+Alt+I** (or status bar → **Open Chat**), then click the Add Context + button in the action bar. + +--- + +## Prerequisites + +- Eclipse IDE with the GitHub Copilot for Eclipse plugin installed and + activated. +- A signed-in Copilot account. +- A Java project is open in the workspace with at least two files. + +--- + +## 1. Attach files, verify chips, remove, and send with context + +### TC-001: Attach files via the Add Context button, verify file chips, remove a file, and send a prompt with attached context + +**Type:** `Happy Path` +**Priority:** `P0` + +#### Preconditions + +- The Chat view is open. +- The workspace contains at least two Java files. + +#### Steps + +1. Locate the **Add Context** button in the action bar (icon-only button with + tooltip **Add Context...**). +2. Verify the button appears as a flat icon without a visible rectangular + border. +3. Click the **Add Context** button. +4. Verify a file picker dialog opens (title: **Search attachments**). +5. Select two files from the workspace and click **OK**. +6. Verify the dialog closes and two **file chips** appear below the chat input + area, each showing the file name and a close (×) button. +7. Click the × button on one of the file chips. +8. Verify that chip is removed and only one file chip remains. +9. Click on the remaining file chip's file name. +10. Verify the corresponding file opens (or is revealed) in the editor. +11. Type a prompt (e.g. `explain this file`) in the chat input and click + **Send**. +12. Wait for the Copilot turn to complete. +13. Verify the response references or uses content from the attached file. + +#### Expected Result + +- The Add Context button opens the file picker dialog. +- Selected files appear as removable chips below the input area. +- Clicking × removes the chip. +- Clicking the file name opens the file in the editor. +- Sending a prompt with an attached file includes that file as context in the + conversation. + +#### 📸 Key Screenshots + +- [ ] **Add Context button** — flat icon button in the action bar, no border. +- [ ] **File picker dialog** — dialog showing workspace files to select. +- [ ] **File chips** — two file chips visible below the chat input area. +- [ ] **After removing one chip** — only one chip remaining. +- [ ] **Response with context** — Copilot turn that references the attached + file content. + +--- + +## 2. Currently active file is shown automatically + +### TC-002: The currently open file appears as a reference in the action bar + +**Type:** `Happy Path` +**Priority:** `P0` + +#### Preconditions + +- The Chat view is open. +- A Java file is open and active in the editor. + +#### Steps + +1. Open a Java file in the editor (click on it in the Package Explorer or + switch to an already-open tab). +2. Switch focus to the Chat view. +3. Observe the file reference area in the action bar. + +#### Expected Result + +- The currently active editor file is automatically shown as a reference in + the action bar (displaying the file name). +- Switching to a different file in the editor updates the displayed reference. + +#### 📸 Key Screenshots + +- [ ] **Current file reference** — the action bar showing the active file name + as a context reference. + +--- + +## Notes on failure modes + +- File picker dialog does not open → the Add Context button's selection + listener may not be wired; check the Eclipse error log. +- File chips do not appear after selection → the ReferencedFileService may not + have received the files; verify the file selection result is non-empty. +- Clicking × does not remove the chip → the close button's mouse listener + may not be attached; check the ReferencedFile widget. diff --git a/com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md new file mode 100644 index 00000000..ea39eab7 --- /dev/null +++ b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md @@ -0,0 +1,354 @@ +# Chat: Model Picker + +## Overview + +Tests the **model picker** dropdown in the Chat view action bar. The model +picker lets users choose which AI model powers their chat conversations. It +displays available models grouped by tier (Standard, Premium, Custom), shows +per-model details on hover, and includes a **Manage Models…** shortcut for +users with BYOK enabled. + +Entry points exercised: +- **Ctrl+Alt+I** (or status bar → **Open Chat**), then interact with the model + picker button in the action bar. + +--- + +## Prerequisites + +- Eclipse IDE with the GitHub Copilot for Eclipse plugin installed and + activated. +- A signed-in Copilot account. Without a valid account the model list cannot + be fetched and the picker will show no items. + +--- + +## 1. Model picker loads, opens, selects, and routes a message + +### TC-001: Open model picker, browse grouped models, switch model, and verify a response uses it + +**Type:** `Happy Path` +**Priority:** `P0` + +#### Preconditions + +- The Eclipse workbench is open. +- The user is signed in to Copilot. +- No previous Chat view is open. + +#### Steps + +1. Open the **Copilot Chat** view via the keyboard shortcut **Ctrl+Alt+I** + (Windows/Linux) or **Ctrl+Cmd+I** (macOS), or click the Copilot status bar + icon and select **Open Chat**. +2. Wait for the Chat view to fully load (the input area is editable, the model + picker shows a resolved model name instead of "Loading…"). +3. Verify the model picker button on the right side of the action bar displays + a model name (e.g. `Claude Sonnet 4.6`) with a dropdown arrow. +4. Hover the mouse over the model picker button and verify a tooltip appears + (e.g. `Pick Model`). +5. Click the model picker button to open the dropdown popup. +6. Verify the dropdown lists models grouped under labelled headers (e.g. + **Standard Models**, **Premium Models**), with each model showing a cost + multiplier on the right (e.g. `0x`, `1x`, `3x`). +7. Verify the currently selected model is marked with a **checkmark** (✓) + in the list. +8. Hover over a model item in the dropdown and verify a **detail card** + appears showing the model's **family** (e.g. `Family: claude-sonnet-4.5`), + **cost** (e.g. `Cost: 1x premium`), and **category tag** (e.g. + `Versatile`). +9. Click a model different from the currently selected one. +10. Verify the dropdown closes and the model picker button label updates to + show the newly selected model name. +11. Type a prompt (e.g. `hello`) in the chat input and click **Send**. +12. Wait for the Copilot turn to complete (the model-info-label footer + appears). +13. Verify the model info label at the bottom of the completed turn matches + the model selected in step 9. + +#### Expected Result + +- The model picker loads and shows the default model. +- The dropdown opens with grouped models and cost multipliers; the selected + model has a checkmark. +- Hovering a model shows a detail card with family, cost, and category. +- Selecting a different model updates the picker label immediately. +- The response is served by the newly selected model, confirmed by the turn + footer. + +#### 📸 Key Screenshots + +- [ ] **Model picker loaded** — action bar showing the resolved model name. +- [ ] **Dropdown open** — popup showing grouped model list with the active + model marked with a checkmark. +- [ ] **Model hover card** — detail card showing family, cost, and category + tag for a model item. +- [ ] **Model switched** — picker button showing the new model name. +- [ ] **Response with new model** — completed Copilot turn whose footer shows + the new model name. + +--- + +## 2. Chat mode interaction + +### TC-002: Switching chat mode updates the model picker + +**Type:** `Happy Path` +**Priority:** `P1` + +#### Preconditions + +- The Chat view is open. +- Both **Ask** and **Agent** modes are available in the mode picker. + +#### Steps + +1. Note the model currently shown in the model picker. +2. Switch the chat mode from **Agent** to **Ask** (or vice versa) using the + mode picker on the left side of the action bar. +3. Observe the model picker button — it should still show a valid model name + (which may differ from the previous one if the old model is not available + in the new mode). +4. Open the model picker dropdown and verify the model list has loaded (it may + contain different models than before). +5. Close the dropdown (click outside or press Escape). + +#### Expected Result + +- The model picker updates automatically when the chat mode changes. +- No crash, blank picker, or stale model name occurs. + +#### 📸 Key Screenshots + +- [ ] **After mode switch** — action bar showing the updated mode label and the + model picker with a valid model name. + +--- + +## 3. BYOK custom models and Manage Models shortcut + +### TC-003: Custom models appear in the picker and "Manage Models…" opens the preference page + +**Type:** `Happy Path` +**Priority:** `P1` + +#### Preconditions + +- The user is signed in with an account that has BYOK enabled. +- At least one custom model has been added and enabled via **Window → + Preferences → GitHub Copilot → Model Management**. + +#### Steps + +1. Open the model picker dropdown. +2. Verify a **Custom Models** group is present at the bottom of the list, + containing the configured custom model(s). +3. Select a custom model and verify the picker label updates. +4. Open the model picker dropdown again. +5. Click the **Manage Models…** item at the bottom of the dropdown. +6. Verify the **Preferences** dialog opens directly on the **Model Management** + page. +7. Close the Preferences dialog. + +#### Expected Result + +- Custom models are listed under a dedicated **Custom Models** group. +- Selecting a custom model works the same as selecting a built-in model. +- **Manage Models…** navigates directly to the BYOK preference page. + +#### 📸 Key Screenshots + +- [ ] **Custom Models group** — dropdown showing the Custom Models section with + the user's configured models. +- [ ] **Manage Models… action** — Preferences dialog opened to the Model + Management page after clicking the shortcut. + +--- + +## Notes on failure modes + +- Picker button stays blank / never shows a model name → the model list fetch + failed; check that the account is signed in and network access is available. +- Dropdown is empty → auth is valid but the server returned no models; try + signing out and back in to refresh the token. +- Selected model is not reflected in the completed turn's footer → the model + switch event did not propagate; re-select the model and resend. +- Custom Models group does not appear → BYOK is disabled by org policy, or no + custom model has been added; verify in Model Management preferences. +# Chat: Model Picker + +## Overview + +Tests the **model picker** dropdown in the Chat view action bar. The model +picker lets users choose which AI model powers their chat conversations. It +displays available models grouped by tier (Standard, Premium, Custom), shows +per-model details on hover, and includes a **Manage Models…** shortcut for +users with BYOK enabled. + +Entry points exercised: +- **Ctrl+Alt+I** (or status bar → **Open Chat**), then interact with the model + picker button in the action bar. + +--- + +## Prerequisites + +- Eclipse IDE with the GitHub Copilot for Eclipse plugin installed and + activated. +- A signed-in Copilot account. Without a valid account the model list cannot + be fetched and the picker will show no items. + +--- + +## 1. Model picker loads, opens, selects, and routes a message + +### TC-001: Open model picker, browse grouped models, switch model, and verify a response uses it + +**Type:** `Happy Path` +**Priority:** `P0` + +#### Preconditions + +- The Eclipse workbench is open. +- The user is signed in to Copilot. +- No previous Chat view is open. + +#### Steps + +1. Open the **Copilot Chat** view via the keyboard shortcut **Ctrl+Alt+I** + (Windows/Linux) or **Ctrl+Cmd+I** (macOS), or click the Copilot status bar + icon and select **Open Chat**. +2. Wait for the Chat view to fully load (the input area is editable, the model + picker shows a resolved model name instead of "Loading…"). +3. Verify the model picker button on the right side of the action bar displays + a model name (e.g. `Claude Sonnet 4.6`) with a dropdown arrow. +4. Hover the mouse over the model picker button and verify a tooltip appears + (e.g. `Pick Model`). +5. Click the model picker button to open the dropdown popup. +6. Verify the dropdown lists models grouped under labelled headers (e.g. + **Standard Models**, **Premium Models**), with each model showing a cost + multiplier on the right (e.g. `0x`, `1x`, `3x`). +7. Verify the currently selected model is marked with a **checkmark** (✓) + in the list. +8. Hover over a model item in the dropdown and verify a **detail card** + appears showing the model's **family** (e.g. `Family: claude-sonnet-4.5`), + **cost** (e.g. `Cost: 1x premium`), and **category tag** (e.g. + `Versatile`). +9. Click a model different from the currently selected one. +10. Verify the dropdown closes and the model picker button label updates to + show the newly selected model name. +11. Type a prompt (e.g. `hello`) in the chat input and click **Send**. +12. Wait for the Copilot turn to complete. +13. Verify the model info label at the bottom of the completed turn matches + the model selected in step 9. + +#### Expected Result + +- The model picker loads and shows the default model. +- The dropdown opens with grouped models and cost multipliers; the selected + model has a checkmark. +- Hovering a model shows a detail card with family, cost, and category. +- Selecting a different model updates the picker label immediately. +- The response is served by the newly selected model, confirmed by the turn + footer. + +#### 📸 Key Screenshots + +- [ ] **Model picker loaded** — action bar showing the resolved model name. +- [ ] **Dropdown open** — popup showing grouped model list with the active + model highlighted. +- [ ] **Model hover card** — detail card showing family, cost, and category + tag for a model item. +- [ ] **Model switched** — picker button showing the new model name. +- [ ] **Response with new model** — completed Copilot turn whose footer shows + the new model name. + +--- + +## 2. Chat mode interaction + +### TC-002: Switching chat mode updates the model picker + +**Type:** `Happy Path` +**Priority:** `P1` + +#### Preconditions + +- The Chat view is open. +- Both **Ask** and **Agent** modes are available in the mode picker. + +#### Steps + +1. Note the model currently shown in the model picker. +2. Switch the chat mode from **Agent** to **Ask** (or vice versa) using the + mode picker on the left side of the action bar. +3. Observe the model picker button — it should still show a valid model name + (which may differ from the previous one if the old model is not available + in the new mode). +4. Open the model picker dropdown and verify the model list has loaded (it may + contain different models than before). +5. Close the dropdown (click outside or press Escape). + +#### Expected Result + +- The model picker updates automatically when the chat mode changes. +- No crash, blank picker, or stale model name occurs. + +#### 📸 Key Screenshots + +- [ ] **After mode switch** — action bar showing the updated mode label and the + model picker with a valid model name. + +--- + +## 3. BYOK custom models and Manage Models shortcut + +### TC-003: Custom models appear in the picker and "Manage Models…" opens the preference page + +**Type:** `Happy Path` +**Priority:** `P1` + +#### Preconditions + +- The user is signed in with an account that has + BYOK enabled. +- At least one custom model has been added and enabled via **Window → Preferences → GitHub + Copilot → Model Management**. + +#### Steps + +1. Open the model picker dropdown. +2. Verify a **Custom Models** group is present at the bottom of the list, + containing the configured custom model(s). +3. Select a custom model and verify the picker label updates. +4. Open the model picker dropdown again. +5. Click the **Manage Models…** item at the bottom of the dropdown. +6. Verify the **Preferences** dialog opens directly on the **Model Management** + page. +7. Close the Preferences dialog. + +#### Expected Result + +- Custom models are listed under a dedicated **Custom Models** group. +- Selecting a custom model works the same as selecting a built-in model. +- **Manage Models…** navigates directly to the BYOK preference page. + +#### 📸 Key Screenshots + +- [ ] **Custom Models group** — dropdown showing the Custom Models section with + the user's configured models. +- [ ] **Manage Models… action** — Preferences dialog opened to the Model + Management page after clicking the shortcut. + +--- + +## Notes on failure modes + +- Picker button stays blank / never shows a model name → the model list fetch + failed; check that the account is signed in and network access is available. +- Dropdown is empty → auth is valid but the server returned no models; try + signing out and back in to refresh the token. +- Selected model is not reflected in the completed turn's footer → the model + switch event did not propagate; re-select the model and resend. +- Custom Models group does not appear → BYOK is disabled by org policy, or no + custom model has been added; verify in Model Management preferences. diff --git a/com.microsoft.copilot.eclipse.swtbot.test/test-plans/nes/nes.md b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/nes/nes.md new file mode 100644 index 00000000..dc4bc348 --- /dev/null +++ b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/nes/nes.md @@ -0,0 +1,311 @@ +# Next Edit Suggestions (NES) + +## Overview + +Tests the **Next Edit Suggestions** feature, which proactively shows AI-powered +code changes in the editor as the user types. When Copilot detects that a +recent edit implies a follow-up change elsewhere in the file, it highlights the +affected range with a background color, shows a diff popup with the proposed +change in green ghost text, and renders a Copilot icon in the editor gutter. + +Users can: +- **Tab** to accept the suggestion (or jump to it if off-screen). +- **Esc** to dismiss the suggestion. +- **Click the gutter icon** to open an Accept / Reject action menu. +- **Hover the gutter icon** to see a tooltip with keyboard shortcuts. + +--- + +## Prerequisites + +- Eclipse IDE with the GitHub Copilot for Eclipse plugin installed and + activated. +- A signed-in Copilot account. +- **Enable Next Edit Suggestions** is checked in **Window → Preferences → + GitHub Copilot → Completions** (this is the default). +- A Java project is open in the workspace with at least one Java source file. + +--- + +## 1. Trigger a suggestion, inspect it, and accept via Tab + +### TC-001: NES appears after related edits, shows diff and gutter icon, and is accepted with Tab + +**Type:** `Happy Path` +**Priority:** `P0` + +#### Preconditions + +- A Java file with multiple references to the same variable or method is open. +- NES is enabled in Preferences. + +#### Steps + +1. In the editor, make a series of related edits (e.g. rename a variable on + one line, then move the cursor to another line where the same variable is + used — or add a parameter to a method declaration, then navigate to a call + site of that method). +2. After each edit, pause briefly (~1–3 seconds) and observe the editor. +3. Once a NES suggestion appears, verify: + - A **background highlight** is shown on the text range that would be + changed. + - A **diff popup** appears next to the highlighted range showing the + proposed replacement in **green ghost text**. + - A **Copilot icon** appears in the editor gutter on the suggestion line. +4. Hover the mouse over the Copilot gutter icon and verify a tooltip appears + showing keyboard shortcuts: **Tab: Accept suggestion** and + **Esc: Dismiss suggestion**. +5. Press **Tab** on the keyboard. +6. Verify the suggestion is applied: the highlighted text is replaced with the + proposed change, the diff popup disappears, the gutter icon disappears, and + the file is modified (unsaved asterisk in the tab title). + +#### Expected Result + +- NES suggestion appears with highlight, diff popup, and gutter icon. +- Gutter icon tooltip shows Tab / Esc hints. +- Pressing Tab applies the change and clears all NES decorations. + +#### 📸 Key Screenshots + +- [ ] **NES suggestion visible** — editor showing the highlighted range, the + diff popup with green ghost text, and the Copilot gutter icon. +- [ ] **Gutter icon tooltip** — tooltip showing "Tab: Accept / Esc: Dismiss". +- [ ] **After Tab accept** — editor showing the applied change with no + remaining NES decorations. + +--- + +## 2. Reject via Esc and use the gutter icon action menu + +### TC-002: Dismiss a suggestion with Esc, trigger another, and use gutter icon Accept / Reject + +**Type:** `Happy Path` +**Priority:** `P0` + +#### Preconditions + +- A Java file is open in the editor. +- NES is enabled. + +#### Steps + +1. Make a series of related edits to trigger a NES suggestion (same approach + as TC-001 steps 1–2). +2. Once the suggestion appears, press **Esc**. +3. Verify the diff popup, background highlight, and gutter icon all disappear. +4. Verify the file content is unchanged — no text was inserted or deleted. +5. Make another series of related edits to trigger a new suggestion. +6. Once the new suggestion appears, **click the Copilot icon** in the gutter. +7. Verify an action menu appears with two options: **Accept** and **Reject**. +8. Click **Reject** in the action menu. +9. Verify the suggestion is dismissed (same as pressing Esc: all decorations + disappear, file unchanged). +10. Trigger yet another suggestion. +11. Click the gutter icon again and this time click **Accept**. +12. Verify the suggestion is applied (same result as pressing Tab in TC-001). + +#### Expected Result + +- Esc dismisses without modifying the file. +- Gutter icon click opens an Accept / Reject menu. +- Reject via menu dismisses the suggestion. +- Accept via menu applies the suggestion. + +#### 📸 Key Screenshots + +- [ ] **After Esc dismiss** — editor with no NES decorations, file unchanged. +- [ ] **Gutter icon menu** — the Accept / Reject action menu next to the + gutter icon. +- [ ] **After menu Accept** — editor showing the applied change. + +--- + +## 3. Out-of-viewport navigation + +### TC-003: Tab jumps to an off-screen suggestion, then a second Tab accepts it + +**Type:** `Happy Path` +**Priority:** `P1` + +#### Preconditions + +- A long Java file (200+ lines) is open. +- NES is enabled. + +#### Steps + +1. Make a series of related edits to trigger a NES suggestion (same approach + as TC-001). +2. Once the suggestion is visible (highlight, diff popup, gutter icon), + **scroll away** from the suggestion line so it is no longer in the viewport. +3. Verify a **notification bar** (pill) appears at the bottom of the editor + with a message like **"Press Tab to jump to Next Edit Suggestion"**. +4. Press **Tab**. +5. Verify the editor scrolls back to reveal the suggestion line, the diff + popup and gutter icon become visible again, and the notification bar + disappears. +6. Verify the suggestion is **not** accepted yet — the highlight and diff + popup are still showing. +7. Press **Tab** again. +8. Verify the suggestion is now accepted and all NES decorations disappear. + +#### Expected Result + +- First Tab: jumps to the suggestion without accepting it. +- Second Tab: accepts the suggestion. + +#### 📸 Key Screenshots + +- [ ] **Bottom bar visible** — notification pill at the bottom of the editor. +- [ ] **After first Tab** — editor scrolled to show the suggestion with diff + popup and gutter icon. +- [ ] **After second Tab** — suggestion accepted, all decorations cleared. + +--- + +## 4. Suggestion auto-dismiss on further edits + +### TC-004: Editing the file while a suggestion is active causes it to disappear + +**Type:** `Happy Path` +**Priority:** `P1` + +#### Preconditions + +- A Java file is open in the editor. +- NES is enabled. + +#### Steps + +1. Trigger a NES suggestion via a series of related edits (same approach as + TC-001). +2. Once the suggestion is visible (highlight, diff popup, gutter icon), do + **not** accept or reject it. +3. Instead, type additional text anywhere in the file (e.g. add a comment on a + different line, or directly edit the text in the suggested range). +4. Observe whether the NES suggestion disappears immediately after the edit. + +#### Expected Result + +- The NES suggestion (highlight, diff popup, gutter icon) disappears as soon + as the file content is modified by the user. +- The newly typed text is applied normally — no conflict with the old + suggestion. + +#### 📸 Key Screenshots + +- [ ] **Suggestion visible before edit** — NES decorations present. +- [ ] **After typing** — all NES decorations gone, user's new edit applied. + +--- + +## 5. Enable / disable toggle + +### TC-005: Disabling and re-enabling NES in Preferences + +**Type:** `Happy Path` +**Priority:** `P1` + +#### Preconditions + +- NES is currently enabled. +- A Java file is open. + +#### Steps + +1. Open **Window → Preferences → GitHub Copilot → Completions**. +2. Uncheck **Enable Next Edit Suggestions**. +3. Click **Apply and Close**. +4. Make a series of related edits in the Java file (same approach as TC-001). +5. Verify no NES suggestion appears (no highlight, no diff popup, no gutter + icon). +6. Open **Window → Preferences → GitHub Copilot → Completions** again. +7. Check **Enable Next Edit Suggestions**. +8. Click **Apply and Close**. +9. Make a series of related edits in the Java file again. +10. Verify a NES suggestion appears. + +#### Expected Result + +- Disabling: no NES decorations appear after edits. +- Re-enabling: NES suggestions resume without restarting Eclipse. +- Inline tab-completions are unaffected by either toggle. + +#### 📸 Key Screenshots + +- [ ] **NES disabled** — Preferences page with the checkbox unchecked. +- [ ] **No suggestion after edit** — editor with no NES decorations after + editing with NES disabled. +- [ ] **NES re-enabled** — suggestion appears again after re-enabling. + +--- + +## 6. No interference with other editor decorations + +### TC-006: No red rectangular border from Copilot appears on SonarQube issue locations + +**Type:** `Edge Case` +**Priority:** `P1` + +#### Preconditions + +- Both **GitHub Copilot for Eclipse** and **SonarQube for Eclipse** are + installed. +- NES is enabled. +- A Java project is open with a file that contains a SonarQube issue. For + example, a file like: + ```java + import java.util.List; + + public class TestSonar { + boolean isEmpty(List l) { + return l == null || l.size() == 0; + } + } + ``` + where `l.size() == 0` triggers a SonarQube rule. +- SonarQube analysis has completed and its markers are visible in the editor. + +#### Steps + +1. Open the Java file that has SonarQube issue markers. +2. Do **not** make any edits — just observe the editor. +3. Check whether a **red rectangular border** (BOX style) with the label + `"Text to be Deleted"` appears around the SonarQube issue location. +4. Hover over the SonarQube-annotated area and verify the tooltip shows + SonarQube's own message — not `"Text to be Deleted"`. +5. If a SonarQube quick fix is available, apply it. +6. Verify the red border does not persist after the quick fix is applied. +7. Open a different Java file that has compiler warnings but no SonarQube + issues, and verify no Copilot NES decorations appear on unedited code. + +#### Expected Result + +- No red rectangular border (`"Text to be Deleted"` BOX annotation) from + Copilot appears on SonarQube issue locations. +- SonarQube markers display normally with their own styling. +- Hovering SonarQube markers shows SonarQube's message, not Copilot's. +- After applying a SonarQube quick fix, no Copilot annotation persists. +- On unedited files without SonarQube, no NES decorations appear either. + +#### 📸 Key Screenshots + +- [ ] **SonarQube file — no NES** — editor showing SonarQube markers with no + Copilot decorations overlapping. +- [ ] **SonarQube hover** — tooltip showing the SonarQube message, not a + Copilot annotation. + +--- + +## Notes on failure modes + +- No suggestion appears after editing → NES may be disabled in Preferences, or + the Copilot account is not signed in; verify both and retry. +- Red rectangular border appears on SonarQube markers → the NES annotation + type is still mapped to the root `textmarker`; check that `plugin.xml` does + not declare `markerType="org.eclipse.core.resources.textmarker"` on the NES + annotation types. +- Accepting a suggestion has no effect → the suggestion may have become stale + (the underlying code changed again before Tab was pressed); make a fresh + edit to trigger a new suggestion and accept immediately. From f429d484d8b8dc08b12e78c2cb4f7bf10611419c Mon Sep 17 00:00:00 2001 From: xinyi-gong Date: Fri, 8 May 2026 14:38:09 +0800 Subject: [PATCH 2/2] resolve comments --- .../test-plans/model-picker/model-picker.md | 189 +----------------- 1 file changed, 6 insertions(+), 183 deletions(-) diff --git a/com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md index ea39eab7..1d18ab35 100644 --- a/com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md +++ b/com.microsoft.copilot.eclipse.swtbot.test/test-plans/model-picker/model-picker.md @@ -5,7 +5,7 @@ Tests the **model picker** dropdown in the Chat view action bar. The model picker lets users choose which AI model powers their chat conversations. It displays available models grouped by tier (Standard, Premium, Custom), shows -per-model details on hover, and includes a **Manage Models…** shortcut for +per-model details on hover, and includes a **Manage Models...** shortcut for users with BYOK enabled. Entry points exercised: @@ -42,7 +42,7 @@ Entry points exercised: (Windows/Linux) or **Ctrl+Cmd+I** (macOS), or click the Copilot status bar icon and select **Open Chat**. 2. Wait for the Chat view to fully load (the input area is editable, the model - picker shows a resolved model name instead of "Loading…"). + picker shows a resolved model name instead of "Loading..."). 3. Verify the model picker button on the right side of the action bar displays a model name (e.g. `Claude Sonnet 4.6`) with a dropdown arrow. 4. Hover the mouse over the model picker button and verify a tooltip appears @@ -127,7 +127,7 @@ Entry points exercised: ## 3. BYOK custom models and Manage Models shortcut -### TC-003: Custom models appear in the picker and "Manage Models…" opens the preference page +### TC-003: Custom models appear in the picker and "Manage Models..." opens the preference page **Type:** `Happy Path` **Priority:** `P1` @@ -145,7 +145,7 @@ Entry points exercised: containing the configured custom model(s). 3. Select a custom model and verify the picker label updates. 4. Open the model picker dropdown again. -5. Click the **Manage Models…** item at the bottom of the dropdown. +5. Click the **Manage Models...** item at the bottom of the dropdown. 6. Verify the **Preferences** dialog opens directly on the **Model Management** page. 7. Close the Preferences dialog. @@ -154,190 +154,13 @@ Entry points exercised: - Custom models are listed under a dedicated **Custom Models** group. - Selecting a custom model works the same as selecting a built-in model. -- **Manage Models…** navigates directly to the BYOK preference page. +- **Manage Models...** navigates directly to the BYOK preference page. #### 📸 Key Screenshots - [ ] **Custom Models group** — dropdown showing the Custom Models section with the user's configured models. -- [ ] **Manage Models… action** — Preferences dialog opened to the Model - Management page after clicking the shortcut. - ---- - -## Notes on failure modes - -- Picker button stays blank / never shows a model name → the model list fetch - failed; check that the account is signed in and network access is available. -- Dropdown is empty → auth is valid but the server returned no models; try - signing out and back in to refresh the token. -- Selected model is not reflected in the completed turn's footer → the model - switch event did not propagate; re-select the model and resend. -- Custom Models group does not appear → BYOK is disabled by org policy, or no - custom model has been added; verify in Model Management preferences. -# Chat: Model Picker - -## Overview - -Tests the **model picker** dropdown in the Chat view action bar. The model -picker lets users choose which AI model powers their chat conversations. It -displays available models grouped by tier (Standard, Premium, Custom), shows -per-model details on hover, and includes a **Manage Models…** shortcut for -users with BYOK enabled. - -Entry points exercised: -- **Ctrl+Alt+I** (or status bar → **Open Chat**), then interact with the model - picker button in the action bar. - ---- - -## Prerequisites - -- Eclipse IDE with the GitHub Copilot for Eclipse plugin installed and - activated. -- A signed-in Copilot account. Without a valid account the model list cannot - be fetched and the picker will show no items. - ---- - -## 1. Model picker loads, opens, selects, and routes a message - -### TC-001: Open model picker, browse grouped models, switch model, and verify a response uses it - -**Type:** `Happy Path` -**Priority:** `P0` - -#### Preconditions - -- The Eclipse workbench is open. -- The user is signed in to Copilot. -- No previous Chat view is open. - -#### Steps - -1. Open the **Copilot Chat** view via the keyboard shortcut **Ctrl+Alt+I** - (Windows/Linux) or **Ctrl+Cmd+I** (macOS), or click the Copilot status bar - icon and select **Open Chat**. -2. Wait for the Chat view to fully load (the input area is editable, the model - picker shows a resolved model name instead of "Loading…"). -3. Verify the model picker button on the right side of the action bar displays - a model name (e.g. `Claude Sonnet 4.6`) with a dropdown arrow. -4. Hover the mouse over the model picker button and verify a tooltip appears - (e.g. `Pick Model`). -5. Click the model picker button to open the dropdown popup. -6. Verify the dropdown lists models grouped under labelled headers (e.g. - **Standard Models**, **Premium Models**), with each model showing a cost - multiplier on the right (e.g. `0x`, `1x`, `3x`). -7. Verify the currently selected model is marked with a **checkmark** (✓) - in the list. -8. Hover over a model item in the dropdown and verify a **detail card** - appears showing the model's **family** (e.g. `Family: claude-sonnet-4.5`), - **cost** (e.g. `Cost: 1x premium`), and **category tag** (e.g. - `Versatile`). -9. Click a model different from the currently selected one. -10. Verify the dropdown closes and the model picker button label updates to - show the newly selected model name. -11. Type a prompt (e.g. `hello`) in the chat input and click **Send**. -12. Wait for the Copilot turn to complete. -13. Verify the model info label at the bottom of the completed turn matches - the model selected in step 9. - -#### Expected Result - -- The model picker loads and shows the default model. -- The dropdown opens with grouped models and cost multipliers; the selected - model has a checkmark. -- Hovering a model shows a detail card with family, cost, and category. -- Selecting a different model updates the picker label immediately. -- The response is served by the newly selected model, confirmed by the turn - footer. - -#### 📸 Key Screenshots - -- [ ] **Model picker loaded** — action bar showing the resolved model name. -- [ ] **Dropdown open** — popup showing grouped model list with the active - model highlighted. -- [ ] **Model hover card** — detail card showing family, cost, and category - tag for a model item. -- [ ] **Model switched** — picker button showing the new model name. -- [ ] **Response with new model** — completed Copilot turn whose footer shows - the new model name. - ---- - -## 2. Chat mode interaction - -### TC-002: Switching chat mode updates the model picker - -**Type:** `Happy Path` -**Priority:** `P1` - -#### Preconditions - -- The Chat view is open. -- Both **Ask** and **Agent** modes are available in the mode picker. - -#### Steps - -1. Note the model currently shown in the model picker. -2. Switch the chat mode from **Agent** to **Ask** (or vice versa) using the - mode picker on the left side of the action bar. -3. Observe the model picker button — it should still show a valid model name - (which may differ from the previous one if the old model is not available - in the new mode). -4. Open the model picker dropdown and verify the model list has loaded (it may - contain different models than before). -5. Close the dropdown (click outside or press Escape). - -#### Expected Result - -- The model picker updates automatically when the chat mode changes. -- No crash, blank picker, or stale model name occurs. - -#### 📸 Key Screenshots - -- [ ] **After mode switch** — action bar showing the updated mode label and the - model picker with a valid model name. - ---- - -## 3. BYOK custom models and Manage Models shortcut - -### TC-003: Custom models appear in the picker and "Manage Models…" opens the preference page - -**Type:** `Happy Path` -**Priority:** `P1` - -#### Preconditions - -- The user is signed in with an account that has - BYOK enabled. -- At least one custom model has been added and enabled via **Window → Preferences → GitHub - Copilot → Model Management**. - -#### Steps - -1. Open the model picker dropdown. -2. Verify a **Custom Models** group is present at the bottom of the list, - containing the configured custom model(s). -3. Select a custom model and verify the picker label updates. -4. Open the model picker dropdown again. -5. Click the **Manage Models…** item at the bottom of the dropdown. -6. Verify the **Preferences** dialog opens directly on the **Model Management** - page. -7. Close the Preferences dialog. - -#### Expected Result - -- Custom models are listed under a dedicated **Custom Models** group. -- Selecting a custom model works the same as selecting a built-in model. -- **Manage Models…** navigates directly to the BYOK preference page. - -#### 📸 Key Screenshots - -- [ ] **Custom Models group** — dropdown showing the Custom Models section with - the user's configured models. -- [ ] **Manage Models… action** — Preferences dialog opened to the Model +- [ ] **Manage Models... action** — Preferences dialog opened to the Model Management page after clicking the shortcut. ---