Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update TESTING.md #1283

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Update TESTING.md #1283

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e69d616
Update TESTING.md
aramaraju Apr 4, 2024
917791e
Created two cases for Enterprise testing, one for Cody Gateway instan…
aramaraju Apr 12, 2024
4f9adb9
Update TESTING.md
aramaraju Apr 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 35 additions & 6 deletions TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
- [ ] [Automatic repository recognition](#automatic-repository-recognition)
- [ ] [Persistent custom repository](#persistent-custom-repository)
- [ ] [Colour scheme changes](#colour-scheme-changes)
- [ ] [Claude 3 Enterprise Testing](#claude-3-enterprise-testing)
- Context
- [ ] [PLG / SYMF / Local Keyword search](#local-keyword-search)
- Model dropdown
Expand Down Expand Up @@ -476,20 +477,48 @@ To open the context menu:
2. Create new chat
3. User doesn't see model dropdown

### Enterprise account
### Cody Enterprise - Cody Gateway Instance

#### Chat

1. Login to Cody Enterprise account
2. Create new chat
3. User should see the default model in the dropdown but is unable to change it
1. Login to the Cody Enterprise Gateway Instance
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where's the credentials for this, if I wanted to perform these tests myself? Ditto below for "Cody Enterprise Cody Gateway Instance". Is that a different instance?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are they getting the credentials today? I thought they were already shared outside of this doc. @kalanchan consolidated them into 1Password

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"where can I get the credentials" kind of information should live in the handbook, but we can also add a link to 1password here. For the sake of completeness, let's just add it here 😄

2. Create a new chat
3. User should not be able to change the LLM model on the chat window
4. Ask Cody a question in the new chat window, Cody should respond.

#### Commands

1. Login to Cody Enterprise account
1. Login to the Cody Enterprise Cody Gateway Instance
2. Go to commands panel
3. Trigger command
4. Command should be executed with default model
4. Command should be executed

#### Completions

1. Login to the Cody Enterprise Cody Gateway Instance
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go).
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go). A completion should appear.

@aramaraju

  • Document the keyboard shortcut to press or the name of the command so the testers can look up the keyboard shortcut.
  • This test should specify having a particular repo cloned and open, and mention specific files to open. It will make testing easier and make reproducing any bugs much easier.

@pkukielka does the tester's setup include other autocomplete providers? If so we need to ensure the completions are coming from Cody specifically somehow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opt \ is the manual trigger hotkey for autocomplete.

The QA testers have experience with this, I don't think we need to assume they have another completion provider. The Cody logo also "spins/loads" when it triggers a completion, so theres indication from the UI that it's coming from Cody



### Cody Enterprise - AWS Bedrock Instance

#### Chat

1. Login to the Cody Enterprise AWS Bedrock Instance
2. Create a new chat
3. User should not be able to change the LLM model on the chat window
4. Ask Cody a question in the new chat window.

#### Commands

1. Login to the Cody Enterprise AWS Bedrock Instance
2. Go to commands panel
3. Trigger command
4. Command should be executed

#### Completions

1. Login to the Cody Enterprise AWS Bedrock Instance
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go).
Comment on lines +502 to +521
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be better if you removed the duplication and said something like: There are these kinds of instances (list them) and then had one set of instructions, like "For each kind of instance: ..."

Copy link
Contributor Author

@aramaraju aramaraju Apr 21, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to note is that we are evaluating the LLM dropdown for Enterprise customers using Cody Gateway, which might not work as elegantly if combine the instance work. I agree combining them reads better, but I'm not sure if the team is going through a check list and having it split up by instances is easier in that case. If we do consolidate, shouldn't this also apply to the entire document? Do the following with these instances: Free, Pro, Enterprise 1, and 2? Will require a rewrite of the whole thing and I don't know why we didn't do that in the first place.


## Other

Expand Down
Loading