make slash command required in command configs #841

taras-yemets · 2023-08-28T15:39:34Z

Next iteration on #631
Follow-up on #798
Design

Changes commands format in default-prompts.json and .vscode/cody.json:
- slash command now is required and acts as a key in commands map, slashCommand field on a command is removed
- command name previously used as a key is now used as a command description.
Updates the custom command creation process making the slash command input required and going first before the description and prompt inputs.
If spotted user or workspace Cody config with commands in "old" format, transforms them to the new one, saves updated config file and informs user about it.

Screen.Recording.2023-09-01.at.16.28.04.mov

Test plan

CI passes.

…sh-command-required

taras-yemets · 2023-08-31T09:12:46Z

vscode/src/custom-prompts/CustomPromptsStore.ts

            for (const key in prompts) {
                if (Object.prototype.hasOwnProperty.call(prompts, key)) {
                    const prompt = prompts[key]
-                    prompt.name = key
                    prompt.type = type
-                    // replace any '/' from the start
-                    const slashCommand = prompt.slashCommand?.replace(/^\//, '')
-                    if (slashCommand?.length) {
-                        prompt.slashCommand = `/${slashCommand}`
-                    }
+                    prompt.slashCommand = key.startsWith('/') ? key : '/' + key
                    this.myPromptsMap.set(key, prompt)
                }
            }


Open question: how to ensure these changes are backward compatible (users with the previous format of cody.json with command names as keys, users with recipes)?
cc: @abeatrix

A possible solution could be to apply new logic if the description field exists in a command. This would mean that slash command is used as a key.
If the description field is missing, store the command only if the slashCommand is defined.
However, this will lead to partial use of cody.json content (in "old" format commands only with slash commands will be stored). Maybe we should rewrite (make slash command a key) the cody.json content when we spot the old format? WDYT?

lib/shared/src/chat/prompts/index.ts

vscode/src/custom-prompts/menus/CustomCommandBuilderMenu.ts

abeatrix

Left comments in line regarding migrating current file into next format

abeatrix · 2023-08-31T23:21:11Z

vscode/src/custom-prompts/CustomPromptsStore.ts

-                    }
-                    this.myPromptsMap.set(key, prompt)
+                    // ensure there is only one leading forward slash
+                    prompt.slashCommand = key.replace(/^\/+/, '').replace(/^/, '/')


Do you think we should also update the save method to handle backward compatibility?

I think we should also extract the first words from all the current keys and turn them into the prompt.slashCommand option here to auto-migrate the current commands without the "slashCommand" settings for our users?

Maybe we can add a CTA to let them know how they can change the command keyword afterward. What do you think?

Thanks for pointing this out! Updated the PR. Now we transform "old" format commands to the new format, update the corresponding config files, and inform the user about the change.
This PR converts names to dash-case (not the first word in the command as it's not guaranteed to be unique) and uses them as slash commands.

…sh-command-required

taras-yemets · 2023-09-01T13:38:05Z

@toolmantim, except for the overall implementation, I'm looking for your feedback on:

the updated custom command creation flow;
"old" format configs transformation and notification shown after.

abeatrix · 2023-09-01T15:05:16Z

tried to run the branch but i'm getting:

Looks like issue with sentry?

taras-yemets · 2023-09-01T15:11:41Z

Looks like issue with sentry?

@abeatrix, it seems so. But it doesn't seem to be caused by te current PR changes.

toolmantim

Works great! I gave it a thorough test locally, and popped on some commits to update a bunch of the UI labels.

abeatrix

Works beautifully! Thanks @taras-yemets!

I have a PR opened that would be affected by the changes in this PR. Since my changes are closer to the main branch as it is using the old format, can you review my PR and see if we can merge mine before yours? Or would it cause other conflicts that would not matter which one gets merged first?

Here is the PR: #907

Part of https://github.com/sourcegraph/sourcegraph/issues/56182 Here's how to get this running: 1. Azure keys * https://portal.azure.com > Azure OpenAI > sourcegraph-test-oai > Keys and Endpoint * Copy the key and `export AZURE_API_KEY="<paste the key here>"` 1. `export SRC_ENDPOINT=https://sourcegraph.sourcegraph.com` 1. `export SRC_ACCESS_TOKEN="<s2 access token goes here>"` 1. Run like so: `pnpm run start --label="Beam search" --output="$HOME/test.results" --provider=azure` * single beam search test runs faster than total # test cases * output file actually allows you to drill into the LLM interaction details ## Test plan Manually verified: ### success scenario with Azure OpenAI and ``` ➜ e2e git:(cb/s/56182-azure) ✗ pnpm run start --label="Beam search" --output="$HOME/azure.results" --provider=azure > @sourcegraph/cody-e2e@0.0.1 start /Users/cbart/cody/e2e > pnpm run --silent build && node dist/e2e "--label=Beam search" "--output=/Users/cbart/azure.results" "--provider=azure" Run 1 of 1 Testing (1/1): Beam search (github.com/huggingface/transformers) Incorrect answers (LLM judge): 0/2 (0%) Incorrect or partial answers (LLM judge): 1/2 (50%) Missing facts: 5/9 (56%) Hallucinated entities: 9/9 (100%) Run 1 of 1 summary: Incorrect answers (LLM judge): 0/2 (0%) Incorrect or partial answers (LLM judge): 1/2 (50%) Missing facts: 5/9 (56%) Hallucinated entities: 9/9 (100%) ``` ### fast fail when not enough env vars provided ``` ➜ e2e git:(cb/s/56182-azure) unset AZURE_DEPLOYMENT_ID ➜ e2e git:(cb/s/56182-azure) ✗ pnpm run start --label="Beam search" --output="$HOME/azure.results" --provider=azure > @sourcegraph/cody-e2e@0.0.1 start /Users/cbart/cody/e2e > pnpm run --silent build && node dist/e2e "--label=Beam search" "--output=/Users/cbart/azure.results" "--provider=azure" Missing required environment variables: AZURE_DEPLOYMENT_ID 1. https://portal.azure.com > Azure OpenAI > sourcegraph-test-oai > Keys and Endpoint 2. Go to Model Deployments > Manage Deployments. Find the deployment name and: export AZURE_DEPLOYMENT_ID="<paste deployment name here>" ELIFECYCLE Command failed with exit code 1. ```

…sh-command-required

taras-yemets · 2023-09-03T20:42:53Z

I have a PR opened that would be affected by the changes in this PR. Since my changes are closer to the main branch as it is using the old format, can you review my PR and see if we can merge mine before yours? Or would it cause other conflicts that would not matter which one gets merged first?
Here is the PR: #907

@abeatrix There is an ongoing discussion about whether the referenced PR will be included in the upcoming release, so I'm this one to have a bit more time to test it on main. I'm happy to help with resolving merge conflicts!

taras-yemets added 6 commits August 28, 2023 18:38

update prompts config shape

7a6356d

make slash command required in custom commands

9954f83

Merge branch 'main' into taras-yemets/make-slash-command-required

d6ba66f

refactor

2af0cc8

remove comment

095510e

fix chat commands labels

5353ec7

taras-yemets mentioned this pull request Aug 30, 2023

rename /fix command to /edit #847

Merged

taras-yemets added 2 commits August 30, 2023 17:15

Merge remote-tracking branch 'origin/main' into taras-yemets/make-sla…

b042227

…sh-command-required

make CodyPrompt["slashCommand"] optional

bd90269

taras-yemets changed the title ~~make slash command reqired~~ make slash command required in command configs Aug 30, 2023

taras-yemets added 3 commits August 31, 2023 10:25

Merge remote-tracking branch 'origin/main' into taras-yemets/make-sla…

a889098

…sh-command-required

Merge remote-tracking branch 'origin/main' into taras-yemets/make-sla…

fb576d0

…sh-command-required

upd custom command creation

7382825

taras-yemets commented Aug 31, 2023

View reviewed changes

lib/shared/src/chat/prompts/index.ts Show resolved Hide resolved

ensure command map key starts with slash

b8b146a

abeatrix reviewed Aug 31, 2023

View reviewed changes

vscode/src/custom-prompts/menus/CustomCommandBuilderMenu.ts Outdated Show resolved Hide resolved

abeatrix reviewed Aug 31, 2023

View reviewed changes

taras-yemets added 5 commits September 1, 2023 11:12

fix copy

66f5b39

refactor and fix errors

6293202

upd

39fc249

do not include deprecated fields

47280bf

add comments

78a28ad

taras-yemets requested review from toolmantim, abeatrix and a team September 1, 2023 13:30

taras-yemets added 2 commits September 1, 2023 16:34

extract slash commands before saving to config file

69c8c05

Merge remote-tracking branch 'origin/main' into taras-yemets/make-sla…

6b01feb

…sh-command-required

taras-yemets marked this pull request as ready for review September 1, 2023 13:35

toolmantim added 5 commits September 1, 2023 14:07

Lowercase the slash command

ae89e5c

Tweak prompt wording

c2fcfd0

Format the JSON

0617987

Update custom command quickpick labels

d273f37

Sentence case description

040497f

toolmantim approved these changes Sep 1, 2023

View reviewed changes

abeatrix approved these changes Sep 3, 2023

View reviewed changes

toolmantim mentioned this pull request Sep 3, 2023

Update Cody VS Code README gifs #906

Merged

cbart and others added 3 commits September 3, 2023 22:19

Merge remote-tracking branch 'origin/main' into taras-yemets/make-sla…

965b47f

…sh-command-required

update changelog

6b9b213

taras-yemets merged commit 519145a into main Sep 3, 2023
6 of 7 checks passed

taras-yemets deleted the taras-yemets/make-slash-command-required branch September 3, 2023 20:36

abeatrix mentioned this pull request Sep 5, 2023

improve generate unit tests command output quality #907

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make slash command required in command configs #841

make slash command required in command configs #841

taras-yemets commented Aug 28, 2023 •

edited

Loading

taras-yemets Aug 31, 2023

taras-yemets Aug 31, 2023

abeatrix left a comment

abeatrix Aug 31, 2023

taras-yemets Sep 1, 2023

taras-yemets commented Sep 1, 2023

abeatrix commented Sep 1, 2023

taras-yemets commented Sep 1, 2023 •

edited

Loading

toolmantim left a comment

abeatrix left a comment

taras-yemets commented Sep 3, 2023

make slash command required in command configs #841

make slash command required in command configs #841

Conversation

taras-yemets commented Aug 28, 2023 • edited Loading

Test plan

taras-yemets Aug 31, 2023

Choose a reason for hiding this comment

taras-yemets Aug 31, 2023

Choose a reason for hiding this comment

abeatrix left a comment

Choose a reason for hiding this comment

abeatrix Aug 31, 2023

Choose a reason for hiding this comment

taras-yemets Sep 1, 2023

Choose a reason for hiding this comment

taras-yemets commented Sep 1, 2023

abeatrix commented Sep 1, 2023

taras-yemets commented Sep 1, 2023 • edited Loading

toolmantim left a comment

Choose a reason for hiding this comment

abeatrix left a comment

Choose a reason for hiding this comment

taras-yemets commented Sep 3, 2023

taras-yemets commented Aug 28, 2023 •

edited

Loading

taras-yemets commented Sep 1, 2023 •

edited

Loading