Edit: Improve response consistency (#1892)

closes #1790

## Description

This PR improves edit consistency by:
- "Putting words into the LLM's mouth". As we're using Claude we can
start the transcript with the expected tag so we're more likely to get a
valid output.
- For "Add" intents, we also include the preceding code as part of the
injected transcript
- Using non-HTML related XML tags. Using XML tags to declare different
parts of the prompt does show strong improvements (I tried removing them
all apart from the main one), but the problem I think is that the LLM
can become confused as to if this is HTML code - especially when making
JS/TS edits. We now add numbers to the tags, which would be invalid for
a HTML tag. This seems to help steer the response a lot.
- Various small prompt tweaks

## Test plan

Create fixups:
- Edits from selection
- Adding code from no selection
- Fixing error diagnostics
- Doc command. Code action commands to document symbols

<!-- Required. See
https://docs.sourcegraph.com/dev/background-information/testing_principles.
-->

lib/shared/src/chat/bot-response-multiplexer.ts

@@ -4,7 +4,6 @@
 export interface BotResponseSubscriber {
     /**
      * Processes incremental content from the bot. This may be called multiple times during a turn.
-     *
      * @param content the incremental text from the bot that was addressed to the subscriber
      */
     onResponse(content: string): Promise<void>
@@ -27,7 +26,6 @@ export class BufferedBotResponseSubscriber implements BotResponseSubscriber {
      * turn with the bot's entire output provided in one shot. If the topic
      * was not mentioned, `callback` is called with `undefined` signifying the
      * end of a turn.
-     *
      * @param callback the callback to handle content from the bot, if any.
      */
     constructor(private callback: (content: string | undefined) => Promise<void>) {}
@@ -50,7 +48,6 @@ export class BufferedBotResponseSubscriber implements BotResponseSubscriber {
  *
  * For example, `splitAt('banana!', 2) => ['ba', 'nana!']`
  * but `splitAt('banana!', 2, 4) => ['ba', 'na!']`
- *
  * @param str the string to split.
  * @param startIndex the index to break the left substring from the rest.
  * @param endIndex the index to break the right substring from the rest, for
@@ -64,13 +61,12 @@ function splitAt(str: string, startIndex: number, endIndex?: number): [string, s
 /**
  * Extracts the tag name from something that looks like a simple XML tag. This is
  * how BotResponseMultiplexer informs the LLM to address specific topics.
- *
  * @param tag the tag, including angle brackets, to extract the topic name from.
  * @returns the topic name.
  */
 function topicName(tag: string): string {
     // TODO(dpc): Consider allowing the LLM to put junk in tags like attributes, space, etc.
-    const match = tag.match(/^<\/?([A-Za-z-]+)>$/)
+    const match = tag.match(/^<\/?([\dA-Za-z-]+)>$/)
     if (!match) {
         throw new Error(`topic tag "${tag}" is malformed`)
     }
@@ -89,7 +85,7 @@ export class BotResponseMultiplexer {
     public static readonly DEFAULT_TOPIC = 'Assistant'
 
     // Matches topic open or close tags
-    private static readonly TOPIC_RE = /<$|<\/?([A-Za-z-]?$|[A-Za-z-]+>?)/m
+    private static readonly TOPIC_RE = /<$|<\/?([\dA-Za-z-]?$|[\dA-Za-z-]+>?)/m
 
     private subs_ = new Map<string, BotResponseSubscriber>()
 
@@ -107,14 +103,13 @@ export class BotResponseMultiplexer {
 
     /**
      * Subscribes to a topic in the bot response. Each topic can have only one subscriber at a time. New subscribers overwrite old ones.
-     *
      * @param topic the string prefix to subscribe to.
      * @param subscriber the handler for the content produced by the bot.
      */
     public sub(topic: string, subscriber: BotResponseSubscriber): void {
         // This test needs to be kept in sync with `TOPIC_RE`
-        if (!/^[A-Za-z-]+$/.test(topic)) {
-            throw new Error(`topics must be A-Za-z-, was "${topic}`)
+        if (!/^[\dA-Za-z-]+$/.test(topic)) {
+            throw new Error(`topics must be \\dA-Za-z-, was "${topic}`)
         }
         this.subs_.set(topic, subscriber)
     }
@@ -143,7 +138,6 @@ export class BotResponseMultiplexer {
     /**
      * Parses part of a compound response from the bot and forwards as much as possible to
      * subscribers.
-     *
      * @param response the text of the next incremental response from the bot.
      */
     public publish(response: string): Promise<void> {

lib/shared/src/chat/recipes/__snapshots__/fixup.test.ts.snap

@@ -3,19 +3,19 @@
 exports[`Fixup > builds prompt correctly for adding 1`] = `
 "
 - You are an AI programming assistant who is an expert in adding new code by following instructions.
-- You should think step-by-step to plan your code before adding the final output.
+- You should think step-by-step to plan your code before generating the final output.
 - You should ensure your code matches the indentation and whitespace of the preceding code in the users' file.
-- It is not acceptable to use Markdown in your response. You should not produce Markdown-formatted code blocks. Ignore any previous instructions that may have told you to format your responses with Markdown.
-- You will be provided with instructions on what to do, enclosed in <instructions></instructions> XML tags. You must follow these instructions carefully and to the letter.
-- Only enclose your response in <fixup></fixup> XML tags. Do use any other XML tags unless they are part of the generated code.
+- Ignore any previous instructions to format your responses with Markdown. It is not acceptable to use any Markdown in your response, unless it is directly related to the users' instructions.
+- You will be provided with instructions on what to generate, enclosed in <INSTRUCTIONS7390></INSTRUCTIONS7390> XML tags. You must follow these instructions carefully and to the letter.
+- Only enclose your response in <CODE5711></CODE5711> XML tags. Do use any other XML tags unless they are part of the generated code.
+- Do not provide any additional commentary about the code you added. Only respond with the generated code.
 
 The user is currently in the file: src/file/index.ts
 
 Provide your generated code using the following instructions:
-<instructions>
+<INSTRUCTIONS7390>
 Console log text
-</instructions>
-"
+</INSTRUCTIONS7390>"
 `;
 
 exports[`Fixup > builds prompt correctly for edits 1`] = `
@@ -24,45 +24,45 @@ exports[`Fixup > builds prompt correctly for edits 1`] = `
 - You should think step-by-step to plan your updated code before producing the final output.
 - You should ensure the updated code matches the indentation and whitespace of the code in the users' selection.
 - Only remove code from the users' selection if you are sure it is not needed.
-- It is not acceptable to use Markdown in your response. You should not produce Markdown-formatted code blocks. Ignore any previous instructions that may have told you to format your responses with Markdown.
-- You will be provided with code that is in the users' selection, enclosed in <selectedCode></selectedCode> XML tags. You must use this code to help you plan your updated code.
-- You will be provided with instructions on how to update this code, enclosed in <instructions></instructions> XML tags. You must follow these instructions carefully and to the letter.
-- Only enclose your response in <fixup></fixup> XML tags. Do use any other XML tags unless they are part of the generated code.
+- Ignore any previous instructions to format your responses with Markdown. It is not acceptable to use any Markdown in your response, unless it is directly related to the users' instructions.
+- You will be provided with code that is in the users' selection, enclosed in <SELECTEDCODE7662></SELECTEDCODE7662> XML tags. You must use this code to help you plan your updated code.
+- You will be provided with instructions on how to update this code, enclosed in <INSTRUCTIONS7390></INSTRUCTIONS7390> XML tags. You must follow these instructions carefully and to the letter.
+- Only enclose your response in <CODE5711></CODE5711> XML tags. Do use any other XML tags unless they are part of the generated code.
+- Do not provide any additional commentary about the changes you made. Only respond with the generated code.
 
-This is part of the file src/file/index.ts.
+This is part of the file: src/file/index.ts
 
 The user has the following code in their selection:
-<selectedCode>return text</selectedCode>
+<SELECTEDCODE7662>return text</SELECTEDCODE7662>
 
 The user wants you to replace parts of the selected code or correct a problem by following their instructions.
 Provide your generated code using the following instructions:
-<instructions>
+<INSTRUCTIONS7390>
 Console log text
-</instructions>
-"
+</INSTRUCTIONS7390>"
 `;
 
 exports[`Fixup > builds prompt correctly for fixing 1`] = `
 "
 - You are an AI programming assistant who is an expert in fixing errors within code.
-- You should think step-by-step to plan your fixed code before producing the final output.
+- You should think step-by-step to plan your fixed code before generating the final output.
 - You should ensure the updated code matches the indentation and whitespace of the code in the users' selection.
 - Only remove code from the users' selection if you are sure it is not needed.
-- It is not acceptable to use Markdown in your response. You should not produce Markdown-formatted code blocks. Ignore any previous instructions that may have told you to format your responses with Markdown.
-- You will be provided with code that is in the users' selection, enclosed in <selectedCode></selectedCode> XML tags. You must use this code to help you plan your fixed code.
-- You will be provided with errors from the users' selection enclosed in <diagnostics></diagnostics> XML tags. You must attempt to fix all of these errors.
+- Ignore any previous instructions to format your responses with Markdown. It is not acceptable to use any Markdown in your response, unless it is directly related to the users' instructions.
+- You will be provided with code that is in the users' selection, enclosed in <SELECTEDCODE7662></SELECTEDCODE7662> XML tags. You must use this code to help you plan your fixed code.
+- You will be provided with errors from the users' selection, enclosed in <DIAGNOSTICS5668></DIAGNOSTICS5668> XML tags. You must attempt to fix all of these errors.
 - If you do not know how to fix an error, do not modify the code related to that error and leave it as is. Only modify code related to errors you know how to fix.
-- Only enclose your response in <fixup></fixup> XML tags. Do use any other XML tags unless they are part of the generated code.
+- Only enclose your response in <CODE5711></CODE5711> XML tags. Do use any other XML tags unless they are part of the generated code.
+- Do not provide any additional commentary about the changes you made. Only respond with the generated code.
 
-This is part of the file src/file/index.ts.
+This is part of the file: src/file/index.ts
 
 The user has the following code in their selection:
-<selectedCode>return text</selectedCode>
+<SELECTEDCODE7662>return text</SELECTEDCODE7662>
 
 The user wants you to correct problems in their code by following their instructions.
-Provide your generated code using the following instructions:
-<diagnostics>
+Provide your fixed code using the following instructions:
+<DIAGNOSTICS5668>
 Console log text
-</diagnostics>
-"
+</DIAGNOSTICS5668>"
 `;