systemMessageComposition experimental configuration for more control over constructed system message

[ferenci84]

Description

Continuation of #4506

Replacement of #3787

On top of PR 4506 it adds experimental configuration systemMessageComposition. Changed documentation explains how it works.

Legacy logic remains working if user do not change the configuration.

Checklist

• The relevant docs, if any, have been updated or created
• The relevant tests, if any, have been updated or created

Screenshots

[ For visual changes, include screenshots. ]

Testing instructions

Experiment with config:

  "experimental": {
    "systemMessageComposition": "prepend"
  }

In the model description, you can also set:

     "promptTemplates": {
        "codeBlockInstructions": "My own instructions for code blocks",
        "toolUseRules": "My tool use rules"
      },

[netlify]

✅ Deploy Preview for continuedev canceled.

Name	Link
🔨 Latest commit	89329af
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/67c976d6cc11b3000807d045

[sammyvoncheese]

very cool change, I have been looking at doing something similar to customize/override all of the hidden system messages sent with each request.

[sestinj]

@ferenci84 I'm a bit concerned about all of the complexity that this adds. I'd like to get to some agreement about a plan that is easier to reason about before starting more work.

With something like the below you don't need to think about prepending, composing, etc.. It's a field that can be safely removed or altered at some point, and it seems to satisfy the needs of removing and replacing the default system prompt:

  - name: gpt-4o-mini
    provider: openai
    model: gpt-4o-mini
    apiKey: sk-xxx
    roles:
      - chat
      - edit
      - apply
    promptTemplates:
      defaultRules: ""

If defaultRules is set, then we would just entirely replace our defaults with this, which would effectively remove them if you set it to "". If you want to write your own tool use rules, then you can do so in the rules section as usual.

What do you think?

[ferenci84]

Do you want to remove systemMessage alltpgether? If not removed/deprecated, how would that work if any value is set?

[ferenci84]

If you want to write your own tool use rules, then you can do so in the rules section as usual

This is not clear to me. The tool use rule is added dynamically, only when tools are enabled. The rules section contains messages that are added always. Or, do I miss something?

Looking at the rules config option, it raises further questions:

rules:
  - uses: myprofile/my-mood-setter
    with:
      MOOD: happy
  - Always annotate Python functions with their parameter and return types
  - Always write Google style docstrings for functions and classes

Would it appear before or after the "defaultRules" specified in the model, or if "defaultRules" are defined, it replaces everything? What is the advantage of having a list of rules over having a single systemMessage?

(It's a side question, but not sure whether the "uses/with" combination is documented anywhere?)