System-wide custom model settings (lightweight GPTs)

[davidmarble]

@danny-avila This project is awesome. I compared a handful of options before settling on deploying this on a VPS as a way to help family members explore GPTs under a single OpenAI API key and minimal setup for them.

User Story

As an administrator, I'd like more options to customize the set of models offered to all users on my LibreChat instance.

Proposed Features

1. Support defining a custom set of models per endpoint with options, some common to all models, some specific to particular models. For example:
   • Common options among all models:
     • name (default if blank/unspecified: model)
     • model (required)
     • desc
     • prompt_prefix (i.e. system prompt)
     • max_tokens
     • default: true (for one model that should be the initial default for all users; we can have a separate discussion about your work in feat: Implement Default Preset Selection for Conversations 📌 #1275 and whether to also support persisting the last selected model for each user)
   • OpenAI specific:
     • temperature
     • top_p
     • freqency_penalty
     • presence_penalty
   • etc.
2. Support defining default options for all models of a particular endpoint
3. Support selecting a default model per endpoint for all users.
4. Display models with name and optional desc

Mockup

Based on Example 2 below

Example YAML configs

I haven't considered how this could best fit within the current dotenv extends/overwrites docker yaml config approach, or if it merits a re-think of how config is done (e.g. migrate env to yaml or put model config in a separate new yaml file or some other approach).

Config docs draft

Instructions: Add models and add/edit options as needed. For each endpoint, e.g. OpenAI, defaults lists the default settings used by LibreChat. You can set defaults for all models of that endpoint type with the defaults key, or override values within individual model definitions. Set default_model: true on 1 model to make it the default for all users. If default_model is not set, the default will be ____ (not sure what the logic is). Consult endpoint provider API docs for documentation about options.

endpoints:
  OpenAI:
    #defaults:
    #  temperature: 0.8
    #  top_p: 1
    #  freqency_penalty: 0
    #  presence_penalty: 0
    #  system_prompt:          # blank by default
    #  max_tokens:             # defaults defined in maxTokensMap in /api/utils/tokens.js, else 4095
    models:
      #- model:                # required, OpenAI API model name
      #  name:                 # optional display name
      #  prompt_prefix:        # optional system prompt
      #  temperature:          # optional
      #  top_p: <#>            # optional
      #  freqency_penalty: <#> # optional
      #  presence_penalty: <#> # optional
      #  default_model: true   # optional, can only be set on 1 model
      #- model:                # ... add models as needed ...
  ...

Example 1 (minimal)

endpoints:
  OpenAI:
    models:
      - model: gpt-4-1106-preview
      - model: gpt-3.5-turbo-1106

Example 2 (custom options)

endpoints:
  OpenAI:
    models:
      - model: gpt-4-1106-preview
        name: "GPT-4 Turbo Accurate"
        description: A logical AI who keeps answers dry and precise
        temperature: 0
        prompt_prefix: You are Accurate AI, a helpful, clear, concise, scientific minded assistant.
        default_model: true
      - model: gpt-4-1106-preview
        name: "GPT-4 Turbo Creative"
        description: A creative AI who loves to gab
        temperature: 1
        prompt_prefix: You are Creative AI, a helpful, friendly, and very creative assistant who loves to chat.
      - model: gpt-3.5-turbo-1106

Example 3 (modified defaults)

endpoints:
  OpenAI:
    defaults:
      temperature: 0.8
      top_p: 1
      freqency_penalty: 0
      presence_penalty: 0
    models:
      - model: gpt-4-1106-preview
      - model: gpt-3.5-turbo-1106
        default_model: true

Example 4 (modified defaults + custom options)

Note the use of a multiline YAML string for instructions. Prob a good idea to include an example prompt_prefix like this in the config file or instructions.

endpoints:
  OpenAI:
    defaults:
      temperature: 0.8
      top_p: 0.8
      freqency_penalty: 0
      presence_penalty: 0
    models:
      - name: GPT-4 Turbo
        model: gpt-4-1106-preview
        temperature: 0.5
        prompt_prefix: |
          You are LibreChat, a large language model based on the GPT-4 architecture by OpenAI.
          Knowledge cutoff: 2023-04
          Current date: 2023-12-05

          You are to provide clear, concise, and direct responses. Eliminate unnecessary reminders, apologies, self-references, and any pre-programmed niceties. Maintain a casual tone in your communication. Be transparent; if you're unsure about an answer or if a question is beyond your capabilities or knowledge cutoff, admit it.
      - model: gpt-3.5-turbo-1106
        default_model: true

---

Additional Notes

Think of this as somewhere between a nice config enhancement only possible for a self-hosted ChatGPT-like experience (multiple instances of the same model with customized defaults available to all users of the LibreChat instance) and supporting lightweight custom GPT characteristics like a highly customized system prompt. I could even see this laying a foundation for later supporting more off-the-shelf GPTs-like enhancements.

While the config in this proposal is presented as YAML, I could see this kind of config being part of a future web console for admins, perhaps stored in mongo. That probably wouldn't replace the need for an initial easily-modifiable config file though.

[danny-avila]

@davidmarble Thanks so much for checking out the project, I'm glad it could serve your family as it does mine.

Also great write up, it's clear what you're asking for and providing options, really appreciate how comprehensive this is.

To me, this makes the most sense as a replacement of all the Header options into 1 dropdown. I like the config idea, and we could start off with a new YAML. This makes it as simple as ChatGPT where there is only one LLM selector. Perhaps an advanced mode could be made accessible through settings, if allowed by the admin, when the custom config is used.

[davidmarble]

Great!

My first draft of this proposal actually included a request to simplify UX by consolidating dropdowns. But I removed that language because I felt it wasn't necessary to achieve the main user value of this request.

If you do go down the UX consolidation route, some considerations:

1. How much to consolidate. I'd advocate for (i) or (ii). Punt on (iii) till plugins evolve further.
   1. Merge UX for endpoint + model into a single dropdown.
   2. Merge UX for endpoint + model + presets. These are all more or less selection of the same type of thing. Probably put Presets at the top of the list, or make the menu item order admin-definable.
   3. Merge UX for endpoint + model + presets + plugins. Prob best to hold off on taking it this far. A few thoughts:
      • Consolidate: The best end user experience is the simplest. Teach/reinforce a mindset of "pick the 1 right tool in the toolbox to solve a problem". Users will primarily pick a model + 1 or maybe 2 plugins and seldom jump around to other plugins with the same model.
      • Don't consolidate: The best end user experience caters to tool composition mindset. Teach end users to compose several tools to solve problems. Future product plans might include expanding a plugin system to work with a broader array of models (remote and self-hosted), and over time the concept of a plugin should remain decoupled from models, particularly if plugins can eventually be leveraged by multiple models (model = different types of drills; plugins = interchangeable bits). In addition to plugins, might the notion of workflows at some point be added to LibreChat/ChatGPT/etc. (heading towards no/low-code territory...)? If so, prob best to keep plugins conceptually separate from models.
2. UI options
   1. One UI for all screen sizes: Perhaps a two-level menu with one level viewable at a time and drill-down/up that works on mobile and wider screens, whether touch or hover/click-driven.
   2. Responsive UI that differs on touch/mobile/tablet vs wide/hover/click-driven. Perhaps a two-stage flyout on hover for wider/hover-based, and a single menu with drill-down/up for touch/mobile screens.

[danny-avila]

I laid some groundwork for this here: #1297

I would like the most configurable option where the admin decides:

• if endpoints/models are combined or not, which endpoints/models to use
• access to settings & presets (which are singular to me despite having 2 buttons, since they're so interlinked)
• if/which plugins are allowed. Plugins can be disabled entirely by omitting the endpoint, but some should be flipped on/off as needed
• use pre-configured model options (essentially presets) as a dropdown option, like in your example
• role based config to all of the above

There are several key features incoming (RAG/Assistants API/Code interpreter/non-OpenAI Agents, all of which are in active development) that are my priority but I really see the app evolving in the way you've helped detail, with better/easier configuration.

So I really want to see this through sooner than later, and I really like the idea of a yaml file for a local setup

[davidmarble]

Good stuff! Single level list seems fine (versus 2-level in my prior comment). It might get long in the tooth if there are many options, but simplification is probably best for now.

Regarding the design above, some thoughts:

• Would like to see custom name & description fields supported (per my initial post). Logic could be if name is undefined or blank, display model. And if desc is undefined or blank, display a default/canned description for that model that includes the provider name.
  • I don't think it's necessary to always show the model name as defined in API. Unnecessary/confusing for installations that don't have technical/developer-type users. An admin can decide what detail to provide in name and desc fields, or keep one or both empty in the config and have the model and/or a canned description with provider shown.
  • Showing that a User API key is set is a nice touch in your design, but would need a bit of a change if incorporating a custom description (which might be long-ish) -- perhaps a user+key icon?
• Would like to see support for defining multiple instances of the same model with different settings (this is assumed in the config and mockup in my initial post, though perhaps I should have spelled this out more directly)
• Might be a good idea to extract provider model defaults into a setting/constants file, e.g. models.yml. That way there'd be a single source for admins to see what models are supported, default descriptions, max tokens, etc. to get ideas for what to enable and overwrite in their settings.[debug|production].yml (or whatever the replacement or augment to .env ends up being named).
  • I see a few places in code where models are enumerated for one reason or another. Some of them don't match up. Might be a good idea to refactor into a single file (like models.yml) that gets pulled in to such code paths as necessary.

Additional future idea:

• Option to set a custom icon (or use a default LibreChat model icon) for each model enabled/defined. This would be especially useful for local and remote lesser-known or custom/fine-tuned models. This might also play into a future scenario where an admin wants to lock things down and not expose the setup, e.g. per-model custom name, description, icon (with no mention of endpoint/provider) + disabled/not-shown settings and presets menus.

[SailFlorve]

Configuring description, prefix, temperature, max tokens, and other parameters even the URL, individually for each model, is indeed a very necessary requirement. I currently hardcode these.

[eburnette]

It would be nice to change models and model versions without having to re-enter prompts (especially) and other settings. I know this is a bit complicated by different models having different settings but there is a lot of commonality. On the other hand, you want it to be dirt simple for the most common case where the user isn't going to want to mess with settings. And you want it to work with prompt libraries and GPT-like functionality, again without making it complicated. Perhaps the way OpenAPI presents GPTs could be a good way to handle prompts too. You see the GPTs/personalities in the top left and have a way to explore to add more. If you never use them, you never see them.

[ceverson70]

I’m glad this request was already input. I would be okay even with just name tonstart

[ArmykOliva]

Is it already possible to add a predefined custom system message for a model?

[arishwin]

Hi, would like to follow up on this, when is the planned release? There is a limitation when using OpenAI with plugins where we can't set a system message, hope this feature would at least allow us to write a predefined custom message in the config file instead.

[danny-avila]

I would like to start on this within the next 2 weeks.

[Tazlad]

also very interested in this development id luv to see the option to add a custom system message and ability to lock it down for other users eg. restrict users to using chatgpt with custom prompt.

[danny-avila]

Absolutely, my plan right now is addressing some open issues necessary for release. Then, after the release is cut, I can work on this as a first item of priority.

[Tazlad]

that would be awsome

[AliSMAmin]

Is there any update here?

[danny-avila]

Yes: https://www.librechat.ai/docs/configuration/librechat_yaml/object_structure/model_specs

[danny-avila]

Was officially implemented in #2578