General discussion & summary of project owner positions

[malfoyslastname]

Use this thread for general discussion and debate regarding the Character Card Spec V2. Anyone may freely use this thread to discuss the spec. However, if you are an owner or representative for a project in the ecosystem (frontend, card editor, card repository...) please identify yourself in your first post, as we hope to achieve a consensus before validating the spec.

I may also be contacted by email at malfoyslastname@proton.me or through your own dev discussion channels (email me a link if I'm not already there).

• Spec (shorter, read first)
• Explainer (longer, justifies technical decisions)

Project positions expressed

Current state of the spec/explainer approved (this is where we'd like everyone to be)

• RisuAI (@kwaroran)
• characterhub.org (@lloorree)
• Agnai (@sceuick)
• SillyTavern (@Cohee1207)
• Pygmalion booru (devh8)
• TavernAI (@TavernAI)

Conditional or partial approval, details included

Will not support the spec

Pending

• ZoltanAI (@ZoltanAI potentially impossible to contact)

If your project has been omitted, please get in contact.

Note: The main Tavern branch and their character repository Characloud have already drifted from this ecosystem, so although we aim to remain compatible with their ecosystem, consensus between V1-compatible projects is the main priority.

[lloorree]

Hey, thanks for doing all of this. Things are obviously a lot easier to adapt on my end (I'm the dev for CHub) than on the actual chat projects', so I ultimately defer to whatever is best to work with for Risu/ST/Agnai/etc. The only thing I have reservations about is the intentional breaking change (moving the fields inside 'data'). I can pretty much already see myself or someone else winding up doing the same thing ZoltanAI did where the analogous fields are all just duplicated to make things pretend to be backwards compatible.

[malfoyslastname]

Thanks. I'm also not sure about the intentional breaking change.

If we keep V2 as a superset of V1 (i.e. V1 cards are valid V2 cards), then:

• V1-only frontends will be able to use V2 cards, but they will be silently missing some data (should not be a real issue: we aren't aware of V1-only frontends that will not support this spec)
• V1-only editors will be able to import V2 cards, but the editor will silently destroy the V2 data (might be a real issue if we can't get ahold of the ZoltanAI dev to get approval to PR the main repo)

Conversely, if we keep v2 incompatible with v1, then:

• V1-only apps will not be able to use V2 cards (should not be a real issue: we aren't aware of V1-only frontends that will not support this spec)
• V1-only editors will not be able to import V2 cards (can be a good or bad thing depending on how you look at it, but I'm leaning towards good)

If we can get ahold of the ZoltanAI dev, as well as his approval to PR the main repo, then I'll be very happy to abandon the breaking change.

[lloorree]

• V1-only editors will be able to import V2 cards, but the editor will silently destroy the V2 data *

Oh never mind I understand now. This is all good then.

[malfoyslastname]

Check out the new properties suggested in #2 by @acidbubbles

Please chime in that thread if you have any concerns or suggestions.

I will add them to the spec in a couple of days.

[malfoyslastname]

Big news

• Explainer updated with additional fields
• First draft of the V2 spec written

The spec is much shorter than the explainer. If you want to save time, I recommend reading only the spec, and checking the explainer's individual section for fields which you have concerns about.

Project owners ( @sceuick @kwaroran @lloorree @Cohee1207 @ZoltanAI and the pyg booru admin ), please give your official position on the current state of the spec and discuss any concerns raised. ZoltanAI is MIA, but once a consensus between the remaining project owners has been reached, the next steps will be:

• Officialize a character book format compatible with both Agnai and Silly's current lorebook features (I will work on this in parallel)
• Start implementing, discuss who needs help with writing PRs, and coordinate on the deployment timing.

[sceuick]

Thanks for coordinating this.
This all looks good to me. The only thing causing me hesitation is the system_prompt only supporting {{user}} | {{char}} placeholders. This will require some extra work to support as the Agnai equivalent contains placeholders for all other portions of the prompt, but I'm sure there is some elegant solution to work around this.

[kwaroran]

RisuAI dev here,
Looks all good to me, no concerns.

[malfoyslastname]

@Cohee1207 said in #2

What should the frontend do with these v2 fields system_prompt, post_history_instructions if the user-selected generation API doesn't use system prompts? Should we let them be displayed but with an additional note that they won't be used during the reply generation?

Might be an ignorant question, but can you say what APIs those are, if they are not hypothetical?

My gut feeling suggestion is:

• If you'd normally send the user's global system prompt in some way (prepending it to character defs), then do that with the character system_prompt. If not, then ignore the character system_prompt. You may communicate that (or not) to the user in any way you want.
• same for jailbreak/post_history_instructions

[Cohee1207]

Might be an ignorant question, but can you say what APIs those are, if they are not hypothetical?

Anything that is not OpenAI, and not oobabooga or KoboldAI running instruct models.
Chat and novel models will not benefit from force-including these. It may break formatting and cause leaks.

[malfoyslastname]

@Cohee1207 If those models don't currently get "System prompt" and/or "Jailbreak" inserted, then it sounds natural to not insert the character card equivalent (we'll add that to the spec). Character editors should show for those fields a subtext like "For compatible models only (model list goes here)". This will give botmakers a hint that everything crucial should remain inside the description field.

In certain rare cases, that means certain character cards won't work well on those models. But I'd say cards are already often very model-specific (creator_notes helps here, and is often already used this way on characterhub). I think making cards more model-agnostic would be a very difficult goal to work towards, perhaps even futile.

With V2, creators will keep on making separate cards (or forks) for different models if the need exists, as before. I don't think introducing those fields will cause such a need to increase substantially (if indeed at all).

[Cohee1207]

If those models don't currently get "System prompt" and/or "Jailbreak" inserted, then it sounds natural to not insert the character card equivalent (we'll add that to the spec).

Makes sense. Other than that - no comments, looks good.

[lloorree]

No further concerns on my end.

[TavernAI]

Hello everyone.

The common format is a very good idea, I think it's important that users can upload the same characters, regardless of the interface. TavernAI already uses its own format, designed taking into account the online synchronization of charaCloud and the use of the smallest webp file size. Since online and local user character libraries grow rapidly, this is starting to play a significant role.

I will be happy to add support for the new format to TavernAI. Since the spec field will appear, it will be easy to import various versions.

If the idea comes to fruition, I will add spec field(spec: "tavernai") in current card format of TavernAI for backward compatibility.

[malfoyslastname]

@Cohee1207 @sceuick @kwaroran Here is my proposed typing for the CharacterBook type, please confirm if this works for you, if you have any concern or suggestion, or if any of it is unclear or requires clarification.

/**
 * ? as in `name?: string` means the `name` property may be absent from the JSON
 * (aka this property is optional)
 * Optional properties are allowed to be unsupported by editors and disregarded by
 * frontends, however they must never be destroyed if already in the data.
 *
 * the `extensions` properties may contain arbitrary key-value pairs, but you are encouraged
 * to namespace the keys to prevent conflicts, and you must never destroy
 * unknown key-value pairs from the data. `extensions` is mandatory and must
 * default to `{}`. `extensions` exists for the character book itself, and for
 * each entry.
 **/
type CharacterBook = {
  name?: string
  description?: string
  scan_depth?: number // agnai: "Memory: Chat History Depth"
  token_budget?: number // agnai: "Memory: Context Limit"
  recursive_scanning?: boolean // no agnai equivalent. whether entry content can trigger other entries
  extensions: Record<string, any>
  entries: Array<{
    keys: Array<string>
    content: string
    extensions: Record<string, any>
    enabled: boolean
    insertion_order: number // if two entries inserted, lower "insertion order" = inserted higher

    // FIELDS WITH NO CURRENT EQUIVALENT IN SILLY
    name?: string // not used in prompt engineering
    priority?: number // if token budget reached, lower priority value = discarded first

    // FIELDS WITH NO CURRENT EQUIVALENT IN AGNAI
    id?: number // not used in prompt engineering
    comment?: string // not used in prompt engineering
    selective?: boolean // if `true`, require a key from both `keys` and `secondary_keys` to trigger the entry
    secondary_keys?: Array<string> // see field `selective`. ignored if selective == false
    constant?: boolean // if true, always inserted in the prompt (within budget limit)
    position?: 'before_char' | 'after_char' // whether the entry is placed before or after the character defs
  }>
}

[malfoyslastname]

@Cohee1207 @sceuick @kwaroran @lloorree

Other questions while everyone's reviewing the character book typing.

• Can we commit to supporting APNG files? (Hopefully we can simply encode the data in the same way it is currently encoded for PNGs)
• A note about WEBP: I propose we leave the WEBP question to the future. At the moment, Main Tavern is using a new encoding which CharacterHub does not want to support for technical reasons. Silly is using the old Tavern encoding. Agnai has no support.

[lloorree]

No issues with this book format, with committing to supporting APNG, or with putting WEBP aside for now.

[kwaroran]

No issues about format too.
Risu already supports APNG, so it won't be a problem for Risu.
About webp, Ok.

[Cohee1207]

For the book format, it may eventually require an option for case-sensitive key matching. Right now keys are assumed to be case-insensitive. Not many frontends other than KoboldAI supports this, but this may be a useful addition to power user. Should that be an extension field or a part of the format? Everything else looks okay to me.

1. Jimp, the library Silly uses to handle images, doesn't support APNGs. @kwaroran can you please give me some clues on how Risu does that?
2. WEBPs should go away from our radars eventually. PNGs were already a de-facto standard for character sharing since late January. Despite WEBP being a pretty old format already (spawned in 2010), its support across different tools and libraries is very poor.

[malfoyslastname]

@Cohee1207 Then I propose adding, if @sceuick and @kwaroran agree, the optional ENTRY field case_sensitive.

Again, optional field = it's not required to support it, just be careful to never destroy it if it's present in the JSON.

[kwaroran]

@Cohee1207 Risu only uses png-chunks-encode / png-chunks-text / png-chunk-text' (which was used by tavern long ago) and doesn't uses other library. APNG doesn't goes out of the code of it.

[kwaroran]

@malfoyslastname no problem. I agree with it.

[malfoyslastname]

@sceuick @kwaroran @lloorree @Cohee1207

The spec has been APPROVED. Implementation and deployment starts now. We may still amend the spec if unexpected problems are raised during implementation, but let's hope that is unnecessary.

Last caveat regarding .APNG: Agnai and SillyTavern (@Cohee1207) will report back on their attempt to support it. If both frontends had no trouble adding support, it will be added last minute as part of the V2 spec.

[Cohee1207]

I added very basic and limited support for APNG into SillyTavern (the image needs to be cropped and resized by the user).
https://github.com/Cohee1207/SillyTavern/commit/c48af795daa0cafac84ab5c1b707f1b326163389

[acidbubbles]

I was thinking, it might be a good idea to provide a json schema (https://json-schema.org/) so you can validate a card easily, as well as get auto complete in editors.

Here's what GPT-4 gave me when asking to convert your latest typescript post:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "description": {
      "type": "string"
    },
    "scan_depth": {
      "type": "number"
    },
    "token_budget": {
      "type": "number"
    },
    "recursive_scanning": {
      "type": "boolean"
    },
    "extensions": {
      "type": "object",
      "additionalProperties": true
    },
    "entries": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "keys": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "content": {
            "type": "string"
          },
          "extensions": {
            "type": "object",
            "additionalProperties": true
          },
          "enabled": {
            "type": "boolean"
          },
          "insertion_order": {
            "type": "number"
          },
          "name": {
            "type": "string"
          },
          "priority": {
            "type": "number"
          },
          "id": {
            "type": "number"
          },
          "comment": {
            "type": "string"
          },
          "selective": {
            "type": "boolean"
          },
          "secondary_keys": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "constant": {
            "type": "boolean"
          },
          "position": {
            "type": "string",
            "enum": ["before_char", "after_char"]
          }
        },
        "required": ["keys", "content", "extensions", "enabled", "insertion_order"]
      }
    }
  },
  "required": ["extensions", "entries"]
}

What do you think?

[malfoyslastname]

Agnai status update:

• APNG: stalled until higher priority stuff is implemented
• V2 implementation: under way, will require some time

A note about characterhub/chub: @lloorree might be busy for a while

@acidbubbles i'd like to make a new file containing code examples. a json schema for character book and tavern card would fit very well into such a file. i'd also like to include some conversion functions in there (i have them written locally, just need to polish them), and perhaps some parsers using popular parsing libraries like zod/runtypes/io-ts. Though it seems no one favors the use of parsers in the ecosystem atm...

[acidbubbles]

Yeah I guess I was referring to using $schema in the character card json to easily know what that json is. With this, it's easy to simply import a json and know what that json contains. The actual schema is less important :)

See this as a recognized, standard way of saying what a json file contains.

[malfoyslastname]

@acidbubbles
Ah ok, I didn't get that. I've never seen this method in use before--I don't feel comfortable investigating its utility and adding it to the spec this late into the process (sorry)

[acidbubbles]

Don't be sorry, I don't have a stake in this :)
To answer the utility part, this is widely used for Azure, AWS, and Google Cloud configuration files, in docker compose files, swagger API, kubernetes. Also, npm package.json, tsconfig.json and a few others make it optional. This is useful when you may need to distinguish which json contains what, if the filename cannot. Rather than making up a field yourself (e.g. the kind field in agnai, or $type for Newtonsoft serializer) it's a standard one, which means in Agnai, if you want to import a bunch of characters, books, chats, etc. you'd be able to.
Please note, I'm sharing info but I understand and won't argue with your decision :)

[kwaroran]

RisuAI Status Update: supported unclear v2 format from 0.8.0, now fully supports spec v2 after 1.16.0

[malfoyslastname]

Agnai update: Hoping to complete V2 implementation this week.

@kwaroran @lloorree @sceuick @Cohee1207 @TavernAI

Utility you might find useful: Online parser/validator, v1 field backfiller, v1 card updater, and example valid JSONs.

https://malfoyslastname.github.io/chara-card-utils-web/

Comes with a library if you want to use it, and extensive documentation

[malfoyslastname]

@lloorree @TavernAI

Notice of a late addition to the spec that was agreed on by Silly, Agnai, Risu, and miku.

system_prompt

Frontends MUST support the {{original}} placeholder, which is replaced with the "system prompt" string that the frontend would have used in the absence of a character system_prompt (e.g. the user's own system prompt).

post_history_instructions

Frontends MUST support the {{original}} placeholder, which is replaced with the "ujb/jailbreak" string that the frontend would have used in the absence of a character system_prompt (e.g. the user's own ujb/jailbreak).

[luizsan]

Hello everyone! I'm a developer of a smaller frontend for personal use, and have made some small contributions to the original TavernAI project. I really appreciate everyone's efforts for creating standard formats!

I wanted to clarify an issue that was a pain point earlier in my project: there's no field to store the creation nor the last modification date of a character, but it's something that it's already implemented in SillyTavern and other frontends as well, each one with a different format.

If it's something that was overlooked, I propose the implementation of the create_date and last_modified fields to the V2 spec, to store their respective dates in the Unix Epoch format.

[SrJuggernaut]

Don't you think it would be more beneficial to have multiple scenarios, each with different greetings?

[johnbenac]

So, with regard to the readme...

This document provides a great foundation for the character card spec. Its strength lies in its clear definition of the character card structure, allowing anyone to easily check their card against the spec.

However, I've noticed the spec includes numerous requirements for frontends, which seems outside the intended scope. As someone looking to integrate front-end capabilities into Autogen from Microsoft, which currently supports only the description and system message, I find these frontend-specific requirements a bit misplaced.

For instance, the spec mandates that frontends must offer "swipes" on character first messages, with each string in the 'alternate_greetings' array serving as an additional swipe. While I understand the rationale behind this, it doesn't align with the primary purpose of a data format spec. In my case, where I have automated agents interacting without human intervention, such features like alternate greetings don't necessarily add value, though the core concept of the character card does.

In my view, the spec should focus on defining and constraining the character card structure itself, not dictating the implementation details of the applications that utilize it. This separation ensures clarity and allows for greater flexibility in application design.

I believe that usage suggestions and examples, such as the intended use of 'alternate_greetings', could be valuable. These could be framed as non-mandatory guidelines or best practices for frontends. This way, they serve as helpful advice without being prescriptive or appearing as requirements within the spec.

Perhaps a separate document or section for frontend guidelines or best practices could be beneficial. This would keep the character card spec focused and concise, while still providing useful implementation ideas for developers.

In summary, while the spec provides a strong basis for character card formatting, I recommend revising it to focus solely on the data format. Frontend implementation details, if necessary, could be addressed separately to avoid confusion and maintain the spec's clarity.

It's akin to setting a standard for gasoline without specifying how car engines that use this gasoline should function. While engine recommendations can be useful, they belong in a different context, not within the gasoline specification itself.

[johnbenac]

Also, are you familiar with the "semantic web," schema.org, and JSON-LD?

It seems like most (all?) of the definitions of the field should be in a context document that is referenced at the top of each character card with a URL.

Here is an example of a character key (property), whose value shall be a person type, and whose definition can be linked to at schema.org: https://schema.org/character

And here are all the attributes of the person type: https://schema.org/Person

Now, you dont have to use schema.org, as their person type doesnt have all the fields that are useful for a frontend.

But if you have the definitions defined in a context that is linked to in a character card, that would be the way of the future.

Not that this is necessarily what anyone is going for right now, but the reason that Google, Bing, Yahoo, etc. made schema.org is to help search engines navigate and parse content. So a character card search engine would benefit from having character cards v3 be according to JSON-LD and linked data and field definitions that are defined in context documents (maybe only one context document, hosted right here in this repository?) that are linked to directly from the character card.

And which you are at it, could you add fields that make it useful for programs like autogen: https://github.com/microsoft/autogen/blob/main/autogen/agentchat/conversable_agent.py and Taskweaver: https://github.com/microsoft/TaskWeaver/blob/main/taskweaver/planner/planner_prompt.yaml and ChatDev: https://github.com/OpenBMB/ChatDev/blob/main/CompanyConfig/Default/RoleConfig.json

I don't think that anybody does it better than these tavern style cards when it comes to getting AIs to behave as the user wants them to behave. Character cards make this a solved problem, and this character card spec can be extended to make it useful beyond the existing frontends that use it today. Just look at these other examples, and you will see that they aren't using all the tools in the character card toolkit! And it shows! As someone who has used these other tools, it's frustrating for me to have these agents break character when I'm trying to get them to do useful tasks. I'm trying to incorporate features from these tavern style frontends and want to use character cards, but I'm going to have to add other fields such as function maps, and other "llm_config" information that is not currently in the character card spec.

[johnbenac]

And of course, how could I leave out the OpenAI Assistants API!!

Create assistantBeta
POST

https://api.openai.com/v1/assistants

Create an assistant with a model and instructions.

Request body
model
Required
ID of the model to use. You can use the List models API to see all of your available models, or see our Model overview for descriptions of them.

name
string or null
Optional
The name of the assistant. The maximum length is 256 characters.

description
string or null
Optional
The description of the assistant. The maximum length is 512 characters.

instructions
string or null
Optional
The system instructions that the assistant uses. The maximum length is 32768 characters.

tools
array
Optional
Defaults to []
A list of tool enabled on the assistant. There can be a maximum of 128 tools per assistant. Tools can be of types code_interpreter, retrieval, or function.

Show possible types
file_ids
array
Optional
Defaults to []
A list of file IDs attached to this assistant. There can be a maximum of 20 files attached to the assistant. Files are ordered by their creation date in ascending order.

metadata
map
Optional
Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format. Keys can be a maximum of 64 characters long and values can be a maxium of 512 characters long.

Returns
An assistant object.

At some point, users may be able to drop in character cards that are tied directly to discrete OpenAI assistants!

I believe that OpenAI is going to add lots of frontend goodness for assistants in the future, including possibly group chat amongst assistants. Users will store their "lore books" on OpenAI servers.