Prompt Library

[calebsheridan]

Suggestion. In #18 multiple prompts are introduced. This allows users to add multiple prompts, but users must manage their own prompt library to enable/disable prompts by adding/removing prompts.

Instead, prompts could be stored locally and enabled/disabled for certain runs (same way that model selection works now).

• Each prompt may be extended to include a uuid and display title
• Prompt selection in grid params should use checkboxes to select/deselect prompts by their title
• Prompt selection in grid params should open a prompt editor to manage the library
• Prompt editor should be extended to fully manage prompts (edit, add, remove, export as JSON)

tbd

• Should system prompt + user prompt be combined at the prompt level, or remain separated?
• Should individual prompts be linked to models, such that the grid search can be limited for known combinations that do not produce good results?

[dezoito]

One quick idea before commenting on the above... you mentioned that at some point it would be cool to have multiple system prompts.

What if use used that same library UI for "regular" prompts and system prompts?
We could have a checkbox or toggle where the user defines a prompt as one or the other.

---

Instead, prompts could be stored locally and enabled/disabled for certain runs (same way that model selection works now).

I love the idea of a prompt library... It would be a good case to introduce a local database without needing to rewrite what's already working (and could be migrated later, after DB kinks are sorted out).

Each prompt may be extended to include a uuid and display title

Agreed.

Prompt selection in grid params should use checkboxes to select/deselect prompts by their title

I see this in a slightly different way maybe... I think the prompt library should be entirely contained in a modal - so we don't overcrowd the "grid params" form - we add just an IconButton to use the prompt library.

When the user uses the library, the prompts they select could be added to the list of prompts already in the form, and not overwrite any existing entries (maybe they just want to quickly test a new idea against one of the prompts that is already "stable").

If the default prompt is empty, it could be overwritten by the first selection.

Prompt selection in grid params should open a prompt editor to manage the library
Prompt editor should be extended to fully manage prompts (edit, add, remove, export as JSON)

By "prompt editor", do you mean the component defined in prompt-dialog.tsx ?

For the sake of checking my assumptions, here's how I understand this UI and what the user can do (please correct me if I am missing something):

From the prompt library UI:

• Select and remove prompts from the current experiment
• Perform CRUD operations

From the grid params interface:

• Add and remove prompt fields from the current experiment
• Edit the contents of a prompt in the context of the current experiment

From the prompt-dialog component

• Edit the contents of a prompt in the context of the current experiment and apply the edits to the original field
• Select one of the existing prompts in the library and using its contents
• [?] Save the current edit, overwriting the entry in the library
• [?] Save (maybe "Add" would be a better label), the current text as a NEW entry in the library
• [?] Delete the current entry from the library (even if it was edited to something else)
• [?] Export as JSON

If you are proposing prompt-dialog to be a single component used by the library and the form (which makes sense), the ones I marked with [?] are the ones I think could lead to user confusion:

Saving and deleting should clearly indicate that this action refers to the entry in the library, not in the current experiment.

This would also require this component to keep in state the UUID of the prompt (a null would indicate that it's working with a manually entered prompt like we have now).

Again, just making sure we are reading the same page on this feature :)

---

Should system prompt + user prompt be combined at the prompt level, or remain separated?

I'd rather keep them separately, for two reasons:

1- Some models may react differently to a system prompt vs. the string of the system prompt passed as a user prompt.

2- Personally, I have lists groups of prompts (tasks) that are grouped under a system prompt (that define a role or "personality")

Should individual prompts be linked to models, such that the grid search can be limited for known combinations that do not produce good results?

Oh man, that could be a can of worms :)
How about leaving this for future consideration, until the prompt library is working ?