FAQ Additions: First 30 Days

[nrfulton]

Let's use this issue to track good FAQ questions/answers.

Just comment here with the following format:

## General Topic Number One

**Specific question**

specific answer

**Another question**

another specific answer

...

## General Topic Number Two


**Specific question**

specific answer

**Another question**

another specific answer

...

You don't need to make the answers perfect; we'll spruce stuff up before adding to docs/websites.

[nrfulton]

Context Management FAQs

What Type of Context Should I Use?

A SimpleContext does not retain any chat-style history. The context for any request in a SimpleContext session is only and exactly the content that you explicitly include in the request. Examples from the stdlib include:

• passing grounding_context or requirements into an Instruction component
• passing arguments into a @generative

A LinearContext is a chat history. There's an optional argument called window_size. You can use that to specify how many messages should be included.

If you do not specify a context, you are using SimpleContext by default.

Why Default to SimpleContext?

Using a single-turn context is usually a good idea in situations where you designing "agents" or other fully-automated pipelines. We make this the default because accumulating too much of the wrong context is a common failure modes of agentic programming / fully-automated LLM workflows.

Managing context using Component s instead of chat history almost always makes more sense than representing a multi-step automated pipeline as a "chat history".

And remember that you aren't stuck with only the stdlib -- you can and often should define your own application-specific Components when you find yourself managing a large amount of application context.

Then Why use Linear Context?

A couple reasons:

1. Chat formats usually work better when there truly is user input and the input modality is dialogic.
2. Some models are really aggressively trained on the "chat" format. If you need to use one of those models, you might find that a LinearContext with fairly "slim" Components works better than a SimpleContext with "fat" Components.

[nrfulton]

I-V-R

Question: My requirements are always failing. How can I make them pass?

Ah, welcome to the art of debugging generative programs 🙂

First, let out a sigh of relief. At least you know that these failures are happening. The alternative could be producing garbage output in production. Generative programs are programs, and we debug programs by... debugging! So, let's dust off the old debugger.

Tip

LLMs are stochastic; you may want to specify a seed in your instruct call during this debugging process

Sometimes requirements have false positives or false negatives. This is especially true for default Requirements because Mellea requirements use LLM-as-a-judge (LLMaJ) as the default requirement checker. LLMaJ is nice because it gets you started quickly, but the methodology is fundamentally limited -- at the end of the day, you're judging a model's output using that same model.

Our first step is to determine whether the bug is in our model's output is in the requirement checking:

1. Set a breakpoint right before the relevant call and fire up your favorite debugger.
2. In your debugger's REPL, run your exact m.instruct call without any strategy (i.e., removing any strategy= parameter, if you have one). Then, run m.last_output().

Does the instruction output look okay? If so, then the fault is in your LLMaJ check. If possible, implement a different validation strategy; see Chapters 2 and 6 of the tutorial for a couple of options. (edited)

Does the instruction output look like it really does fail one or more of your requirements? Great; the requirement check is working! Now we just have to figure out how to cajole the LLM into not failing. More on that below.

You can now do one of a few things (or all):

• try to improve the model's response.
• You can try using a different model.
• You can try tweaking your prompt.
• Use a SamplingStrategy to generate a larger number of diverse samples and hope one of them succeeds

Tip

Be aware of the Purple Elephant Phenomenon: if you tell a model "do NOT talk about purple elephants", it often can't help but think about purple elephants. Not unlike us in that respect. If these seems like it could be the root cause can replace req("my requirement") with check("my requirement") for the requirement that's behaving like a purpose elephant.

[psschwei]

we have a faq on docs