Optimize "should-respond" evaluation AI prompt and decision logic

[cubrift]

Currently, the decision prompt evaluating whether Quart should jump into an ongoing conversation on a group chat suffers from occasional false positives (responding to irrelevant background chatter) and false negatives (ignoring context where a reply would be natural).

We need to optimize the system prompt and context structure used for the "should respond" check to make autonomous interventions more consistent, natural, and cost-efficient.

We need:

• Reduced false-positive rate in group chats.
• Clear, reproducible prompt template provided in the PR.
• Benchmarked against standard conversation scenarios (direct mentions, room-wide questions, non-relevant banter).

Quart should respond when:

• Quart is clearly being mentioned directly/indirectly ("the AI" or "Quart" or mentioned directly with @ or any other given nicknames)
• A completely open statement/question without context ("I'm feeling happy today" or "What happened on the news last week?")
• A user is talking to everyone at once ("Does anyone know when I said the event was?" or "Can everyone give me some encouragement?")
• An announcement is made ("Happy birthday David!" or "Merry Christmas everyone!")
• Anything else that Quart should be able to humanly react to (a funny meme or joke, a person describing an unfortunate event, a recent news update, etc.)

Try to keep it short and crystal clear, though it isn't terrible if it is long (the tokens get cached).

Important

Please check for edge cases. No PRs that have not gone through necessary edge case checks will not be merged. Quart should respond sufficiently, like a human, to cases that can be unfamiliar.
Also check that your prompt is static and does not change. The evaluation prompt should not change no matter the circumstances.

[nolanefe]

Hi, I’d like to work on this issue if it is still available.

I’ll first trace the current “should-respond” decision path, prompt construction, context selection, and any existing tests or fixtures. Then I plan to:

• keep the evaluation prompt static and reproducible
• make the response criteria explicit for direct mentions, open-ended room-wide questions, announcements, and naturally relevant conversational moments
• reduce false positives from unrelated background chatter
• preserve cases where a human participant would reasonably respond
• keep the prompt concise and avoid unnecessary context/token use
• add a focused evaluation set covering direct mentions, indirect mentions, room-wide questions, irrelevant chatter, announcements, jokes/memes, personal updates, and ambiguous edge cases
• compare the current and revised behavior using the same scenarios and document the results
• add regression tests for the accepted prompt/decision logic

Before I begin, do you want the PR to change only the prompt and evaluation fixtures, or is it also acceptable to make a small deterministic change to the surrounding decision logic if the benchmark shows that prompt-only changes are not sufficient?

[cubrift]

Go ahead! I seriously do not mind if you make a couple extra changes, just make sure you make them as separate commits so the changes are visible.
Your plan sounds absolutely excellent, probably out the scope of what I can think of. Sounds brilliant to me!