Repository navigation
Can a model distinguish a stop signal from a stale or unrelated alert? #1858
sylvesterkaczmarek
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I’ve built 50 paired cases around one decision: should an action with two valid approvals still go ahead after a new alert? In half the pairs the alert changes the correct decision. In the other half the approval still stands. For example, an editor’s hold should stop a brochure release if it concerns the current revision. The same hold on the previous revision should leave the release alone.
The 100 prompts cover item identity, sequence, revision, decision authority and expiry. They use the existing JsonMatch grader. Each label can also be checked independently. I’d report missed holds and unnecessary stops separately.
In a local pilot, Llama 3.2 1B Q4_K_M with constrained JSON output got 18/100 correct and both halves of 1/50 pairs correct. It requested another check on 22 of the 25 updated cases where approval still stood. I haven’t run the qualifying OpenAI baseline yet.
Dataset, grader configuration and pilot outputs
Would this be a useful registry submission if the OpenAI baseline also produces repeatable errors?
All reactions