Repository navigation
What narrow AI task should the harness test next? #5
fusiontechstrategies
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
AI Accuracy Harness v5.0.0 currently demonstrates a deliberately narrow claim: bounded workers completed 39/39 qualification calls, while an open-ended security-review adapter was rejected because its evaluation was not deterministic enough.
I want the next example to come from a real task that people need, but it still has to meet a strict bar:
Good candidates might include exact configuration extraction, choosing from a preregistered option set, or a transformation checked by a project-owned parser and invariants.
If you have a candidate, please share:
The new deterministic-oracle issue form is also available for a more structured proposal:
https://github.com/fusiontechstrategies/AI-Accuracy-Harness/issues/new/choose
Roadmap:
https://github.com/fusiontechstrategies/AI-Accuracy-Harness/blob/main/ROADMAP.md
I am especially interested in examples where a fluent, schema-valid answer can still be wrong in a way that matters.
All reactions