[Feature] Semantic Validation for the EvalEvalBot #138
nelaturuharsha
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I think the easiest option would be for batch processing by a small light-weight LLM (a quantized version is fine that can run on CPU) which is able to process one record at a time - as a prefill task and is able to provide summarized feedback. We can iterate on a system prompt which would allow for an 'agentic' pipeline where the same model can perform different stages of semantic validation (controlled by sub-prompt).
I propose we can also look into https://arxiv.org/abs/2507.19457 to make it better aligned.
Beta Was this translation helpful? Give feedback.
All reactions