Logger plugin interface #456
AlisinaDevelo
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Following up on #451. I focused too much on response logging there. The good/bad categories shouldn't be part of a general logger interface.
I'd start with one log(event) method and event types for run/trial lifecycle, progress, scores and generated responses. The CLI logger would format those events, while JSONL and WandB could use the raw values. Events would carry run/trial context where relevant, and scorer-specific fields would stay with the scorer.
I'd keep prompt and response capture opt-in, so a logger used for metrics doesn't have to collect text or trigger extra generations.
Does that shape make sense, or would you prefer separate callbacks for the different events? I'll leave the implementation alone until #446 and the remaining plugin changes settle.
All reactions