Update Measure in AI eng #465
Conversation
| } | ||
| ); | ||
|
|
||
| Eval('text-match-eval', { |
There was a problem hiding this comment.
Should we encourage users to add capability name to metadata?
There was a problem hiding this comment.
IMO capability should be either
- parsed automatically by traversing the trace
- put somewhere more prominent than metadata, which imo is a place for the user to stick whatever they want
There was a problem hiding this comment.
@c-ehrlich we need to write something about cli auth: axiom login, maybe @manototh can help us structure this in a good way. There would be 2 ways to authenticate the SDK:
- Either using CLI, and then you only need to specify a dataset in
axiom.config.ts - Specifying an API token and dataset in axiom.config.ts
There was a problem hiding this comment.
@thesollyz Thanks for this. I would specify API token only because that's what we use in the rest of the AI eng docs anyway.
There was a problem hiding this comment.
@manototh not sure what do you mean? CLI auth is a new feature that we want the users to use.
There was a problem hiding this comment.
Do you mean CLI auth is new feature in the context of evals?
If users need to specify the API token anyway in many other places in the setup, I think it would complicate things if we asked them to authenticate the CLI as well. Or am I missing something?
There was a problem hiding this comment.
yes the CLI auth is a new feature in context of evals. They don't always have to pass a token, its either:
- pass a token
- or login with CLI (then you don't need to pass a token)
Co-authored-by: Islam Shehata <islam@axiom.co>
|
Closing in favor of #473, feel free to re-open if that's wrong |
3 participants
No description provided.