feat: promptfoo eval --filter-failing outputFile.json #742
Conversation
mikkoh
changed the title
mikkoh
force-pushed
the
mikkoh/run-failing
branch
2 times, most recently
from
921864d
to
3bd36a3
Compare
| const {results} = await readOutput(outputPath); | ||
|
|
||
| if (results.version < 2) { | ||
| throw new Error(`Unsupported output version: ${results.version}`); |
There was a problem hiding this comment.
I put this here just cause I'm not familiar with the differences between versions
There was a problem hiding this comment.
Should be safe to remove this check 👍
src/main.ts
Outdated
| firstN: cmdObj.firstN, | ||
| pattern: cmdObj.pattern, | ||
| failing: cmdObj.failing, |
There was a problem hiding this comment.
I'm wondering if these command line params should be changed to be something like:
filterFirstN
filterPattern
filterFailing
There was a problem hiding this comment.
Ok... I'll make the change in this PR
src/main.ts
Outdated
| firstN: cmdObj.firstN, | ||
| pattern: cmdObj.pattern, | ||
| failing: cmdObj.failing, |
| const {results} = await readOutput(outputPath); | ||
|
|
||
| if (results.version < 2) { | ||
| throw new Error(`Unsupported output version: ${results.version}`); |
There was a problem hiding this comment.
Should be safe to remove this check 👍
| @@ -1228,3 +1244,26 @@ export function getStandaloneEvals(): StandaloneEval[] { | |||
| }); | |||
| return flatResults; | |||
| } | |||
|
|
|||
| export function providerToIdentifier(provider: TestCase['provider']): string | undefined { | |||
There was a problem hiding this comment.
Really appreciate you adding these helper functions. I'm aware that ApiProvider vs ProviderOptions and similar variations are some of the ugliest parts of the code :(
There was a problem hiding this comment.
No prob. I love the flexibility of promptfoo. You can use it many ways. But it also does brings on complexity. I suspect little utility functions to simplify logic can go a long way.
| @@ -552,8 +552,9 @@ async function main() { | |||
| 'Run providers interactively, one at a time', | |||
| defaultConfig?.evaluateOptions?.interactiveProviders, | |||
| ) | |||
| .option('-n, --first-n <number>', 'Only run the first N tests') | |||
| .option('--pattern <pattern>', 'Only run tests whose description matches the regular expression pattern') | |||
| .option('-n, --filter-first-n <number>', 'Only run the first N tests') | |||
There was a problem hiding this comment.
This will be a breaking change so will need to do a version bump for the next release. Unsure if theres a mechanism you use eg Github labels to determine if a version bump is needed for the next release
Should we keep -n? Should the other two filters have short forms also?
There was a problem hiding this comment.
My selfish preference is to keep -n because it feels familiar, like head -n :). Open to other short forms but I don't think it's necessary. If we find ourselves getting tired of typing everything out let's add it separately.
I do version bumps manually - since we're pre-1.0 I've included breaking changes in minor versions, but generally trying to avoid them. I think this is acceptable though. I'll merge with feat! breaking change notation and make a note in the release notes.
mikkoh
changed the title
Makes it so that you can iterate more quickly by simply running failing tests given an output file.
Example Flow
First run:
promptfoo eval --output result.jsonNext run:
promptfoo eval --failing result.json