What is wrong
Three working CLI flags are missing from evalgate --help.
The HELP constant in src/cli/index.ts (lines 106-134) lists --provider, --model, --tags, --out, --md, --json, --no-fail, and the compare flags. It never mentions:
--concurrency <n>: read in cmdRun (line 161) via numFlag(args, "concurrency", 1) and passed to runSuite. It is the flag that makes real network-bound provider runs finish in reasonable time, and it is documented in RunOptions in src/runner.ts (lines 462-467).--junit <file>: read in writeArtifacts (lines 149-150) and rendered by src/reporters/junit.ts. This is how you get results into a CI test-report UI.--degrade: read in registryFor (line 59) and used to make the mock provider return corrupted output (src/providers/mock.ts lines 91 and 110). It is the fastest way to see evalgate actually fail a build, which is the whole pitch of the tool.
README.md line 206 lists --concurrency and --junit, so the README and --help disagree. --degrade is documented nowhere at all.
Why it matters
--help is where people look, and evalgate's headline use case (prove the gate catches a regression) is gated behind an undocumented flag.
Steps
- Add
--concurrency <n>, --junit <file>, and --degrade to the "Common flags" block of HELP in src/cli/index.ts, with one-line descriptions in the existing style. Note in the --degrade line that it only affects the mock provider.
- Add
--degrade to the flag list at README.md line 206 so the two agree.
- Worth a sentence in the README showing the demo it enables: run a suite, then rerun with
--degrade and watch the exit code flip to 1.
- No code change, no test change.
Comment below if you would like this one. I usually reply within a day.
What is wrong
Three working CLI flags are missing from
evalgate --help.The
HELPconstant insrc/cli/index.ts(lines 106-134) lists--provider,--model,--tags,--out,--md,--json,--no-fail, and the compare flags. It never mentions:--concurrency <n>: read incmdRun(line 161) vianumFlag(args, "concurrency", 1)and passed torunSuite. It is the flag that makes real network-bound provider runs finish in reasonable time, and it is documented inRunOptionsinsrc/runner.ts(lines 462-467).--junit <file>: read inwriteArtifacts(lines 149-150) and rendered bysrc/reporters/junit.ts. This is how you get results into a CI test-report UI.--degrade: read inregistryFor(line 59) and used to make the mock provider return corrupted output (src/providers/mock.tslines 91 and 110). It is the fastest way to see evalgate actually fail a build, which is the whole pitch of the tool.README.mdline 206 lists--concurrencyand--junit, so the README and--helpdisagree.--degradeis documented nowhere at all.Why it matters
--helpis where people look, and evalgate's headline use case (prove the gate catches a regression) is gated behind an undocumented flag.Steps
--concurrency <n>,--junit <file>, and--degradeto the "Common flags" block ofHELPinsrc/cli/index.ts, with one-line descriptions in the existing style. Note in the--degradeline that it only affects the mock provider.--degradeto the flag list atREADME.mdline 206 so the two agree.--degradeand watch the exit code flip to 1.Comment below if you would like this one. I usually reply within a day.