Skip to content

v4.0.0

Choose a tag to compare

@SAY-5 SAY-5 released this 10 Sep 09:00
· 42 commits to main since this release

playbook bank keeps every scenario a procedure has been run on, with its tags and how each prompt version scored it, so a case retired from scenarios.yaml keeps its history and stays in the replay. playbook regress replays the bank against a version and compares each scenario with the last version that scored it. A scenario that used to pass and now fails is a regression and exits the run non-zero, naming what broke. A scenario the bank has never scored is new, so its failure is reported separately and does not trip the guard. Pass rates are reported per tag, and --tag or --failures narrow the replay to one segment or to the cases with a recorded failure.