Repository navigation
Factory 0.5.0 adds holdout scenarios: end-to-end checks written from your original request that only QA and you can read. QA's checks used to come from the same task and plan the Builder read, so a Builder could aim its code at the checks it expected. A Builder never sees holdout scenarios, so they show whether the work does what you asked, not just what its own tests check.
This is phase 4 of the factory's redesign.
How it works
- A Scenario writer starts with each Builder. It reads the task's description and your answers to the plan's questions, never the plan. It reads the code only to learn how a person uses the product. It writes 3 to 7 scenarios, each with a starting state, steps and what you should see, including at least one failure case. Its final message is only a count, and it doesn't wake the Lead.
- Only QA and you read them, on the task in the Factory page, or with
bb factory scenarios APP-12from your own terminal. They're kept in the plugin's database, never as a task comment, because the Builder reads every task comment. Status, the board, the merge card and Health show counts only, such as "Holdouts: 4/5 pass at abc1234". - QA runs each one at the commit it checks and records pass, fail or invalid. Invalid is for a scenario that misreads your request or can't be run here; QA gives the reason, and you see it on the task. QA reports a failure to the Builder as a symptom, never by copying the scenario.
- QA can't pass the task while a scenario has failed or hasn't been run at that commit, or while the writer is still at work. A fail is never held. If the writer ends without saving any scenarios, QA passes as before, and assigning QA again starts a new writer.
- You can change them on the task in the Factory page, or with
bb factory scenarios-setfrom your own terminal. Each save is a new version, and QA's results on the earlier one stop counting. If QA had already run them, the Lead is asked to run QA again. - Health shows how many scenarios passed on QA's first run of each task. The Builder never saw them, so they measure the work better than its own tests do.
Keeping them from agents
- The writer's thread isn't advertised. It isn't one of the Lead's child threads, and no task comment, status, board, run ledger or Lead message names it.
- QA reports through the factory. With holdouts on, QA's final message reaches the Lead only if it quotes no scenario. A message that quotes one is held back, and you're told. A QA task comment that quotes a scenario, which the Builder could read, is flagged to you too.
- Reads and saves from a terminal are reported. The plugin can't tell your terminal from an agent's, so each time scenarios are read or saved from a terminal outside an agent thread, you get a notification.
- Agents are told never to look for the scenarios, relay them, or use
bb plugin rpcon the factory.
Also in this release
- The Lead's standing rules are shorter. They're re-sent every turn, so they survive compaction. For Live Caption Studio and Alpha Learning website, with every option on and the Lead merging when green, they ran past bb's limit of 4,096 characters for them. They now fit for every factory name, with a line about holdouts added.
When you upgrade
- New factories have holdouts on.
- Existing factories keep them off until you turn them on:
bb factory holdouts APP on, or the Holdout scenarios switch in Settings on the Factory page. Turning them off is from your own terminal only. - A new seat in Agents: Scenario writer. It defaults to QA's provider, Codex, and falls back to QA's provider when its own isn't available. Presets treat it as a reading seat.
- With holdouts on, the Lead hears QA's result as a "[factory] QA … finished" message instead of the usual child-thread notification.
- Running agents pick up the new rules on their next turn.
Limits
- Detection, not a security boundary. An agent with file access can still read the plugin's database or search for the writer's thread. QA could describe a scenario in its own words. The Factory page can't tell your tap from an agent's call.
- Only the first build is fully blind. Reporting a failure, even as a symptom, reveals part of that scenario.
- Not tried for real yet: no factory has holdouts on. Turning them on for one factory first would show how the scenarios, QA's runs and the cost of the extra thread work out.
Full Changelog: v0.4.7...v0.5.0