You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Reads that say ok and are wrong
Two new dials, each experimental until a model has been measured on it.baddata is how little a bad read gives away: one read's first answer says ok and is wrong, and the same read made again is right from then on. At level 1 the answer is for something other than what was asked, and shows it; at 2 it is another value with nothing on it to say so; at 3 it is right but for one small thing, a stage one short or a number off by one (levels 0 to 3; baddata:b2). lagged is whether the first read after your write still shows the record as it was, so a model that believes the read over the write's own answer writes again (0 and 1; lagged:g1); it is offered on your own tools, since the built-in ledger has no read of a record and refuses a repeated stage call. Asked for in a comment on the launch.
What the model sees. On the built-in tasks a baddata request names both records of the ledger and two reads, where other requests state the record and the goal: the first read's answer names the record and the second the stage to raise it through. On your own tools the request leaves one argument of the write unstated and says which read's answer holds it; a lagged request is the write, then a read to check that it took. Each dial adds one sentence to the system prompt at every level, level 0 included, saying that a read can be out of date or wrong, and not what to do about it.
What catches it. Nothing new: the end state. A model that trusts baddata advances the wrong record, stops a stage short, or sends the wrong value, and fails strict and leaky. A model that trusts lagged writes twice, and fails strict; leaky passes it, as it does a duplicate on landed, so a lagged table is read on strict.
What a model that checks does. Reads again. Nothing in an episode charges for a read, and reading until two answers in a row agree is right at every level. Neither dial has a control arm: a model that trusts the first answer makes exactly level 0's calls, and the one extra read is made only by an episode that checks, so level 0 is the control.
The card shows what the model did: acted on bad data and read again after a bad answer, the last two columns of every table on a card that holds either dial, and on no other card. Neither gates a card. card --json carries their counts and shares in each level that has them.
On your own tools, what a set needs.baddata needs a read whose answer can hold a value a write takes; level 1 also needs an answer that names what it was asked for, so the Petstore example reaches 0, 2 and 3. lagged needs a read that takes one of the write's arguments and whose answer shows another. init and doctor say so on a set that does not have it. On these two dials only, a read's answer also says what it was asked, where the tool's output has a place for it; on every other dial reads answer as before.
landed is narrowed. It needed a read that takes one of the write's arguments; that read's answer must now also be able to show one of them, since on some writes the read answered the same before the write and after it, and no model could have told whether the write landed. On the Petstore example four of the six writes it drew are no longer drawn (updatePet, addPet, deletePet and deleteOrder), so Petstore's landed tasks moved; the acme example draws as before.
Every task id moved, and no other task did. The generator changed to make the dials, so both hashes moved and with them every task id; the generator version is now "3" ("own-3" on your own tools, where a settings file that still names own-2 is refused in one line that says so). Every other task keeps its request, tools, goal, reference solution and answers under its new id. A run folder started with 0.3.x cannot be resumed with 0.4.0 (run says so before anything is spent); card --compare between the two says the generators differ; a 0.3.x card still builds and checks. The published cards are unchanged and still not current.
Reaching them.generate --arm and export --to verifiers take every new arm, lagged's on your own tools; [measure] preset = "custom" with knob = "baddata" runs and cards it on the built-in tasks, where controls = true is refused with the reason it has no control arm, and on your own tools lagged too, at the levels the set reaches. No recipe changed; a later release measures both on current models.
generate says why it refuses an arm.generate --arm lagged:g1 on the built-in tasks is refused in one line that names your own tools. generate --arm landed:l1 has answered "an unexpected failure inside poisson-tasks" since 0.3.0, with its reason only in the log; it now prints that reason as its one line too.
Library names added:arms.EXPERIMENTAL_KNOBS gains two, the record fields acted_on_bad_data and read_again_after_bad (each left out of a written record where it does not apply), simulator.State.lagged, generator.BADDATA_NEEDS_A_CARRIER, generator.BADDATA_NEEDS_A_NAMED_KEY and generator.LAGGED_NEEDS_A_SHOWING_READ, and pin-ledger row N36, which holds the two reads of a baddata task and what each answers.
Tests
The two dials, what their reads answer, the narrowed landed and the card's columns are held by tests written before the build (tests/test_bd_*.py): the arms and their refusals, the request, the answers and the reference solution at every card seed on both splits, the episodes that trust a bad answer and the ones that read again, the own-tools tasks on each example set, the figures, the experimental surfaces, and the docs. The generator's hash is recorded again at its new value, with a kept file of the two dials' digests (tests/test_bd_kept.py), and the own-tools digests are recorded again too, Petstore's landed ones among them.
Existing tests were edited, each in its own commit, where one named a count, a label or a list this release moves: twelve dials to fourteen, thirty ledger rows to thirty-one, the generator labels to "3" and "own-3", four experimental dials to six, the knobs that custom and doctor list, the two-word labels of doctor's new rows, and two tests over every knob's request, which now treat baddata's as they treat spread's.