Enhance CI integration and security measures - #2695
Merged
Merged
Conversation
Seven findings from the review of this PR.
Strict-secret mode redacted a list of known-risky tools and let everything
else through. That list can't hold: an OTP typed into a visible field comes
straight back out of the next get_accessibility_tree or read_page, and the
model repeats it in its own prose. Strict mode now denies by default — tool
results reduce to success plus HTTP status, tool arguments to the scalar keys
grading reads, and text/text_delta/run_status text is dropped. The clarify
question is the one carve-out, since a caller has to read it to answer it.
Consecutive text deltas were also appending raw model output onto the stored
row without passing through redaction at all.
run.result kept the raw structured result while run.summary was blanked, so
strict mode destroyed the readable summary and published the field a model is
most likely to park a credential in. It now publishes the redacted copy;
booleans and numbers, which is what grading reads, survive intact. The same
went for scheduled jobs: a child task's prompt, result, and target URL went
over the bridge untouched by anything applied to the parent run.
done_json advertised the caller's output_schema, and closeToolDefinition
stamped additionalProperties:false onto every object node — including
free-form ones. A schema declaring a plain {"type":"object"} field rejected
every key that field carried, so the run could never finish. An object that
declares no properties means "any object", not "empty object", and is now
left open. Separately, a shorthand schema was read as JSON Schema whenever it
happened to contain a field named description or required, which produced the
same unsatisfiable shape; only structural keywords disambiguate the two forms
now, and both trees share one helper for it.
On the CI side, each answered clarification restarted the full run timeout, so
five questions on a 10-minute scenario licensed 72 minutes. There's one budget
now. A run row read right after an answer can also still report the
clarification we just answered — a stale read that got graded as a user
handoff, which the poll now waits through. The smoke job's 120-minute timeout
sat under the pack's own 133-minute budget; both workflow timeouts are sized
from the catalog, and ci/test.mjs derives that bound so adding a scenario
fails the test rather than quietly shrinking the margin. A killed job skips
every cleanup path, so run.mjs now releases in-flight browser sessions and
fixtures on a signal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two follow-ups from Codex on d27ee25. Widening isJsonSchemaSpec to recognize const, anyOf, oneOf, allOf, and $ref made validateCloudOutput classify those specs as JSON Schema without evaluating any of them, so 7 validated clean against {anyOf:[{type:'string'},{type:'boolean'}]} and done_json could complete a run with a result the caller's contract forbids. It was a regression: before, an unrecognized combinator fell through to the shorthand branch and got rejected, albeit for the wrong reason. The four composition keywords are implemented now, and $ref is reported as unsupported rather than passed over, so a schema this validator can't evaluate fails closed. Optional shorthand advertised the bare type. validateCloudOutput accepts an explicit null for `nickname: 'string?'`, but the advertised {type:'string'} meant validateToolArguments rejected the call before the run's own validator saw it — the same advertised-stricter-than-enforced asymmetry as the free-form-object bug, in the other direction. An optional field now advertises its type plus null, so both validators agree.
Three follow-ups from Codex on 6cb5c65. createIncognitoBrowser already surfaces webbrain_config_result and nothing read it. A rejected or partially applied config left strictSecretMode at its default false while the signup scenario handled a password, mailbox token, and OTP — the exact run whose redaction the earlier commits were about. A scenario that declares session_settings now aborts before startRun unless provisioning confirms every one of them. The cloud's response schema is not in this repo, so the reader accepts the shapes it might take (echoed settings, an applied list, any of the usual rejected/ignored lists) and treats silence as failure rather than success. The session is still torn down by the existing finally, so aborting there leaks nothing. Recognizing a combinator by name alone repeated the bug it was meant to fix: const, anyOf, oneOf, allOf, items, enum, and $ref are all legal shorthand field names, so {const: 'string'} was read as JSON Schema and its intended required field went unchecked. A keyword now has to carry a schema-shaped value — arrays of objects for the composition keywords, and for the two that take free values, anything that is not a shorthand type token. const compared with Object.is, so an object or array const could never match a separately parsed done_json argument and a valid result failed twice before the run gave up. Both const and enum use structural equality now; enum had the identical defect and is fixed with it.
Three follow-ups from Codex on 7cce77c, and a change of approach on the first. The clarify carve-out was the last update type publishing raw model text in a strict run, and Codex is right that the strict system prompt is not a redaction boundary. But enumerating types is what produced this finding and the ones before it: each round redacted the types someone had thought of. So strict runs now redact by value instead. Whatever the run types into a page, plus any value sitting under a key the scrubber already treats as sensitive, is remembered and struck from the text of every later update — clarifications, warnings, captcha diagnostics, anything added later. The literals live in a map outside the run object so they cannot reach session storage, and are dropped when the run ends. A clarification stays answerable: the question survives with the secret gone. One limit is now explicit in the source rather than undiscovered. A secret the model only ever reads out of an ordinary field — an OTP in the body text of an inbox message — never reaches this code in a recognizable form, so a clarification quoting it still rests on the prompt boundary. Closing that would mean blanking clarification text, which makes a strict run unanswerable, or guessing at entropy in prose. The enum fix in the last commit was half a fix: validateToolArguments still compared with Object.is, so an object-valued enum was rejected at the argument gate before the cloud validator's new deep equality could accept it. Both now share one exported jsonDeepEqual, and a sweep over fourteen schema/result pairs asserts the two validators agree rather than spot-checking a keyword at a time. Finally, every post-startRun throw was being attributed to setupError, and inferStuckAt short-circuits on that before it reads the run — so a run timeout or a failed trace export was reported as a provisioning failure. Setup now ends where it actually ends.
Two P1s from Codex on b338fd0. Both predate this branch's review — they live in code from 0cc5619 — but the second is one my own review walked past. run.content, run.result, run.finalUrl and run.error reach a caller by the same two routes as an update row, and only the structured branch was redacting them. An unstructured strict run returned its final answer raw, which is the single most likely place for a model to repeat the credential it was told not to. All four now go through the run's value redaction, structured or not. Blanking would have been wrong here: with no output schema that text is the result, so it comes back readable with the literal struck instead. The structured redactor replaced string leaves only, so an OTP serialized as verification_code: 481920 went out untouched — no key pattern recognizes that field name, and the known-value redactor rewrites strings, not JSON numbers. Numbers are redacted now too. Booleans and null stay, because neither can carry a credential and a boolean outcome flag is what scenario grading reads; a strict run's structured output is assertable on booleans only, which the signup scenario already is. Worth noting the earlier fix here described preserving numbers as a feature — it is, for grading, and it was also a hole. Codex's secondary point is the sharper one and is covered: a done_json tool_call is published before argument validation can reject it, so the args are a publication path of their own. The new fixture asserts through both.
Two P1s from Codex on a1dfec9. The second is mine, and it is a correction to my own last commit rather than a new defect. The secret collector only inspected text-entry arguments, so a credential that is minted but never typed stayed unknown to the value redactor. That is exactly the disposable-signup flow: the Mail.tm account password is created inside the JSON body of POST /accounts and never touches a form field, so a clarification quoting it was published intact. Sensitive-keyed values in any tool call's arguments are registered now, and a request body is parsed as JSON or as form encoding to reach the ones nested inside it. Redacting the public result by leaf type was an over-correction. It made every string and number in a schema-valid answer a placeholder while the run still reported completed — strict mode satisfied by making the run useless. The two consumers want different things, so they get different redactors: the update row is trace and persistence with no contract to honour and keeps the blunt leaf-type redaction, while run.result and run.summary are the caller's answer and take value redaction, striking registered credentials and leaving everything the schema declared. The value redactor now also matches a number against a registered secret's string form, since an OTP typed as text comes back as JSON number — which keeps the case behind the previous commit closed without flattening `count: 3` along with it. run.summary is value-redacted for the same reason rather than blanked, so it stops being the one terminal field with different rules.
`{ const: 'string' }` is ambiguous — a shorthand field named `const`, or the
JSON Schema literal "string" — and the detector had to pick one. It picks
shorthand, which left the literal reading unreachable. A caller who means it
now says so with `$schema`, which turns off shorthand interpretation for the
whole tree rather than node by node. Unmarked specs are unchanged, so the
earlier disambiguation holds. The marker describes the document, not the
argument, so it is stripped from the advertised done_json schema.
The docs were the other half of this. `security-model.md` claimed strict mode
"prevents credential exfiltration even if the model is jailbroken into quoting
secrets", which was never true of the prompt-only implementation and is still
not true of the redaction layer this branch adds. It now says which parts are
instructions to the model, which parts are enforced underneath, and where the
enforcement stops: a secret the model only reads out of ordinary page text is
invisible to a redactor that matches on known values, so repeating it in a
clarification is caught by the prompt or not at all. Closing that would mean
blanking clarification text, which makes a strict run unanswerable, or guessing
at entropy in prose — neither is worth it for the runs this protects.
The two shorter claims in privacy-and-data-flow.md and architecture.md were
narrowed to match.
Add WebBrain Cloud CI integration
|
@esokullu is attempting to deploy a commit to the esokullu's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.