Deploy follow-ups: classify-call thinking pin + regression test, inbound runbook fix, publications backfill - #36
Merged
Merged
Conversation
…d in the prod investigation Investigation of the dead reply-to-review flow (2026-08-11) found latent defects that would break or undermine the pipeline even once its missing AWS/DNS infrastructure is provisioned: - The SEC-5 anti-spoofing gate merged verdicts across ALL Authentication-Results headers with "a pass wins", so a sender-forged pass header overrode SES's fail verdicts. Now only the topmost header (the one SES prepends on receipt) is trusted, and it must carry the amazonses.com authserv-id. - HTML-only replies (no text/plain part) extracted an empty body and were silently dropped. Now fall back to tag-stripped HTML, with structural quote removal (blockquote/gmail_quote). - Auto-submitted mail (RFC 3834, e.g. out-of-office) was processed and could be answered with a help email - a mail loop. Now ignored. - MAX_REPLIES_PER_TOKEN_PER_HOUR was declared but never enforced. Now a sliding one-hour in-memory window per token. - A poison S3 object was retried every poll forever. Now quarantined to failed/ after 3 attempts for manual inspection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prod sent 129 review emails telling PIs to "reply to this email to rate it" while ENABLE_INBOUND_EMAIL was off and the reply infrastructure (MX record, S3 bucket, receipt rule) did not exist - every PI who replied got silence plus an eventual bounce, which is the reported failure. Gate the reply-soliciting copy and the Reply-To header on settings.enable_inbound_email in the proposal-review reminder, the new-proposal alert, and the welcome email. When the flag is off, all three direct PIs to the web dashboard only, so outbound email can be safely re-enabled before (or without) provisioning inbound. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-08-11 investigation found every infrastructure layer of the reply-to-review flow missing on prod: no MX record on reply.copi.science, no copi-inbound-email S3 bucket, no SES receipt rule, send-only perms on the copi-ec2-ses-role instance role, and ENABLE_INBOUND_EMAIL unset. scripts/setup_inbound_email.py --check reports each layer; --provision (admin creds) creates the bucket/policy/receipt rule and prints the DNS records and the IAM policy that must be applied by hand. docs/inbound-email.md is the architecture + bring-up runbook, including the ordered re-enable steps and the end-to-end verification procedure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports prod hotfix 0e2ed84 byte-identically (the merge with copi-prod stays trivial) and adds the regression test that was missing there: Sonnet 5 thinks by default and max_tokens caps thinking + text together, so without the pin content[0] is a thinking block, the .text read raises, and every inbound reply classifies as unparseable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The welcome email (built by the app) reads ENABLE_INBOUND_EMAIL for its reply-vs-dashboard copy, so recreating only the worker leaves new signups with stale instructions. Also spell out why 'up -d' and not 'docker restart': env_file is resolved at container creation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ons rows Issue #29 rollout prerequisite. Eleven active labs have empty ORCID works, so the profile pipeline (the only publications ingest path) persisted nothing and the fail-closed authorship guard mutes their first-person paper claims. The script takes a human-curated agent_id -> PMID JSON mapping (PubMed author search cannot disambiguate names like Wu or Wilson), fetches records via fetch_pubmed_records, canonicalizes DOIs, and inserts idempotently; dry-run by default. Rows go live on the next ~30s roster sync, no restart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ahueb
marked this pull request as ready for review
August 14, 2026 15:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #31 (
email-fix): merge #31 (and #32) first — until then this PR's diff includes #31's commits. Only the threedeploy-followupscommits are new here. Targetscopi-prod.Executes the remediation-plan follow-up tracks from the 2026-08-14 deploy audit.
What this adds
fix(email): pinthinking={"type": "disabled"}on the inbound classify call — ports prod hotfix0e2ed84byte-identically into theemail-fixline (so the merge withcopi-prodstays trivial) and adds the regression test that was missing: without the pin, Sonnet 5's default thinking makescontent[0]a thinking block, the.textread raises, and every inbound reply classifies asunparseable. The test was written first and failed onemail-fixexactly as the latent bug predicted.docs(inbound): theENABLE_INBOUND_EMAILflip must recreate app AND worker — the welcome email (built by the app) reads the same flag for its reply-vs-dashboard copy; the runbook previously recreated only the worker. Also documents whyup -dand notdocker restart(env_fileresolves at container creation).ops(publications):scripts/backfill_publications.py— the issue-29 rollout prerequisite for the 11 muted labs (badran, cravatt, good, kern, lotz, maillie, pwu, saez, schultz, williamson, wilson). Root cause, measured on prod 2026-08-14: their ORCID works are empty (profile jobs completed, cravatt ×3 / lotz ×4 retries, still zero publications) and their profiles contain zero DOIs, so the DB ∪ profile grounding union is empty and re-running the pipeline cannot fix it. The script takes a human-curated{agent_id: [pmid, ...]}JSON (PubMed author search can't disambiguate names like Wu or Wilson), fetches records viafetch_pubmed_records, canonicalizes DOIs, inserts idempotently, and is dry-run by default. Inserted rows go live on the next ~30s roster sync — no restart.What this does NOT do
0024).Testing
All code written test-first: the pin test failed red on
email-fix, the backfill's five tests failed red against a stub. Full./scripts/ci.shgate green on this branch: 1681 passed / 120 skipped, 20 snapshots, migration round trip clean; also re-run by the pre-push hook on push.🤖 Generated with Claude Code