Skip to content

feat(webmd_doctor): add WebMD Doctor mirror (site 25, port 40024) - #95

Merged
Raibows merged 22 commits into
aiming-lab:mainfrom
evanz37:feat/webmd_doctor
Sep 13, 2026
Merged

feat(webmd_doctor): add WebMD Doctor mirror (site 25, port 40024)#95
Raibows merged 22 commits into
aiming-lab:mainfrom
evanz37:feat/webmd_doctor

Conversation

@evanz37

@evanz37 evanz37 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Mirrors https://doctor.webmd.com/ (the WebMD Care physician directory only; PR #62 mirrors www.webmd.com and is unrelated). Registered at index 24 / port 40024, bringing the registry to 25 sites. Ships 20 benchmark tasks in sites/webmd_doctor/tasks.jsonl. Every provider, practice, hospital, address, phone, NPI, review and account is synthetic; only the specialty / condition / insurer / city names are real vocabulary, and only site chrome (layout, icons, CSS) was harvested.

What's included

  • Flask app (app.py, SQLAlchemy models, CSRFProtect, Flask-Login) plus Jinja templates for home, results with filters and sorting, physician profiles (Basic and Enhanced layouts), hospital and practice hubs, specialty index / state / city pages, Choice Awards recipients, login / signup, saved providers, appointment requests and reviews.
  • Build-generated seed DB: .build-generated-seed marker and a Dockerfile RUN block in the same shape as walmart_careers / compass (PYTHONHASHSEED=0 python seed_data.py, rm -rf instance __pycache__); one seeded RNG, no wall-clock reads, hardcoded benchmark password hashes.
  • Rows per model: doctors 226 (202 within 40 mi of Newark, DE 19711 + 24 in Baltimore, MD), locations 348, reviews 1227, hospitals 12, practices 30, specialties 10, conditions 82, procedures 62, insurers 12 (28 plans), users 4 (alice.j / bob.c / carol.d / david.k @test.com, password TestPass123!), plus 4 saved providers, 1 appointment request and 1 pending user review pre-seeded.
  • 20 tasks with the contributor keys only (web_name, id, ques, web, upstream_url); no verifier, rubric or answer lives in any tracked file.
  • Registry edits (websyn_start.sh, control_server.py, Dockerfile EXPOSE 8101 40000-40024) plus a separate docs/tests bump commit (docs: bump site count to 25 (ports 40000-40024)).
  • HF assets PR https://huggingface.co/datasets/ChilleD/WebHarbor/discussions/68: static/images/ only (317 generated PNGs: 226 initials avatars + 91 video poster frames, md5-list digest e1b14d92); no DB in the tarball. .assets-revision is pinned to refs/pr/68 until it merges.

Verification

Check Result
Build, all 25 sites GET / 25 x 200
/health alive 25 / 25
POST /reset/webmd_doctor {"pid":293,"ready":true,"site":"webmd_doctor"}
In-image md5 instance/ == instance_seed/ 12b4c269cae06ccfd41e3d8116565e3a before reset, after reset, after docker restart
Seed rebuild determinism identical DB md5 under PYTHONHASHSEED=0 and =1; two --write-images runs give identical PNG md5 lists equal to the shipped digest e1b14d92
Fresh clone at 3c22d4a (git clone --depth 1 + per-site fetch_assets.sh + build.sh) 25/25 alive, reset and restart byte-identical
Rendered-HTML leak audit (home, results, specialty / state / city, hubs, recipients, account pages, 0-result state) 255 pages, 20/20 tasks clean
Tasks driven from the ques text with Playwright (click paths, no URLs) 20/20, expected row deltas for the stateful tasks, byte-identical after the following reset
Build-time distractor / breadth assertion (scripts_dev/, not shipped) 23 task checks + global rules pass (>= 6 results, target never rank 1, <= 50 % full matches, every filter value >= 20 in radius)
Cross-field consistency 31 invariants pass (city / zip / coordinates, board vs specialty, training timelines, one-city street names, hours, phones, awards)

Notes for the reviewer

  • No name search: upstream returns 0 results for a doctor's name, and so does the mirror. Every ques names its route through specialty + city, a hub page or the awards page.
  • T17: gate the verifier on the appointment_requests row (carol.d, Dr. Sarah Keller, "Riverfront Heart & Vascular - Wellness Center", New Patient, 2026-09-14 10:30 AM), not on the printed confirmation string. T6: the oldest review is on the profile's second review page, so ?rpage=2 is a valid navigation gate.
  • ./scripts/fetch_assets.sh in all-sites mode currently fails on every revision ("expected 25 site archives, found 26") because the dataset already holds drugs_com.tar.gz, which is not in GitHub main. Per-site fetch works: for s in sites/*/; do ./scripts/fetch_assets.sh $(basename $s); done.
  • sites/walmart_careers/tests immutable-pin assertion is red while .assets-revision points at refs/pr/68; it passes again once the pin is bumped to the HF merge sha (same history as feat(walmart_careers): add Walmart Careers mirror (site 23, port 40022) #86).
  • sites/rotten_tomatoes/tests registry assertion relaxed from the literal array tail to an ordered prefix (ted osu rotten_tomatoes compass walmart_careers); the literal-tail form breaks for any appended site.
  • Deviations: 226 doctors (above the 200 guideline; needed so every distance bucket and every filter value keep >= 20 rows at the default 40-mile radius); non-binary gender seeded at 21; Pillow initials avatars and gradient poster panels in place of photography (no real people); specialty-by-state and specialty-by-city hub pages are thin (1 to 14 doctors), mirroring upstream's per-city structure.
Tasks (20)
  • WebMD Doctor--0 — Search for Dermatologists near Newark, DE 19711 and open the profile of Dr. Jonah Dimitriou. Report the medical school Dr. Dimitriou graduated from and the graduation year.
  • WebMD Doctor--1 — Find Dr. Julian Zamora, a Cardiovascular Disease specialist whose primary office is in Wilmington, DE. Report the NPI number shown on the profile and the languages spoken.
  • WebMD Doctor--2 — Search for Family Medicine doctors near Newark, DE 19711 and open Dr. Ruth Thackeray's profile. Report the phone number listed for the primary office and that office's Saturday hours.
  • WebMD Doctor--3 — Open the profile of Dr. Mateo Alvarado, a Neurologist in West Chester, PA. Besides the primary office, the Locations section lists one other office. Report that office's name and street address.
  • WebMD Doctor--4 — Find Dr. Charles Villanueva, an Orthopedic Surgeon in Elkton, MD. From the Certifications, License, & Education section, report the board that certified them, the certification year, and the institution where they completed their residency.
  • WebMD Doctor--5 — Search for Gastroenterologists near Newark, DE 19711 and open Dr. Caroline Danforth's profile. Among the five most-treated conditions shown, exactly one is marked "More Than Most". Which condition is it, and which condition is listed first under "View Top 20"?
  • WebMD Doctor--6 — Open the profile of Dr. Fatima Jensen, a Psychiatrist in Media, PA, and read all of their reviews. What is the date shown on the oldest review, and what star rating did that reviewer give?
  • WebMD Doctor--7 — Find Dr. Lillian Acosta, an Obstetrics & Gynecology specialist in Salem, NJ. Which of the seven Patients' Perspective criteria received the most needs-improvement votes, and what average wait time is shown on the profile?
  • WebMD Doctor--8 — Log in as alice.j@test.com (password: TestPass123!) and open Saved Providers. Exactly one of your saved providers is a Dermatologist. Open that profile and report the institution where they completed their residency, then remove that provider from your saved list.
  • WebMD Doctor--9 — Search for Dermatologists near Newark, DE 19711 who are female, accept new patients and accept Blue Cross Blue Shield. Among the results, open the profile of the doctor with fewer than 5 years of experience and report their medical school and the year of their board certification.
  • WebMD Doctor--10 — Find Psychiatrists near Newark, DE 19711 who accept Medicaid and have a rating of 4 stars or higher. Open the profile of the one who offers virtual visits and report the average wait time and the residency institution listed.
  • WebMD Doctor--11 — Set the distance to 10 miles from Newark, DE 19711, search for Family Medicine doctors and sort by Number of Ratings. Open the profile of the doctor with the second-highest number of ratings and report their NPI number and the institution where they completed their residency.
  • WebMD Doctor--12 — From the Find Providers by Specialty menu open Cardiovascular Disease, then Pennsylvania, then West Chester. Filter to doctors rated 4 stars or higher. Open the profile of the only male doctor in that list and report his fellowship institution and the year he completed it.
  • WebMD Doctor--13 — Among Dermatologists in Wilmington, DE, Dr. Gregory Greenwood and Dr. Emerson Huang both accept Blue Cross Blue Shield. Which of the two graduated from medical school earlier? Report that doctor's name and graduation year.
  • WebMD Doctor--14 — Open the hospital page for Christina Creek Medical Center (Find a Facility > Hospitals > Delaware). Two of its listed physicians are Psychiatrists; open both profiles. Which one was board certified more recently? Report that doctor's name and the certification year.
  • WebMD Doctor--15 — From the header, open Award Winning Hospitals, then view the WebMD Patient's Choice recipients. Find the recipient who practices Orthopedic Surgery in Media, PA, open their profile, then open the practice page linked from their primary office. Report the practice's website address and its Saturday hours.
  • WebMD Doctor--16 — Log in with the demo account (email: bob.c@test.com, password: TestPass123!), search for Pediatricians near Newark, DE 19711, open the profile of Dr. Anita Castellano and save the provider. Then open Saved Providers and confirm Dr. Castellano appears there.
  • WebMD Doctor--17 — Log in as carol.d@test.com (password: TestPass123!). Open the profile of Dr. Sarah Keller, a Cardiovascular Disease specialist in Newark, DE, and request an appointment as a New Patient at the Riverfront Heart & Vascular - Wellness Center office on Mon, Sep 14 at 10:30 AM. Report the confirmation reference shown after submitting.
  • WebMD Doctor--18 — Log in as david.k@test.com (password: TestPass123!). Find Dr. Tariq Huang, a Dermatologist in Elkton, MD, and leave a 4-star review with the text "Short wait and a clear explanation of my treatment options." Confirm the profile now shows your review as Pending review.
  • WebMD Doctor--19 — Register a new account with an email and password of your choice. Then search for Neurologists near Newark, DE 19711 who offer virtual visits, open the profile of Dr. Monica Carrington, and save the provider. Report the NPI number shown on Dr. Carrington's profile.

Screenshots

Reference captures from doctor.webmd.com (left) vs the mirror on port 40024 (right), both at 1440 px.

Page Upstream Mirror
Home 01_home 01_home
Results (Dermatologist filtered with new patients) 02b_results_viewport_filter_newpatients 02b_results_viewport_filter_newpatients
Profile (Enhanced) 03_doctor_enhanced_hollander 03_doctor_enhanced_hollander
Hospital detail 07_hospitals 07_hospitals

evanz37 and others added 17 commits September 10, 2026 14:57
Offline Flask mirror of https://doctor.webmd.com/ ("WebMD Care"): deterministic
synthetic directory of 224 doctors across 10 specialties and 8 cities around
Newark, DE 19711, with hospitals, group practices, Choice Awards, real auth,
saved providers, appointment requests and pending reviews.

- Build-generated seed (.build-generated-seed): the Dockerfile regenerates
  instance_seed/webmd_doctor.db plus 224 Pillow initials avatars and 90 video
  poster frames from seed_data.py; no Hugging Face assets, .assets-revision
  untouched.
- One seeded RNG (20260910), literal reference date 2026-09-10, hardcoded
  scrypt hashes for the four benchmark users, whole-function seed gates,
  seed_metadata version + expected-count validation; byte-identical after
  /reset and docker restart.
- Deterministic longest-match query parser (specialty / condition / procedure
  / insurer / gender / virtual), conjunctive filters, Best Match / Distance /
  Average Rating / Number of Ratings ranking, numbered pagination.
- Registered as index 24 in websyn_start.sh and control_server.py; Dockerfile
  EXPOSE raised to 40024, header comment bumped to 25 sites.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
Contributor-side task definitions only (web_name, id, ques, web, upstream_url):
9 lookups (0-7, 15), 4 multi-constraint (9-12), 2 compares (13-14), 5 stateful
(8, 16-19). Every ques routes through a specialty + city, a hub page or an
awards class before naming a doctor (name search is a non-goal, as upstream);
credentials are embedded for login tasks; no answers, verifiers or rubrics.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
- seed: two extra West Chester cardiology slots appended after the grid
  (task 12 now has 8 doctors on the city page, 6 rated 4+ with exactly one
  male); deterministic language top-up for physicians covering three
  offices (task 1 target speaks 3 languages); EXPECTED_COUNTS refreshed
  (226 doctors, 202 in radius)
- pagination: disabled prev/next rendered as spans instead of live links
  to an empty page
- filter bar: empty and default params are dropped from auto-submitted URLs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
Section-by-section pass against scraped_data/reference at 1440 px:
- profile: booking widget overhangs the hero into the rail, tabs span the
  main column only, colleagues tile grid / hospital affiliation card in the
  stack, Basic rail below the tabs, NPI folded into the certification card,
  nearby + transparency panels full width, star picker, review controls,
  check-circle perspective icons, two-column bulleted top-20 lists
- home: icon tile row for popular specialties, filled uppercase View Profile
  buttons, upstream heading sizes
- filter pills in dark text with navy active state; typeahead no longer
  clipped by the search bar, prefix highlighting
- hospitals / group practices hubs rebuilt as landing pages (state chips,
  name search, top-4 cards, search-all button, care-type chips); state lists
  drop the sliders button and gain a name filter
- hospital / practice details: specialty select, map-on-top locations,
  ratings card, text flag lines
- awards page art panels + larger headings; recipients grouped by state
  with a state filter; guidelines as a centred numbered card; sign-up terms
  line and mm/dd/yyyy date field; login/state/city copy per upstream
- landing "Highest Rated" strip now the 25-mile ring so no task target
  is showcased there

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
README row counts match EXPECTED_COUNTS of the final seed (226 doctors,
348 locations, 91 posters). Freezer byte-identical across two runs and
PYTHONHASHSEED=0/1; in-image instance == instance_seed before and after
docker restart.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
Shared docs (README, AGENTS, CONTRIBUTING, CLAUDE, agent_demo/README) and the
registry assertions in the walmart_careers and rotten_tomatoes tests now
cover the appended webmd_doctor slot (index 24, port 40024).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTvmqJcShyy3v3KfEPxMPe
…hanged)

Switch to the walmart_careers asset shape: the Docker RUN block still
regenerates instance_seed/webmd_doctor.db deterministically from
seed_data.py, but the 226 initials avatars and 91 video posters under
static/images/ now arrive through the pinned Hugging Face tarball
(.requires-images) instead of being written at image-build time.

- seed_data.py: image generation moves behind `--write-images` /
  build_images() for the local freezer only; PNGs are saved with fixed
  compression and no ancillary chunks (IHDR/IDAT/IEND only), identical
  under PYTHONHASHSEED=0 and 1 (317 files, list digest e1b14d92).
- Dockerfile: assert the fetched avatars/posters are present, then build
  the DB only. In-image seed md5 unchanged (364c15c35fff64f0c25ef62ac0bb64fc).
- Markers: .build-generated-seed reworded, .requires-images added so
  check_assets.sh / fetch_assets.sh treat the site like walmart_careers;
  .assetpaths globs already cover static/images/. scripts/ untouched.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qq2aHgZwwYWAS7cCepWKiu
…iscussions/68; temporary)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qq2aHgZwwYWAS7cCepWKiu
…ed-search notice, city-page sections, profile scale

- seed: Medicare/Medicaid Insurance-card rows now follow the primary office flags
  (deterministic post-pass, no RNG; doctor_insurances 2274 -> 2208; in-image md5
  8bf76b696f8e49ba59848a51bf47f24a; avatars/posters byte-identical, no HF change)
- results: unresolved search text keeps the "ignore" semantics but the heading reads
  "All Providers" and a mirror-specific notice names the unmatched input
- specialty city page: description blurb, FAQ and related-specialties grid below the
  list (FAQ removed from the landing page), Distance control + distance on cards
- profile: type scale matched to upstream, sticky tabs + compact provider bar on
  scroll, "View Less" toggle, certification-card dividers, grey nearby/transparency panels
- chrome: blue filter-pill labels, banner dismiss, larger stars, inline View Profile on
  regular cards, grey current page, footer badge cluster + privacy toggle, hub hero /
  Sort By / info-block styling, extra hospital & practice FAQ entries, recipients show
  only the active award chip, awards sub-nav chevrons, guidelines page without search
  row/footer, styled booking select, auth overlay dims the header

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGf8U2ULBCevvaYfbBBuVt
…ve/hours consistency

- results: a search term in which no token resolves to a vocabulary term now renders
  the upstream-style empty state (heading with the typed term, "0 Results", notice,
  filter bar kept, browse-by-specialty link); partially matched queries still ignore
  the unmatched tokens
- seed (RNG-free post-passes, row counts unchanged): every primary office gets its own
  direct line (no office shares its practice's main number, all phones unique); a
  practice and every office located at it share one schedule (the longest weekly
  office schedule); Patients' Perspective votes per criterion are bounded by the
  ratings count and floored at the visible review marks
- in-image md5 86cd0e7a8669151405f5c6b541fb91d0; avatars/posters byte-identical
  (no HF change); no task id shifted; T2 phone + Saturday hours and T15 practice
  Saturday hours change value, every other answer is unchanged

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGf8U2ULBCevvaYfbBBuVt
…ecialty-consistent condition lists)

- T15 ques names the real controls (Award Winning Hospitals > WebMD Patient's Choice);
  T4 ques uses the on-page heading "Certifications, License, & Education" (upstream label).
- Booking reference is eight base-32 chars packed injectively from (row id, doctor id,
  office index, day, slot) with fixed constants - two bookings can never share a reference,
  and a first booking after a reset is no longer the same string for every doctor.
- Most-Treated / Top-20 conditions and procedures draw only from the doctor's own specialty
  plus its SECONDARY_CHOICES pool (RELATED_SPECIALTIES removed); under-5-years bucket is
  2-4 years; five training-institution names that doubled as hub hospitals replaced.
- "1 Year Experience" pluralised on cards, profile and hub mini-cards.
- <=720 px CSS: header nav, review controls, footer badges wrap; promo panel and location
  blocks stay inside the viewport.
- Re-frozen seed (PYTHONHASHSEED 0 and 1 identical; images byte-identical to the shipped
  HF tarball); EXPECTED_COUNTS + README row counts updated; re-picked targets for tasks
  4, 11 and 14 after the RNG reshuffle.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8Mt6G2qwvqncZiKyVC5pR
…aining timelines, per-city streets)

- SECONDARY_CONDITIONS / SECONDARY_PROCEDURES: curated per-specialty pools (9-10
  conditions, 6-8 procedures incl. the primaries); Top-20 lists draw only from the
  primaries + the doctor's own specialty pool (conditions 40 -> 82, procedures 30 -> 62)
- TRAINING + training_plan(): residency length per specialty after the MD, fellowship
  (Cardio/GI 3 y required, optional 1-3 y elsewhere), board = end of training + 0/1,
  years of experience = 2026 - board year; subspecialty board only after a fellowship
- SECONDARY_STREETS per city; every street name belongs to exactly one city
  (Deer Park Digestive Health moves from E Main St to Library Ave)
- _bound_review_stars(): review means stay within 1.0 of the profile average (RNG-free)
- doctor stream unchanged (names, slugs, NPIs, cities, ratings, years, offices, phones);
  EXPECTED_COUNTS + README rows updated; images unchanged (list digest e1b14d92)
- tasks: T4 -> Dr. Charles Villanueva, T11 reports NPI + residency institution,
  T13 pair -> Greenwood vs Dr. Emerson Huang, T14 compares the two Psychiatrists

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YKXfhRp7xMmrgixw5Ccohe
…next, no __pycache__ in the image)

- logout(): session.clear() before logout_user()
- safe_next(): reject leading/trailing whitespace (raw and decoded), strip fragments explicitly
- Dockerfile: rm -rf __pycache__ after the build-time seed run

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YKXfhRp7xMmrgixw5Ccohe
20 verifiers (verify_0..19.py) plus verify_lib.py and ground_truth.py under
sites/webmd_doctor/verify/, in the walmart_careers contract: argparse CLI, zero
LLM calls, fail-closed snapshot discovery (run_dir or docker cp), pinned
28-table schema hash + seed marker + counts, catalog tables row-identical,
read-only tasks require the four runtime tables unchanged, stateful tasks
enforce exact row deltas (saved provider removed/added, appointment row
matched by user/doctor/office/slot with the answer's reference taken from the
new row, review row by rating/text/status, registration by the e-mail typed on
/signup). Route gates follow the task text (results query params incl. the
Newark rule and combined filters, specialty menu path, hospital and awards
pages, second review page); comparison tasks gate on both profiles.
ground_truth.py re-derives every target from the initial snapshot and fails
closed on disagreement.

verify/tests: 372 subprocess tests (genuine, shortcut, wrong answer, state
mismatch, over-action, read-only write, packaging and contract negatives) plus
25 library tests; no docker, no API key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017yDUT66ukMXuASicT7yd6H
tasks.jsonl now carries the seven reviewer keys. The five contributor keys are
byte-identical to the submitted rows; verifier_path points at
sites/webmd_doctor/verify/verify_N.py; judge_rubric opens with the shared
scoring preamble (step list authoritative, unverified is not contradicted, the
verifier owns values and the database) followed by rule-only checkpoints. A
dev-side validator asserts the key shape, ids 0-19, verifier files, byte
identity of the contributor keys, and that no rubric contains a ground-truth
token or a doctor, hospital, practice or city name absent from its own ques.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017yDUT66ukMXuASicT7yd6H
LLM-free matrix (116 scripted Playwright runs graded through eval_judge.py
--verifier True) showed the five stateful verifiers ran row-identity /
exactly_one_* checks before <table>_exact_delta, so a run that wrote nothing
failed under the wrong name. Exact delta now comes first; the redundant
exactly_one_* count checks are gone; unit tests updated (397 green).
README gains the validation summary (counts only).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WM3oHCY7Vi612v5c6JSfay
Rules-only rewrite from 40 dual-graded agent runs (gpt-5.4-nano and
gpt-5.4-mini, judge on nano): the shared preamble now states the
screenshot window, that the final answer text is the evidence, and that
an uncontradicted checkpoint set means success; "must be visible"
checkpoints (8-11, 16-19) and "as shown in the section" facts (0, 4, 5,
9) are phrased against the step list. validate_tasks: contributor keys
byte-identical, no ground-truth token. README records the real-run
counts (verifier 1/20 nano, 7/20 mini; 0 verifier defects; 0 judge false
PASSes; agreement 19/20 and 15/20).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UB1bWfYQV98foy5iCKxP1H
@evanz37

evanz37 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Track B — grading contract for webmd_doctor

Following #86's pattern, this adds the reviewer-side contract to the same PR: 5657fdd verify_lib + 20 LLM-free verifiers + 397 unit tests, 82af7dd verifier_path/judge_rubric backfill (rules only), df19ae7 hardening from a 116-cell LLM-free validation matrix, e7862ed rubric polish from 40 real agent runs.

Reading the pass rates. Verifier PASS is 1/20 on gpt-5.4-nano and 7/20 on gpt-5.4-mini. Every task is solvable: the scripted Playwright walk from the ques text is 20/20 in the matrix, and every real-run FAIL adjudicates to the agent — a name search that returns 0 results (as on upstream) followed by a give-up, max_steps (15) exhausted on the long profile pages, a wrong same-surname doctor, or a mandated filter/sort/awards path skipped with an otherwise correct answer. No verifier defect surfaced from the real runs.

The number that matters: 0 judge false PASSes before and after polish. All 7 verifier↔judge divergences were judge false FAILs (evidence-window or self-contradictory nano-judge rationales); a gpt-5.4-mini judge agrees with the verifier on 5 of the 6 residual ones.

Observation for maintainers, not actioned: the 0-result name search is the single largest source of agent give-ups. It is faithful to upstream and the empty state already points to specialty search; if you would prefer a more explicit hint there, it is a template-only change and I can push it separately so it does not muddy these runs.

Total spend $1.17 (cap $2.50). Details below.

Full report

Verdict on the site (A): unchanged from the PR description's Verification table: 25/25 alive, POST /reset/webmd_doctorready:true, in-image md5 pair 12b4c269cae06ccfd41e3d8116565e3a before the runs, after every reset between runs and after the last run.

Contract (B): sites/webmd_doctor/verify/verify_lib.py + verify_0.py … verify_19.py + ground_truth.py, deterministic and LLM-free. What they guard: package validity (exact task id, nonempty answer, agent_done, same-origin HTTP URLs, decodable PNGs), snapshot validity (28-table schema hash, seed marker, frozen counts, 24 catalog tables row-identical, target re-derived from the initial snapshot and cross-checked against the hardcoded constants, fail-closed with infra_error when a snapshot is missing), navigation gates (the named profile and its tab aliases; the specialty results page under the "near Newark, DE 19711" rule; every mandated filter on one results URL for 9–11; the menu / hub / awards paths in order for 12, 14, 15; the second review page for 6; ordered workflows for 16), negation-aware answer matchers (institutions, years as standalone tokens, 10-digit NPI, phone digits, both clock endpoints, wait minutes, star + date, first + last name, website with domain ending), and exact after-state deltas (<table>_exact_delta first, then row identity — owner, doctor, office, slot, stars, text, signup e-mail — then <table>_unchanged on the other runtime tables; read-only tasks require all four runtime tables row-identical). 397 unit tests, no docker or key needed.

Validation (C):

  • LLM-free matrix through eval_judge.py --verifier True: 116 cells — no-op 20/20 FAIL, scripted genuine 20/20 PASS, shortcut 20/20 FAIL on the gate, wrong-answer 18/18 FAIL on the answer check, state-skipped 5/5, over-action 5/5, same-table duplicate 5/5, read-only write 15/15, mandated route bypassed 8/8 — every cell on the intended check.

  • Real agent runs: agent_demo/agent.py unchanged (default max_steps 15), one attempt per task, gpt-5.4-nano and gpt-5.4-mini (40 runs), every run graded by the verifier and by the rubric-driven LLM judge on gpt-5.4-nano. Verifier PASS: nano 1/20, mini 7/20. Agent self-reports: nano claimed success on 4 runs (verifier confirmed 1), mini on 8 (verifier confirmed 7). No verifier defect surfaced: every FAIL was adjudicated to the agent (name searches that return 0 results and a give-up, max_steps without an answer, a wrong doctor, a mandated filter / sort / awards path skipped) and the seven PASSes hold exact values and exact row deltas (mini 16: +1 saved_providers row for bob on the named doctor; every other run left the runtime tables untouched except mini 19, which registered a user and ran out of steps before saving).

  • Verifier↔judge agreement: nano 19/20 → 19/20, mini 14/20 → 15/20 after the rubric polish. Judge false PASSes: 0 before, 0 after (the judge's only PASSes, mini 16 and mini 5, are verifier PASSes). All 7 pre-polish divergences and the 6 residual ones are verifier PASS / judge FAIL and adjudicate to the verifier. A supplementary re-grade of the 6 residual runs with a gpt-5.4-mini judge agrees with the verifier on 5 (only nano 4 stays FAIL, on a FAQ line the judge read as a contradiction) — the residue is the judge model and the evidence window, not the rubrics.

Agreement table

Pass — agent nano

task steps termination agent self-report verifier first failing check judge (original rubric) judge (polished rubric) agreement
0 3 agent_done gave up FAIL visited_results_dermatologist_search FAIL (0.95) FAIL (0.99) agree
1 3 agent_done gave up FAIL visited_profile_julian-zamora-d412b77d FAIL (0.97) FAIL (0.96) agree
2 3 agent_done gave up FAIL visited_results_family_medicine_search FAIL (0.95) FAIL (0.90) agree
3 4 agent_done gave up FAIL visited_profile_mateo-alvarado-f727bd61 FAIL (0.98) FAIL (0.99) agree
4 11 agent_done success PASS FAIL (0.83) FAIL (0.86) diverge
5 3 agent_done gave up FAIL visited_profile_caroline-danforth-7093653d FAIL (0.93) FAIL (0.98) agree
6 4 agent_done gave up FAIL visited_profile_fatima-jensen-e5a26d53 FAIL (0.95) FAIL (0.95) agree
7 5 agent_done gave up FAIL visited_profile_lillian-acosta-4a89e15e FAIL (0.98) FAIL (0.98) agree
8 15 max_steps FAIL final_answer_nonempty FAIL (0.95) FAIL (0.97) agree
9 13 agent_done success FAIL visited_filtered_results FAIL (0.78) FAIL (0.78) agree
10 8 agent_done gave up FAIL visited_medicaid_rated_results FAIL (0.75) FAIL (0.95) agree
11 9 agent_done success FAIL visited_results_10mi_sorted_by_ratings FAIL (0.86) FAIL (0.74) agree
12 7 agent_done success FAIL visited_state_page FAIL (0.92) FAIL (0.85) agree
13 15 max_steps FAIL final_answer_nonempty FAIL (0.95) FAIL (0.99) agree
14 15 max_steps FAIL final_answer_nonempty FAIL (0.96) FAIL (0.95) agree
15 15 max_steps FAIL final_answer_nonempty FAIL (0.98) FAIL (0.92) agree
16 15 max_steps FAIL final_answer_nonempty FAIL (0.97) FAIL (0.95) agree
17 15 max_steps FAIL final_answer_nonempty FAIL (0.90) FAIL (0.97) agree
18 15 max_steps FAIL final_answer_nonempty FAIL (0.98) FAIL (0.98) agree
19 3 parse_error FAIL final_answer_nonempty FAIL (0.99) FAIL (0.98) agree

Notes for maintainers: scripts_dev/ and runs/ stay out of git; the container was reset after the last run and the md5 pair is unchanged. Rubrics contain rules only; ground truth lives in verify/ only.

Seed and data integrity:
- All 226 doctor NPIs are now checksum-valid (CMS Luhn rule over 80840+9
  digits) and verified unassigned against the NPPES registry (August 2026
  monthly + weeklies through 2026-09-06 + deactivated report); the verified
  list is embedded in seed_data.py and re-checked on every build. RNG stream
  preserved, so slugs, images and education rows are unchanged.
- Alice holds six saved providers (seven rows total) so the saved-provider
  removal task has real breadth; _seed_is_complete hardened to all immutable
  counts + foreign-key check.
- Booking confirmation references use range-checked deterministic packing.

App hardening:
- SECRET_KEY from env or per-process random; session cookie hardening; login
  timing equalization via a dummy scrypt hash; bounded integer/argument
  parsing; safe_next rejects control characters; IntegrityError and duplicate
  guards on save/booking/review; booking enforces each office's new-patient
  policy.
- Specialty hub pages count primary-OR-secondary specialty (matching results
  search); hub header totals reflect the filtered population; /_health added
  with legacy /health alias; inject_globals degrades safely so error pages
  render even when the database fails.

Verifier hardening (447 tests):
- Clause-level negation matching, first-value parameter binding, signed-in
  checks bound to the login email+password, minimum screenshot 320x240,
  booking reference recomputation, per-task ordering/role/comparison checks,
  and an EXPECTED_FACTS cross-check in ground truth covering all 20 tasks.

UI, accessibility, responsive:
- Sitewide mirror notice and synthetic-data disclaimers; heading-order fixes;
  skip link; focus-visible outlines everywhere; fieldset/legend radio groups;
  role=alert error lists; aria-controls/expanded on disclosures; contrast
  fixes for muted/red tokens, booking steps and map placeholder; 320px
  overflow fixes (scrollable header nav row, single-column mini grids,
  wrapping footer badges); no-JS progressive enhancement (menus/popovers
  render as static lists, noscript submit fallbacks).

Asset governance and repo integration:
- generated_asset_inventory.json (317 PNGs with sha256) + site-local
  check_generated_assets.py, gated in the Dockerfile and check_assets.sh.
- fetch_assets.sh all-sites mode now enumerates local sites and tolerates
  extra archives on HF (drugs_com, fedex).
- .assets-revision pinned to merged HF main sha ad6f424f72cada9e6f5c09a580
  93d0ceeab9c52b (HF assets PR merged); walmart pin assertion updated.
- Root README site count 23 -> 25; site README documents the NPI policy,
  known deviations and benchmark accounts.

Tests: new sites/webmd_doctor/tests (seed quality + determinism + Luhn audit,
task breadth, rendered-page sweep, generated assets); verify suite 447 passed;
cross-site battery green (compass 94, RT 56, walmart 51, ikea 119, walmart
verify 246); 20/20 Playwright E2E tasks pass against the packaged image.
… overflow and saved-link contrast; add answer-leak sweep

Review-round 2 (checklist re-audit):
- Booking widget no longer hardcodes New Patient as checked: task 17
  requires the runner to choose the patient type, so every form field the
  tasks ask for now arrives empty (README documents the policy).
- Enhanced-profile layout forced horizontal overflow below 1181px (absolute
  384px sidebar plus 424px tabs/hero offsets; 791px document width measured
  at a 390px viewport): an appended max-width:1180px block makes the sidebar
  a static full-width card and drops the offsets.
- Saved-providers card links used #7d92ff (2.85:1 on white, below the 3:1
  large-text floor); switched to var(--blue) (5.31:1).
- New tests/test_answer_leaks.py: derives all 20 ground-truth answers from
  the frozen seed and asserts unique answer facts (NPIs, phones, institution
  names, office addresses, websites, review dates, years) appear on none of
  the ~100 pre-discovery chrome surfaces, are absent from templates/CSS/JS,
  and that no booking-reference pattern appears on chrome. Shared-vocabulary
  classes (condition browse tiles, criterion callout chips, small integers,
  hour strings) are covered by entity-bound tests: the T5 target's cards
  never render her condition names or tier wording, the T7 target's own card
  chip never reads the answer criterion, and the card template renders no
  conditions at all. Mutation-tested: injecting a ground-truth school name
  and a TODO marker into index.html fails both guards; corrupting one seeded
  NPI fails the Luhn audit; an extra image file fails the asset checker.
- test_seed_quality and verify/tests/_support surface subprocess stderr in
  assertion messages on build failure.
- walmart shared-documentation test derives the expected 40000-400NN port
  range from the control_server.py SITES registry via ast instead of
  hardcoding the site count.
…inventory test

The T5 entity-bound test asserted that none of the target's most-treated or
Top-20 condition names appear on her pre-profile cards. That is stricter than
the leak model: enhanced cards quote patient reviews, and a quote may mention
a non-answer condition in prose (the target's snippet mentions Irritable Bowel
Syndrome while the task answers are the More-Than-Most condition and the first
Top-20 condition). The prose mention is non-discriminative because the tier
data that identifies the answer lives only on the profile, and the verifier
binds scoring to a required profile visit. The old assertion also depended on
regex extraction from the rendered profile, which made it sensitive to
whitespace changes across Jinja2 versions (3.1.4 vs 3.1.6 gave different
extracted name sets from byte-identical seeds). The test now derives the two
answer facts from ground truth, pins them fail-closed, and asserts they and
the tier wording never appear on the target's own cards.

Also restores test_tarball_contains_only_declared_generated_images, which was
lost to a truncated file write before the previous commit. It asserts the
published tarball at the pinned revision contains exactly the 317 declared
generated images (no seed database, no ground-truth data, no undeclared
members in either direction). Both tests are mutation-verified: injecting the
answer fact into the card template and adding a fake inventory entry each
fail the respective test, and reverting passes.
The enhanced-profile bio lead-in ('Meet Dr. X:') and .meet accent rendered
#d9541e on white, which measures 4.01:1 - below the 4.5:1 WCAG AA threshold
for normal-size text. The replacement #c24a19 keeps the same hue (17 deg)
and measures 4.90:1 on #fff and 4.70:1 on #fafafa. Found by the pixel- and
computed-style-based contrast sweep over all routes at 1440/768/390/320.
…WebMD Doctor branch

Registry: append `webmd_doctor` after `fedex`, so FedEx keeps index 24 / port
40024 and WebMD Doctor moves to index 25 / port 40025 (26 sites, 40000-40025).

Conflict resolutions:
- `.assets-revision`: keep `ad6f424f` (current HF dataset main, which carries
  `webmd_doctor.tar.gz` and the byte-identical `fedex.tar.gz` blob) instead of
  main's FedEx-only pin `68dcbf2c`.
- `scripts/fetch_assets.sh`: take main's registry-scoped fetch implementation.
- `Dockerfile`: keep both site blocks (FedEx asset gate + WebMD generated-asset
  gate and source-built seeds) and widen `EXPOSE` to 40000-40025.
- `README.md`, `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, `agent_demo/README.md`,
  `.claude/skills/*`: 26 mirrors, port range 40000-40025, alt ports 41000-41025.
- `sites/walmart_careers/tests/test_integration.py`,
  `sites/rotten_tomatoes/tests/test_environment_quality.py`: take main's
  registry-derived assertions and extend them for `webmd_doctor` (40025) and its
  generated-asset gate.

Follow-on work required by the port move:
- `sites/webmd_doctor/tasks.jsonl`, `README.md`, `verify/verify_lib.py` comment,
  `tests/test_task_breadth.py` (now derives the port from the registry),
  `verify/tests/test_verify_lib.py`: port 40025.
- new `sites/webmd_doctor/tests/conftest.py`: rebuild the build-generated seed
  when it is absent and install the runtime copy the app binds, so the suite runs
  after `scripts/fetch_assets.sh` alone (the Walmart mirror's convention). In that
  state the suite previously failed 7 tests.

Verified after the merge: webmd site tests 19, webmd verify 447, walmart 51,
fedex 91 (+1341 subtests), rotten_tomatoes 56 (+4129 subtests), compass 94.
@Raibows

Raibows commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Thanks for your contribution! @evanz37 @alidezhihui

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants