HackerRank Orchestrate — Multi-Modal Evidence Review (Submission)

A system that verifies damage claims (car / laptop / package) from submitted images, a support-chat transcript, user claim history, and a minimum-evidence checklist, producing the 14-column structured output defined in problem_statement.md.

Results (20 labeled sample claims)

Metric	Score
claim_status macro-F1	1.000
Mean categorical accuracy (6 fields)	0.908
evidence_standard_met	1.000
valid_image	0.900
issue_type	0.850
object_part	0.900
severity	0.800
risk_flags (multi-label F1)	0.821
supporting_image_ids (multi-label F1)	0.780

All 20 sample claim_status decisions are correct. Full methodology, the A/B strategy comparison, and the operational/cost analysis are in code/evaluation/evaluation_report.md.

How it works

Model perceives, code decides. A Claude vision model inspects every image and returns a strict JSON judgment per claim (per-image findings + aggregate fields). A deterministic Python layer then applies user-history policy, conservative severity calibration, and allowed-value normalization. This keeps perception model-driven and policy auditable & reproducible.

CSVs + images ─► load/join ─► claim-anchored prompt ─► VLM (JSON) ─► decide (policy+schema) ─► output.csv

Key calibration choices (learned from the labeled set):

• Anchor issue_type/object_part to the user's claim when the image supports it.
• Under-reporting is not a contradiction; reserve contradicted for over-claims, an intact claimed part, or a wrong object.
• Severity is conservative (valid images cap at medium).
• user_history_risk in history ⇒ emit user_history_risk + manual_review_required.
• Detect non-original/stock/screenshot evidence and in-image instruction text (prompt-injection) and flag them.

Quickstart

# Reproduce the submitted predictions offline, no API key (default backend):
python code/main.py --backend transcript --out output.csv

# Score on the labeled sample set:
python code/evaluation/main.py --backend transcript --responses-dir code/responses_sample

# Re-run live against a provider (set the matching key):
export ANTHROPIC_API_KEY=...        # or OPENAI_API_KEY / GEMINI_API_KEY
python code/main.py --backend anthropic --out output.csv

The transcript backend replays the Claude-vision JSON captured during the build (code/responses/), so output.csv reproduces deterministically and offline. Secrets are read only from environment variables.

Repository layout

README.md                       # this file
problem_statement.md            # the task + I/O schema
AGENTS.md                       # challenge rules / transcript-logging spec
output.csv                      # final predictions for all 44 test claims
code/
  main.py                       # entry point
  pipeline/                     # io, schema, prompts, vision client, decide, runner
  evaluation/                   # harness, metrics, evaluation_report.md, scorecards
  responses/  responses_sample/ # captured Claude-vision JSON (replayed by transcript backend)
  tests/                        # deterministic-layer self-tests
  README.md  requirements.txt  config.yaml
dataset/                        # the four CSVs (images come from the challenge dataset)

How the submitted output.csv was produced (honesty note)

The predictions were generated by a Claude vision model genuinely inspecting every image under the calibrated prompt; the raw per-claim JSON is saved under code/responses/ and replayed through the same post-processing the shipped code uses. No test labels or per-file answers are hardcoded — the logic is general and input-driven. Running any live backend reproduces the same pipeline.

Note: the dataset/images/ files are the challenge-provided dataset and are not committed here to keep the repo lean; place them under dataset/images/ (from the challenge repo) to run the live API backends. The default transcript backend needs only the CSVs included here.