Amazon Review AI Detector

A niche AI-text detector for Amazon-style product reviews — not another generic essay classifier.

Interview one-liner: false positives (flagging a real customer as a bot) are the costly error, so the product returns a calibrated probability and a high-precision threshold, not a "FAKE" stamp.

Why this niche

General AI-text detectors are a crowded, untrustworthy space. Amazon reviews are a different distribution:

Essays / news	Amazon reviews
Long, formal, sourced	Short, informal, first-person
Topic-agnostic "AI-ness"	Must mention this product or it is pasteable spam
False positive embarrasses a student	False positive accuses a real buyer

Sellers already spin up LLM reviews at scale. A detector that only works on GPT-4 essays will not notice that — and a detector that flags every well-written 5-star review will destroy trust. The scope is the whole product: classifier + /analyze API + Manifest V3 extension that reads the DOM Amazon already rendered in the user's browser (no scraping).

Architecture

Amazon Reviews 2023 (human class)
        +
Ollama llama3.2 / qwen2.5  (AI class, matched product + stars)
        │
        ▼
Engineered features ──┐
TF-IDF word/char  ────┼─► calibrated logistic ─┐
DistilBERT (optional)─┘                         ├─► P(AI) + signals
                                                │
FastAPI POST /analyze          (single review)
        POST /analyze-listing  (trust, clusters, posting bursts)
        GET  /listing-history  (percent_ai snapshots over time)
        │
Chrome MV3: site adapter → listing card + hover report
        adapters: amazon, trustpilot, etsy, sephora, walmart

What a shopper actually sees: a collapsed card — 14% likely AI · 8 share a template — expandable to cluster count, posting-burst warning, a percent_ai sparkline, and a “show flagged only” filter. Per-review badges are secondary. Site DOM is isolated behind adapters (extension/adapters/).

Repo

data/          download + AI generation + splits
model/         features, train, DistilBERT, evasion
api/           FastAPI  /analyze  /health
extension/     Manifest V3
writeup/       false-positive tradeoff + evasion findings

Metrics (this checkout)

Trained on 400 reviews (200 human Amazon Reviews 2023 All_Beauty + 200 Ollama-matched AI), split 280/60/60. Positive class = AI. Threshold frozen at 0.72 (the product default) after val precision was already 1.0 at that cut.

Held-out test (n=60):

	Precision (AI)	Recall (AI)	FPR (human)
Hybrid (sklearn + DistilBERT, blend 0.20)	1.00	0.90	0.00

Val DistilBERT accuracy after 2 CPU epochs: 0.95. In-distribution local-LLM copy is easy. That is not the headline — evasion is:

Recall at the 0.72 production threshold (30 AI test reviews):

Attack	Recall	Drop
Identity (raw LLM)	0.90	—
Typos	0.83	−0.07
Humanize aside	0.83	−0.07
Splice one human sentence	0.77	−0.13
Ollama paraphrase	0.53	−0.37

See writeup/evasion.md. We catch stock product copy; a seller who rewrites it gets through more often than not.

Setup

Python 3.12, Node not required (extension is plain JS). Ollama is used to generate the AI class and to paraphrase for evasion tests.

python -m pip install -r requirements.txt
ollama pull llama3.2:1b
ollama pull llama3.2:3b
ollama pull qwen2.5:1.5b   # optional third fingerprint

1. Human class (no scraping)

Amazon Reviews 2023 (McAuley Lab, UCSD / Hou et al.). Default slice is All_Beauty so the file is hundreds of MB, not tens of GB.

python data/download_amazon.py --categories All_Beauty --per-rating 40

Star ratings are stratified. Product title/features are joined from metadata so the AI class cannot be separated by "these are about different products."

2. AI class (matched)

Each human review seeds one LLM review: same product, same star rating, same category. Prompt style and model rotate (casual_rambling, concise_professional, enthusiastic_marketing, pros_cons × local models).

python data/generate_ai_reviews.py
python data/build_splits.py

3. Train

python tests/test_features.py
python model/train.py --skip-transformer          # minutes on CPU
python model/train.py                             # also fine-tunes DistilBERT

Metrics land in artifacts/metrics/eval.json. The operating threshold is chosen on validation to hit ~0.90 precision on the AI class (or max-F1 if unreachable). Test numbers use that frozen threshold — we do not retune on test.

4. Evasion

python model/evasion.py

Reports recall drop at the production threshold under typos, a grafted shopper aside, human-sentence splicing, and Ollama paraphrase. Output: artifacts/reports/evasion.json.

5. API

python -m uvicorn api.main:app --host 127.0.0.1 --port 8000

POST /analyze
{
  "reviews": [
    {
      "id": "r1",
      "text": "I recently purchased this serum and it exceeded my expectations...",
      "rating": 5,
      "verified_purchase": false,
      "product_title": "CeraPep Repair Serum 30ml"
    }
  ]
}

Response: ai_probability, verdict ("Most likely human" … "Most likely AI"), signals.

POST /analyze-listing
{
  "product_id": "B00TESTASIN",
  "listed_rating": 4.8,
  "reviews": [ { "id": "r1", "text": "...", "rating": 5 } ]
}

Returns a trust object: percent_ai is the share of reviews the classifier scored ≥50% AI, kept separate from template clustering (n_template_reviews, cluster_note). Near-duplicate clusters use union-find (cosine ≥ 0.90, escalate only at size 5+). Short blurbs and Amazon UI chrome (“Verified Purchase”, star labels) are stripped/skipped so generic 5-star titles do not look like a campaign. burst flags 3-day posting spikes (z-score ≥ 3 vs median reviews/day, or 40+ five-stars in 3 days). trend is a sparkline of stored daily snapshots (SQLite) falling back to AI% by review-post month when visit history is sparse. Listings are cached ~12h by product + review-text fingerprints. Embeddings use local nomic-embed-text when Ollama is up, TF-IDF otherwise.

GET /listing-history?product_id=B00TESTASIN

Daily snapshots (percent_ai, adjusted_rating, cluster_count, burst_flag). Seed a demo series with python scripts/seed_listing_history.py --product-id B0032RMX3U.

6. Extension

1. Chrome → chrome://extensions → Developer mode → Load unpacked → extension/
2. Popup: confirm API URL http://127.0.0.1:8000
3. Open a review page on Amazon, Trustpilot, Etsy, Sephora, or Walmart

The content script only reads review nodes already in the page via a per-site adapter (extension/adapters/). It does not crawl. Badges are confidence scores, never a hard FAKE label. See extension/adapters/README.md for the schema and deferred sites.

Features (beyond raw text)

Linguistic: sentence-length burstiness, type-token ratio, hapax rate, character entropy, punctuation/emoji/uppercase, contraction rate.

Amazon-specific:

• Product-specificity — token overlap with title + bullets. Generic enough to paste onto any item in the category = AI tell.
• Rating–sentiment gap — humans are inconsistent (4 stars, still complaining). LLM copy usually matches the stars too cleanly.
• Verified-purchase prior — used in training/eval; live pages expose it sometimes, and the API accepts null.
• Generic phrase lexicon — "exceeded my expectations", "must-have for anyone", "elevate your…", stock openers like "I recently purchased".
• Template clusters — near-duplicate wording across the listing, even when no single review scores as AI.
• Posting bursts — 3-day volume spikes vs that listing's own daily baseline (z-score), independent of the text classifier.

The logistic layer is trained on these plus word/char TF-IDF. DistilBERT (when trained) is blended with a weight tuned only on val PR-AUC.

False-positive stance

Accuracy is the wrong headline. A 95% accurate detector that sprinkles red badges on real 5-star reviews is worse than no detector.

We raise the threshold until validation precision on the AI class is high, and we under-weight the AI class slightly during logistic training. The extension maps scores to three bands so the UI can stay quiet on the middle.

See writeup/limitations.md.

Deploy

The API is a single FastAPI process. Production ships artifacts/models/hybrid.joblib (sklearn path; DistilBERT is optional and heavy).

1. API on Render

1. Push this repo to GitHub (or GitLab).
2. In Render: New → Blueprint → select the repo (uses render.yaml + Dockerfile).
3. Wait for the deploy; open https://<service>.onrender.com/health and confirm "model_present": true.

Local Docker smoke test:

docker build -t review-detector .
docker run -p 8000:8000 -e PORT=8000 review-detector

2. Point the extension at the API

1. Set PRODUCTION_API_URL in extension/config.js to your HTTPS origin (e.g. https://review-ai-detector.onrender.com).
2. Reload the unpacked extension (or rebuild the zip).
3. Popup → confirm API URL → Save. Free Render instances may cold-start (~30s) on first request.

host_permissions allow https://*.onrender.com/*, https://*.loca.lt/*, and localhost for development.

Helper after gh auth login:

powershell -File scripts/deploy-render.ps1

Interim (local API + public tunnel): while developing without Render, run python -m uvicorn api.main:app --host 127.0.0.1 --port 8000 and a tunnel to port 8000, then put that HTTPS URL in extension/config.js.

3. Package the extension

powershell -File scripts/package-extension.ps1

Output: dist/review-ai-detector-1.0.0.zip.

• Sideload: chrome://extensions → Developer mode → Load unpacked → extension/
• Chrome Web Store: Developer Dashboard → upload the zip. Use extension/privacy.html (or host it) as the privacy policy URL. Store listing requires your Google developer account ($5 one-time).

License & data

Code: MIT. Review text comes from Amazon Reviews 2023 for research; do not republish the raw JSONL. Cite:

Hou, Yupeng, et al. “Bridging Language and Items for Retrieval and Recommendation.” arXiv:2403.03952, 2024.

This is a research demo, not an accusation engine. Model card: writeup/model_card.md.