This repository contains a 2026 benchmark of four AI-text detection systems on a balanced English dataset of 1,000 texts. The benchmark is designed to evaluate not only overall accuracy, but also the human false positive rate: the rate at which real human writing is incorrectly flagged as AI-generated.

The tested detectors are:

• GPTHumanizer
• GPTZero
• ZeroGPT
• Sapling AI Detector

All detector runs in this repository were completed on May 14, 2026. The repository includes the benchmark input data, evaluation scripts, and aggregate metrics. Because the complete per-item detector outputs are large, they are hosted as public Google Drive artifacts and linked below for learning and research use.

• GPTHumanizer Official Website
• Related benchmark blog article

GPTZero achieved the highest overall accuracy in this run, while GPTHumanizer had the lowest human false positive risk.

The confusion matrix treats AI as the positive class:

• TP: AI text correctly classified as AI
• FP: human text incorrectly classified as AI
• FN: AI text incorrectly classified as human
• TN: human text correctly classified as human

GPTZero returned two API errors during the completed run, so its accuracy and rates are calculated on 998 evaluable records. Those two non-ok records are still preserved in the output file.

The benchmark uses 1,000 English texts:

The word-count distribution is balanced across the human and AI splits:

The human benchmark set contains 500 human-written texts sampled from Pile-small. The final test file stores each sample with its text, label, source, style field, and perplexity score.

Human source coverage in the final benchmark set:

Why this matters:

• The human data is not drawn from a single writing style or domain.
• The sources include encyclopedic, web, legal, technical, biomedical, email, forum, patent, academic, and subtitle-style text.
• Every final human benchmark record has a human label, non-empty text, a source field, and a retained per-item record in the detector outputs.

The AI benchmark set contains 500 AI-generated texts randomly sampled from a larger pool of 2,600 prompted generations. The AI generation pool was produced on February 5, 2026 by prompting large language models directly. The final benchmark file stores each sampled item with its text, label, prompt, source model, and theme.

AI source model coverage in the final benchmark set:

Why this matters:

• The AI split is not produced by a single model family.
• The model source is retained for every final benchmark item.
• The prompts are preserved in data/ai_detection_test.json, making the sampled AI data auditable at the item level.

All model test results have complete classification data. Because these output JSON files are large, the full artifacts are stored on Google Drive instead of being committed directly to the GitHub repository. They are publicly shared and may be used for learning or research purposes.

Each output JSON contains aggregate metrics and an items object with detailed per-item results, including text, source metadata, true label, predicted label, correctness, word count, word bucket, request status, HTTP status where available, and the detector response fields.

GPTZero non-ok records from the completed run:

Different detectors return different output formats. The benchmark normalizes each detector into a binary human or ai prediction using explicit rules stored in the output metadata.

These rules are implemented in the evaluation scripts and recorded in each output file's meta.prediction_rule field.

The benchmark reports the following metrics:

• Overall accuracy: (TP + TN) / evaluable_items
• AI detection rate: TP / (TP + FN)
• Human false positive rate: FP / (FP + TN)
• AI miss rate: FN / (TP + FN)
• Evaluable items: records with a valid normalized human or ai prediction

The human false positive rate is a central metric because false accusations against human-written text can create serious academic, professional, or institutional risk.

Shorter text is harder to classify because detectors have less linguistic evidence. The benchmark therefore reports results by word-count bucket.

GPTZero has 298 evaluable records in the 500-1000 word bucket because two records in that bucket returned API errors.

1. GPTHumanizer produced zero human false positives in this benchmark.

   It classified all 500 evaluable human-written texts as human. This does not prove a universal 0% false positive rate, but it is the strongest human-safety result observed in this dataset.

2. GPTZero produced the strongest raw accuracy and AI recall.

   GPTZero reached 98.70% accuracy on 998 evaluable records and detected 497 of 499 evaluable AI texts. It also incorrectly flagged 11 of 499 evaluable human texts as AI.

3. ZeroGPT and Sapling showed high human false positive risk.

   ZeroGPT flagged 92 of 500 human texts as AI. Sapling flagged 97 of 500 human texts as AI using the benchmark rule of score > 50%.

4. Short text is the most difficult bucket.

   All detectors showed weaker behavior on 50-200 word samples. GPTHumanizer missed more short AI texts, while GPTZero, ZeroGPT, and Sapling showed higher human false positive rates in that range.

.
|-- data/
|   |-- human_detection_test.json
|   |-- ai_detection_test.json
|-- evaluate_detection_datasets.py
|-- evaluate_gptzero_datasets.py
|-- evaluate_zerogpt_datasets.py
|-- evaluate_sapling_datasets.py
|-- requirements.txt
`-- README.md

Install dependencies:

python -m pip install -r requirements.txt

Run the GPTHumanizer evaluation:

export GPTHUMANIZER_API_KEY="your_key_here"
python evaluate_detection_datasets.py --restart

Run GPTZero:

export GPTZERO_API_KEY="your_key_here"
python evaluate_gptzero_datasets.py --restart

Run ZeroGPT:

export ZEROGPT_API_KEY="your_key_here"
python evaluate_zerogpt_datasets.py --restart

Run Sapling:

export SAPLING_API_KEY="your_key_here"
python evaluate_sapling_datasets.py --restart --drop-token-fields

On Windows PowerShell, set environment variables with $env:NAME = "value" before running the scripts.

The scripts also support explicit headers or API keys through command-line options. Do not commit private API keys to the repository.

This repository is intended to make the benchmark auditable rather than aggregate-only.

Evidence available in this repository and the linked public artifacts:

• The final 500 human test records.
• The final 500 AI test records.
• Source metadata for human samples.
• Prompt, theme, and model-source metadata for AI samples.
• Evaluation scripts for all four detectors.
• Public Google Drive links to the complete per-item classifier outputs for all four detectors.
• Aggregate metrics embedded in each linked output file.

This means the headline results can be checked against the item-level classifications by downloading the linked output artifacts instead of relying only on summary tables.

• The benchmark is English-only.
• The sample size is 1,000 texts, balanced as 500 human and 500 AI.
• Detector APIs can change over time. These results describe the detector behavior observed in the completed May 14, 2026 runs.
• GPTZero had two API errors in the completed run. They are preserved in the output and excluded from evaluable metrics.
• The benchmark uses explicit binary normalization rules, but some detector outputs are more nuanced than binary human/AI labels.
• A measured 0% false positive rate on 500 human samples should be interpreted as an observed benchmark result, not as proof that future false positives cannot occur.
• Exact regeneration of the original random samples may require the original upstream Pile-small snapshot, AI generation pool, sampling code, and random seeds if those are not otherwise archived.

AI detectors should support review, not replace human judgment. A detector result should not be used as the only evidence in high-stakes academic, employment, publishing, or compliance decisions.

This benchmark suggests that detector trust should be evaluated with special attention to human false positives. Catching AI-generated text is useful, but falsely accusing human writers is often the more serious risk.