Repository navigation
This release provides the reference implementation, datasets, and analysis workflows for NERO (Nonparametric Entropy-Rate Oracle), a model-agnostic, training-free framework for characterizing long-form text via entropy-rate estimation.
NERO estimates the Shannon entropy rate of text mapped to a fixed 27-symbol alphabet (26 letters plus space) using a nonparametric, PFSA-based method. When applied to long-form prose, entropy-rate estimates produced by NERO are systematically lower for contemporary large language model (LLM) outputs than for human-authored text. This intrinsic complexity signal enables robust discrimination between human and AI-generated text without access to model internals, token probabilities, or supervised retraining.
This archive accompanies the paper:
Schmidt R, Chattopadhyay I. Complexity Signature of Generated Text (2026).
The release includes:
- the compiled NERO binary implementing the entropy-rate estimator,
- pipelines and metadata for generating long-form AI text (api_data_collection/),
- curated human reference corpora and categorizations (human_text_categories/),
- analysis notebooks reproducing all figures and results reported in the paper,
- derived tables and plot-ready data used for evaluation and visualization,
- manuscript sources for transparency and archival completeness.
NERO is intended for long-form text analysis (on the order of 10⁴–10⁵ characters or more) and provides a principled, model-independent statistic for AI text detection and for tracking changes in generative behavior across model generations.
License: Non-commercial