title	ICSAC Editorial System
emoji	📄
colorFrom	indigo
colorTo	gray
sdk	static
pinned	false
license	mit
short_description	Open-source AI editorial pipeline for ICSAC submissions

ICSAC Editorial System

This is the open-source AI editorial pipeline that accepts and reviews every submission to the Institute for Complexity Science and Advanced Computing and its journal Persistence (/submit).

If you sent a paper to ICSAC — this repository is exactly what received and read it. The submission handler, the rubrics, the prompts, the citation checks, the redaction layer, the audit pass that scores the panel itself. With one defensive exception (the prompt-injection signal in the RQC audit is not published, to avoid teaching adversarial submissions what to avoid — see rubrics/review_quality_control.md for the rationale), nothing about how your work is evaluated is hidden.

Why this is public

Independent and heterodox researchers have legitimate cause to be skeptical of black-box editorial AI. The reasonable response is not to hide the system behind a "trust us." The reasonable response is to publish it.

You can read the rubrics ICSAC reviewers apply (rubrics/). You can read the templates that draft the acceptance and revise-and-resubmit letters (templates/, intake/templates/). You can read the redacted reviews of every accepted paper at icsacinstitute.org/accepted — produced by the exact redaction layer in this repo. You can run the pipeline yourself.

What this repo does NOT include

This repo is the AI reviewer half of the institute. Intentionally not public: the manuscript archive, author records, member directory, internal moderation tooling, the website that fronts all of this, and the operational infrastructure that runs it. Those remain private to protect the privacy and security of authors, members, the team, affiliates, and site visitors. What this repository exposes is how your manuscript was evaluated and what was recorded — not the people, payment, identity, or infrastructure layers around that evaluation.

What it does

The pipeline has two halves: a long-running submission front-end that accepts new manuscripts, and a batch review pipeline that runs on a timer and drains the queue. Both live in this repo.

For each submission:

1. Intake (intake/intake_server.py) — the FastAPI handler behind icsacinstitute.org/submit accepts a DOI / arXiv reference OR a direct PDF upload. The upstream form host verifies the submitter's ORCID via OAuth and signs each request with an HMAC; this handler re-validates and persists the submission record.

2. Manuscript fetch (submission_intake.py) — the worker either downloads the manuscript from the resolver (DOI route) or uses the uploaded PDF as the archive of record. Text and references are extracted.

3. Citation verification (citation_verify.py, citation_misattribution.py) — every cited work is checked against arXiv, Crossref, and Semantic Scholar. Fabricated and misattributed citations are flagged.

4. Five-reviewer panel (review.py) — a panel of independent model instances reviews the manuscript against ICSAC's rubrics (scope, methodology, calibration, tone, AI provenance signal). Each reviewer scores blind to the others. The panel tolerates one slot failure per pass via self-heal (MIN_REVIEWERS=4), so a published review may reflect four valid outputs if a slot errored.

5. Review quality control (review_quality_control.py) — a separate auditor reviews the panel itself. Low-confidence dimensions, missing injection indicators, or systemic drift trigger curator alerts.

6. Redaction (redaction.py) — internal reasoning, vendor names, and operational metadata are stripped before any review is shared with the author or published.

7. Decision (action.py) — the panel recommends one of three outcomes:

   • Accept — published to icsacinstitute.org/accepted/<id> with a redacted copy of the panel's review.
   • Revise and resubmit — author receives the panel's feedback and may resubmit a revised version.
   • Scope reject — reserved for submissions that fall outside the institute's remit (pseudoscience, non-engageable). Not a standard editorial outcome.

   Every panel run produces a recommendation, not a verdict. The recommendation, the full panel report, and the RQC audit are sent to the curator via the operator's configured notification channel. The curator confirms, modifies, or overrides before any author email is drafted, any publications-registry entry is written, or any Zenodo deposit is staged. The panel never auto-delivers a decision.

What it is not

This is the system one institute uses to evaluate its own submissions. It is not a general-purpose academic peer review platform. It is not a service you can submit to without going through icsacinstitute.org/submit. It is not a replacement for human editorial judgment — every panel outcome routes to a human curator for the final verdict, and any author who disagrees with that verdict can appeal to a human editor.

It is also opinionated. The rubrics reflect the institute's editorial scope: complexity science, information theory, persistence dynamics, and adjacent methodology. A submission outside that scope will be flagged as out-of-scope regardless of its merit on its own terms.

Repository layout

The editorial-side modules live at the top level. The submission front-end is a Python subpackage at intake/.

Editorial pipeline (top level)

Path	Purpose
editorial_workflow.py	Top-level workflow orchestration
submission_intake.py	Manuscript fetching and text extraction
citation_verify.py	Cross-reference citation validation
citation_misattribution.py	Catches "real DOI, wrong paper" errors
review.py	The five-reviewer panel
review_quality_control.py	Auditor that scores the panel
redaction.py	Redaction layer for public review output
action.py	Decision dispatch (accept / R&R / scope reject)
repository_deposit.py	Stages accepted papers to a repository backend (Zenodo)
email_send.py, email_render.py	Author correspondence primitives
publications.py, publish_watcher.py	Post-acceptance publication
watch.py	Polls Zenodo for curator state transitions and posts branded comments on accept/decline
notify.py	Curator-channel notification helpers (channel-agnostic dispatcher; forkable)
directory.py	Community directory data layer
stats.py	Aggregate statistics over historical reviews
rubrics/	The editorial rubrics applied to every submission
templates/	Editorial-side correspondence templates (Zenodo-comment posts, etc.)
editorial-batch.{service,timer}, editorial-review.{service,timer}, submission-watcher.{service,timer}	systemd units for the batch workflow, community-request poller, and intake/decision watcher

Submission front-end (intake/)

Path	Purpose
intake/intake_server.py	FastAPI app: POST /api/submit, GET /api/submission/{id}/state, GET /healthz
intake/submission_worker.py	Drains the queue, resolves deferred DOIs, dispatches into the review pipeline
intake/apply_decision.py	Applies the curator's verdict: publications registration, deposit staging, author email
intake/notify_author.py	Author email rendering; attaches redacted panel report + RQC as PDFs
intake/rehydrate.py	Refetch a stubbed DOI submission's bytes from the resolver, verify SHA
intake/templates/	Author-facing email templates
intake/scripts/	Smoke tests for the test-mode pipeline
intake/README.md	Subpackage-scoped documentation

Running it yourself

The pipeline is designed to run on a single host. The submission server runs as a long-lived FastAPI service; the editorial workflow runs from systemd timers (twice daily by default) and drains the queue when it ticks.

Dependencies:

• A submission repository (we use Zenodo — needs an API token with read + deposit scopes)
• An LLM provider (OpenRouter, Anthropic, or compatible)
• An SMTP account for author correspondence
• A registry destination (the institute's website repository, in our case) for post-acceptance publication of accepted papers and redacted reviews

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt

cp config.example.py config.py     # edit for SMTP, Zenodo, LLM provider
export INTAKE_HMAC_SECRET=$(openssl rand -hex 32)   # for the intake HMAC gate

# Submission server (long-running):
uvicorn intake.intake_server:app --host 127.0.0.1 --port 8443 --log-level warning

# Submission worker (drains the queue when markers appear):
python -m intake.submission_worker

# Editorial batch tick (run from systemd timer in production):
python editorial_workflow.py batch-tick

Set secrets in /etc/icsac/editorial.env (loaded by systemd EnvironmentFile=). For manual runs, source the same file. See intake/README.md for the full list of intake-side environment variables.

License

MIT. See LICENSE.

The rubrics and review templates are MIT-licensed code artifacts — feel free to fork, adapt, and use as the basis for your own institute's review system.

A note to authors

If your paper was reviewed by this system and you disagree with the panel's recommendation: write to help@icsacinstitute.org. A human editor reads every appeal. The panel is not the last word — it is a thorough first pass that the curator turns into the verdict, and that the editor overrides on appeal.

If your paper was accepted: the redacted review is published at icsacinstitute.org/accepted/<your-record-id> alongside the work itself.

If you want to know exactly which prompts the panel saw, which rubrics it applied, which citations it verified, and which form fields the submission handler collected before forming its recommendation: read the source. That is why this repository exists.

Support this work

ICSAC operates on bootstrap funding and private donations. Open peer review for independent and heterodox researchers is the kind of infrastructure no one is paid to build. If you want to help keep it running:

manifund.org/projects/icsac-ai-assisted-peer-review-for-independent-researchers