ArchAgent

English | 简体中文

Archaeological Breakthrough Analyzer — an open-source pipeline that runs a two-phase LLM analysis on Chinese Neolithic–Bronze Age excavation papers, extracts chronological / attribution / typological / methodological conflicts, and aggregates them into a normalized knowledge-graph layer with automatic cross-paper conflict scanning.

考古学术突破点分析器 —— 面向中国新石器至青铜时代考古报告的开源分析流水线： 两阶段 LLM 分析，抽取年代 / 归属 / 类型学 / 方法论层面的学术分歧， 并聚合进规范化知识图谱，自动扫描跨文献冲突。 中文文档见 README.zh-CN.md。

Why? Reviewing excavation reports by hand to find where a new C14 date contradicts the established typology, or where two papers disagree on a site's cultural attribution, is slow and error-prone. ArchAgent does the first-pass critic work and leaves a machine-checkable trail (every claim is anchored to a quote + source location).

Features

• Two-phase pipeline: Phase A (structured extraction) → Phase B (critique), per the archagent_v2 prompt.
• Strict evidence anchoring: every breakthrough must carry verbatim quotes with page/figure anchors; empty-result ("no findings") path is supported.
• Confidence rubric: internal-knowledge comparisons are capped, so the agent never over-claims.
• OpenAI-compatible backends: use DeepSeek / Qwen / Kimi / GLM / OpenAI / local Ollama — all via base_url, no code changes. Extraction and critique can use different models.
• Normalized graph layer: each paper becomes site / culture / object / timespan nodes + typed edges with provenance.
• Cross-paper conflict scan: automatically surfaces "same site, conflicting time spans across papers".
• Pluggable storage: JSON or SQLite by default; swap in Neo4j later via the same GraphStore interface.
• Measured, not vibes: a gold-annotation evaluation harness scores precision/recall, conflict-type accuracy, verbatim-quote grounding, and hallucination traps — and gates CI.
• Runs offline: ARCHAGENT_OFFLINE=1 drives the whole pipeline from a deterministic mock, so tests and CI need no API key.

Install

pip install -e ".[llm,pdf]"        # core + LLM calls + PDF fallback
# optional web UI (phase 2):
pip install -e ".[web]"

pdftotext (poppler) is recommended for best PDF extraction.

Configure

Copy config.example.yaml to config.yaml and set your backends:

extraction:
  base_url: https://api.deepseek.com/v1
  api_key: sk-...
  model: deepseek-chat
critique:
  base_url: https://api.openai.com/v1
  api_key: sk-...
  model: gpt-4o
storage: sqlite
storage_path: ./archagent_store
# reference_corpus: ./my_reference_chronology.txt   # optional: lifts the 0.6 cap on vertical comparisons

Env-var equivalent (prefix ARCHAGENT_EXTRACTION_ / ARCHAGENT_CRITIQUE_).

Usage

Analyze one paper:

archagent analyze --input path/to/paper.pdf --output result.json
archagent analyze --input https://example.com/paper.pdf

Aggregate a corpus and scan conflicts:

archagent build --results-dir ./results/
archagent conflicts

Try it without an API key (deterministic mock backend):

ARCHAGENT_OFFLINE=1 archagent analyze \
  --input archagent/examples/sample_paper.txt --output /tmp/out.json

Serve the web UI:

archagent serve --port 8000        # then open http://127.0.0.1:8000

Programmatic:

from archagent import load_config, analyze_file, make_store, scan_conflicts
from archagent.graph import result_to_fragment

cfg = load_config("config.yaml")
result = analyze_file("paper.pdf", cfg)
store = make_store(cfg.storage, cfg.storage_path)
store.add_fragment(result_to_fragment(result))
store.save()
print(scan_conflicts(store))

Evaluation

An LLM pipeline without a baseline is a demo. archagent eval scores output against human annotations in archagent/examples/gold/:

archagent eval \
  --gold archagent/examples/gold/sample_paper.gold.json \
  --result archagent/examples/sample_output.json

Four metrics, all objective:

metric	what it catches
precision / recall / F1	invented findings and missed conflicts
conflict-type accuracy	right finding, wrong label
grounding rate	quotes marked basis: 原文 that are not actually verbatim in the source
forbidden-claim hits	annotator-defined hallucination traps (fabricated calibration curves, labs, half-lives …)

Current baseline on the shipped 12 synthetic samples (12 papers, 26 annotated findings, all fictional):

P / R / F1 : 1.000 / 1.000 / 1.000
type acc.  : 1.00
grounding  : 6/6 (100%)
forbidden  : none

The shipped sample is a synthetic, fictional paper — the public repo contains no copyrighted text. To evaluate on a real published paper, place your own legally-obtained copy plus its gold annotation into the gitignored archagent/examples/_local/ directory and run archagent eval locally (see CONTRIBUTING.md).

One paper is a floor, not a benchmark. Adding gold annotations is the single most useful contribution to this project — see CONTRIBUTING.md.

CI enforces --min-f1 0.9 --min-grounding 0.85 --no-forbidden, so a prompt or schema edit cannot silently degrade extraction quality.

Project layout

archagent/
  prompts/archagent_v2.md   # the analysis prompt (asset, versioned with the code)
  analyze.py                # two-phase pipeline + JSON repair/retry
  ingest.py                 # PDF / URL / text -> text
  llm.py                    # OpenAI-compatible client + offline mock
  graph.py                  # graph fragments, entity resolution, storage, conflict scan
  eval.py                   # gold-annotation scoring harness
  cli.py / api.py           # CLI + FastAPI service
  static/index.html         # no-build single-page UI
  examples/                 # synthetic sample run, mock fixture
  examples/gold/            # human annotations used by `archagent eval`
  examples/_local/          # (gitignored) YOUR real-paper copies for local eval only
tests/                      # 41 tests, all runnable offline

Roadmap

• Phase-2 web UI (upload → graph view → conflict list) — FastAPI + no-build single-page UI
• Evaluation harness + first gold annotation
• Offline mock backend, JSON self-repair, atomic graph writes
• Grow the gold set to 10+ papers across multiple journals
• Entity resolution beyond string rules (alias table, fuzzy match)
• Neo4j backend implementation
• Batch ingestion over a journal volume

Development

See CONTRIBUTING.md for setup, tests, and how to add a model or storage backend. CI runs on GitHub Actions (.github/workflows/ci.yml): test matrix on Python 3.10–3.13, an offline end-to-end smoke run, the accuracy gate, and a compile check.

Security considerations (your paper text goes to a third-party model provider) are documented in SECURITY.md. Release history is in CHANGELOG.md.

Documentation

Document	Language	Contents
README.md	English	This file
README.zh-CN.md	简体中文	完整中文说明
CONTRIBUTING.md	中文 (English summary at top)	Dev setup, gold annotation guide, PR rules
SECURITY.md	English	Threat model, vulnerability reporting
CHANGELOG.md	English	Release history

License

MIT — see LICENSE.