Skip to content

Methodology

Jump to bottom
github-actions[bot] edited this page Jul 7, 2026 · 2 revisions

Methodology

This page gathers, in one findable place, how pyaegean knows what it claims to know: how the Greek NLP models are evaluated, how they are trained and quantized, where every dataset comes from and under what licence, how the training data is kept clean of the evaluation folds, and how the toolkit marks the line between settled scholarship and machine-generated hypothesis.

None of this is new material: it is the methodology already recorded across the project, pulled together here for a reviewer who wants the whole picture at once. Each section names the primary source and links onward:

This is a documentation page, not an academic paper, and it does not claim to be one. It describes what the code and data actually do, with the commands to reproduce every number yourself.

At a glance

Section The question it answers Primary source
Evidence-tier framework Is this a fact, a measurement, or a hypothesis? For Specialists
Evaluation methodology How are the Greek NLP numbers measured, and what do they mean? Benchmarks
Leakage control How is the test set kept out of training? Benchmarks
Training and quantization How is the shipped model built and shrunk without losing accuracy? training/README.md
Provenance, licensing, reproducibility Where does every byte come from, and may I redistribute it? Data & Provenance

1. The evidence-tier framework

Everything pyaegean reports falls into one of three registers, and each result is marked so you always know which one you are looking at. The register also tells you what a wrong result is: a correction, a challenge to a number, or a refutation of a hypothesis.

Register What it covers How it is marked If it is wrong, it is a…
Established Facts carried from editions, lexica, and the Unicode standard: Linear B / Cypriot sign values, the Greek lexicon and morphology (Perseus AGDT, LSJ), bundled transliterations, the find-site gazetteer. Each cites its source (info / cite, Data & Provenance, NOTICE). correction
Measured Model accuracies reported leakage-free on held-out data (the Greek lemmatizer / tagger / parser and the neural pipeline). A number with a reproducible protocol (this page, Greek NLP, docs/benchmarks.md). reproduce or challenge the number
Exploratory Anything decipherment-adjacent over the undeciphered Linear A material (cross-linguistic distances, morphological clusters, structure heuristics, metrological guesses) and all AI-layer output. An explicit [EXPLORATORY …] tag, an exploratory=True flag, a red badge in Jupyter, and an auditable trace(). validation (confirm or refute)

Editorial certainty travels with every token

Every token carries an editorial certainty following Leiden / EpiDoc conventions, so an edition's apparatus survives into the data model and back out through the EpiDoc and JSON round-trips. The four states are exhaustive:

ReadingStatus Meaning EpiDoc / Leiden
certain securely read (the default) —
unclear damaged but read <unclear> / underdot
restored editorially supplied <supplied> / [ ]
lost not preserved / lacuna <gap> / [---]

The bundled corpora are normalized transcriptions, mostly certain with a real fraction damaged. In the bundled Linear A corpus 552 tokens load as LOST and 120 as UNCLEAR (together touching 366 of the 1,721 documents), so any analysis can choose to trust, weight, or exclude them rather than have the damage hidden.

The undeciphered scripts are never presented as read

pyaegean covers two undeciphered scripts, and treats them honestly:

  • Linear A (lineara): of 342 signs in the bundled inventory, 50 carry an empirical sound value, drawn from the 81 signs shared with the Linear B grid and each stamped with a confidence; the rest have no agreed reading. The phonetic transcription uses Linear B sound values only as a working convention. There is deliberately no Greek-reading bridge for Linear A; anything in that direction lives in the exploratory AI layer, labeled as such.
  • Cypro-Minoan (cyprominoan): of its 99 catalogued signs, none carries a settled sound value, so pyaegean offers no transliteration or lexicon for it, only the sign inventory and sign-sequence tokenization.

The two deciphered syllabaries (Linear B, Cypriot) do carry a Greek-reading bridge, because those scripts write Greek; that bridge is established, not exploratory.

Auditing an exploratory result

Because exploratory output is only as good as the evidence under it, the AI layer makes that evidence visible. Every generative result returns an ExploratoryResult you can audit: the [EXPLORATORY …] label travels with the text, trace() groups the exact local facts the model was given by source and ref, and a trace that reads grounding: none (ungrounded generation …) tells you the answer rested on the model's parametric knowledge alone. Grounding fidelity itself is measured like an accuracy number (ai.run_eval, aegean ai eval), scoring groundedness and fabrication rate rather than authority. The full walkthrough is on For Specialists.

2. Evaluation methodology

The Greek NLP accuracy numbers are the project's measured claims. They are scored on the standard Ancient Greek benchmarks under a fixed, reproducible protocol. The full protocol, the cross-tool comparison tables, and their citations are in Benchmarks; this section summarizes it and reports pyaegean's own numbers (per the wiki convention, cross-tool comparisons stay in the benchmarks doc).

Protocol

  • Test sets. The Universal Dependencies Ancient Greek test folds: UD_Ancient_Greek-Perseus (commit 331ddef, CC BY-NC-SA 2.5) and UD_Ancient_Greek-PROIEL (commit a4ab8d4, CC BY-NC-SA 3.0). Both are fetched to the cache for evaluation only: never bundled, never trained on.
  • Scorer. The official CoNLL 2018 shared-task evaluator (conll18_ud_eval.py, MPL 2.0), fetched sha256-pinned and imported from the cache. Reported figures are the evaluator's F1 per metric.
  • Gold tokenization. pyaegean runs over each fold's gold FORM column, so the scores measure tagging / lemma / parsing quality rather than tokenizer agreement. The neural pipeline is also measured end-to-end from raw text through pyaegean's own tokenizer (tokens F1 99.97), and the scores track the gold-tokenization figures closely, so tokenization is not a bottleneck on this fold.
  • No tagset reconciliation on the UD folds. UPOS and lemmas are scored exactly as emitted; convention gaps count against pyaegean here, unlike greek.evaluate_on_proiel, which reconciles tagsets to isolate real errors.
  • Lemma scoring. On the UD folds, lemmas use the evaluator's exact string match with no added normalization (the UD gold is already NFC and carries no homograph-index digits). Convention differences (principal-part choice, movable-nu, proper-noun citation form) therefore count as errors rather than being normalized away. The native-corpus checks (evaluate_on_nt / evaluate_on_proiel) apply a light clean-up (NFC plus homograph-digit stripping) because those golds are not pre-normalized.
  • Train / dev / test discipline. Training is the leakage-clean corpus (below). The dev fold is used for early stopping, checkpoint selection, light schedule tuning, and the quantization gate; the test folds are scored once on the finished model and never used for any selection.

What the metrics mean

All are percentages against a human-annotated gold standard; higher is better.

Metric In plain terms
UPOS Universal part of speech: the basic word class from UD's 17-tag set.
XPOS The language-specific (treebank's own) finer-grained tag; not comparable across treebanks with different tagsets, so sometimes marked n/a.
UFeats The full morphology (case, number, gender, tense, mood, voice, person). A word counts only if every feature is right: the strictest word-level tag.
Lemma The dictionary / citation form (λέγει → λέγω).
UAS Unlabeled Attachment Score: the fraction of words hooked to the correct syntactic parent.
LAS Labeled Attachment Score: UAS, but the dependency link must also carry the right relation label. The usual headline parsing number.

Two supporting terms: the scorer reports F1 (the balance of precision and recall) per metric, and a bootstrap confidence interval (e.g. [89.6, 90.9]) is the range a score would plausibly fall in on similar data, estimated by re-sampling the fold's sentences. A narrow interval means the number is stable, not a lucky fold.

pyaegean's measured numbers (the shipped neural pipeline)

The shipped joint model (grc-joint-v3, activated by greek.use_neural_pipeline(), the [neural] extra) is one GreBerta-encoder checkpoint serving UPOS, XPOS, UD FEATS, dependency trees (single-root Chu-Liu/Edmonds MST decoding, so non-projectivity is handled natively), and lemmas. Measured through the package's own inference code:

Test fold Lemma UAS LAS UPOS UFeats XPOS
UD Perseus (in family) 94.29 90.23 85.64 97.04 96.04 93.48
UD PROIEL (out of domain) 90.50 82.47 63.47 86.71 59.43 n/a

The shipped checkpoint is one of five seed replicates of the recipe; across those seeds the UD Perseus test mean ± standard deviation is LAS 85.58 ± 0.10, UAS 90.15 ± 0.12, UPOS 97.00 ± 0.06, UFeats 96.06 ± 0.04, lemma 94.30 ± 0.02, XPOS 93.52 ± 0.05 (PROIEL LAS 63.50 ± 0.04), so the headline figures are representative, not a lucky seed. Within-fold 95% bootstrap confidence intervals (percentile bootstrap over the fold's sentences, greek.bootstrap_ud's default of 999 resamples) accompany every headline number in docs/benchmarks.md.

Out of domain is always reported alongside in-family. PROIEL is a treebank none of pyaegean's models train on, so its numbers are the honest generalization figure. The remaining PROIEL LAS and UFeats gaps are largely convention divergence between the two treebanks' UD conversions (PROIEL annotates feature types the Perseus scheme lacks, and its XPOS is a different tagset entirely), a measurement boundary rather than a model defect.

A second out-of-domain check scores the pipeline against the Nestle 1904 Greek New Testament's own gold (greek.evaluate_on_nt()): lemma 87.03, UPOS (reconciled) 86.75 over 137,303 tokens. The model never trains on the NT.

The pure-Python baseline is a floor, not the accuracy story

The zero-dependency stack (use_treebank() + use_tagger() + use_lemmatizer() + use_parser()) is the offline, no-heavy-deps path and reads like a baseline (Perseus test UPOS 86.73, PROIEL test UPOS 78.83). Its tagger, lemmatizer, and parser are built from the full AGDT, which contains the UD-Perseus test sentences, so its Perseus-fold scores are an in-training upper bound reported for orientation; the PROIEL fold is its honest number. The neural pipeline, not the baseline, carries the accuracy claims.

Reproduce the numbers

from aegean import greek
greek.use_neural_pipeline()
greek.evaluate_on_ud("perseus", "test")   # in-family
greek.evaluate_on_ud("proiel", "test")    # out of domain
greek.evaluate_on_nt()                     # Koine / NT, out of domain
aegean greek eval nt          # the NT row from the shell

The published offline-stack rows are re-measured against the claims registry by scripts/check_benchmarks.py; every published number lives in training/results/published-claims.json, and tests/test_benchmark_claims.py pins the docs to the registry so a documented number cannot drift silently. Full tables, the field's published numbers, and citations: Benchmarks. The evaluation call sites are documented on Greek NLP → Standard-benchmark evaluation and Greek NLP → Neutral evaluation.

3. Leakage control

UD Perseus is converted from the AGDT, the treebank pyaegean's Greek backends are built from, so a naïve evaluation would leak the test set into training. Two controls keep the neural pipeline's numbers honest:

  • The UD-Perseus exclusion manifest. greek.agdt_ud_overlap() resolves every UD-Perseus dev and test sentence to its AGDT source and verifies the reference by NFC form-sequence comparison: 2,443 sentences across 5 AGDT files, all form-identical. The neural model's training split excludes all of them (cached at ud-grc/agdt-ud-exclusion.json).
  • PROIEL is held out entirely. No pyaegean model trains on PROIEL, so it is a genuine out-of-domain fold. The combined-corpus model adds the Gorman and Pedalion treebanks (both CC BY-SA 4.0); the overlap audit excluded 1,591 Gorman and 155 Pedalion sentences matching either evaluation fold, and Gorman's Herodotus files (the same work as PROIEL's hdt.xml) are excluded at source.

One caveat applies to the pure-Python baseline only, not the neural pipeline: it is built from the full AGDT (which contains the UD-Perseus test sentences), so its Perseus-fold scores are an in-training upper bound, and the PROIEL fold is its honest number. This is stated in place wherever the baseline is reported.

The licence split behind these controls is not incidental: the models train only on the CC BY-SA treebanks (AGDT, Gorman, Pedalion), which permit it, while the CC BY-NC-SA evaluation treebanks (UD Perseus, UD PROIEL, PROIEL) are fetched for evaluation only and never trained on, satisfying both the leakage discipline and the NonCommercial obligation at once (see §5).

4. Training and quantization

The training code lives in training/ and nothing in it ships in the wheel; trained artifacts are published as GitHub release assets and fetched to the cache, never bundled. The evidence for every claim below is in training/results/.

Data protocol

Training data is leakage-clean against the evaluation folds (§3):

  • train = AGDT + Gorman + Pedalion (about 1.41 M tokens), minus every sentence in the UD-Perseus dev+test exclusion manifest (greek.agdt_ud_overlap).
  • dev = the AGDT sentences behind the UD-Perseus dev fold (early stopping, checkpoint selection, light schedule tuning, and the quantization gate).
  • test = in neither train nor dev; final numbers come only from greek.evaluate_on_ud("perseus", "test") against a finished model.

training/data/ and training/out/ are gitignored; the datasets rebuild deterministically from the cache.

The model, in stages

One GreBerta encoder serves every task from a single forward pass. It is built in five stages, each with a dataset builder and a training script:

  • Stage A — encoder selection. Fine-tune UPOS on each candidate encoder under an identical budget; pick on dev accuracy, generalization to unseen forms, size, and licence. GreBerta (a RoBERTa for Ancient Greek, Apache-2.0) is the shipped encoder.
  • Stage B — joint tagger. UPOS + XPOS + UD FEATS on the GreBerta encoder, trained on UD-convention labels from the AGDT→UD converter.
  • Stage C — biaffine parser. Dozat–Manning arc and relation scorers on the shared encoder, with single-root Chu-Liu/Edmonds MST decoding at evaluation (non-projective trees handled natively).
  • Stage D — lemmas. A word-level edit-script classifier (Chrupała edit trees) plus a train-only lookup, on the same checkpoint.
  • Stage E — export and quantize. Export to ONNX (fp32, torch-free at inference: the reproducibility grc-joint-v2 asset), then quantize weight-only to produce the shipped grc-joint-v3 asset.

The quantization discipline

The shipped model is quantized at about 173 MB (tar.gz; 182 MB uncompressed model.onnx), roughly 3× smaller than the fp32 build (518 MB tar.gz / 556 MB uncompressed) and lossless on accuracy: the UD Perseus test scores are unchanged within ±0.02 (UPOS 97.0 / UFeats 96.0 / lemma 94.3 / UAS 90.2 / LAS 85.6). The recipe is weight-only int8 + fp16, activations kept fp32: onnxruntime MatMulNBits (block 128, symmetric) on the MatMul weights, fp16 on everything else (crucially the ~160 MB word-embedding table), activations fp32 by design.

This is the recipe that works because the obvious one does not. Full int8 (quantized activations) collapses the GreBerta encoder: its activation outliers do not survive 8-bit quantization, so every dynamic or static int8-activation recipe tried dropped UPOS from 97 to 16–32 and LAS from 86 to 1–13. That recipe was gated on a ≤0.3-point drop and rejected; keeping activations fp32 and quantizing only the weights ships the size win at no accuracy cost. The rejected recipe is recorded in training/results/gate-report.json, and the shipped sizes plus the lossless comparison in training/results/v3-quantize-report.json.

The trade-off the quantization does carry is CPU throughput, not accuracy: the int8 MatMulNBits kernels run several times slower than fp32 MatMul on this workload (roughly 20–70 words/s quantized versus roughly 300 words/s fp32 on the development machine), so the quantized default optimizes download size and disk, not speed. Throughput is hardware-dependent and illustrative, not a pinned benchmark (unlike the accuracy figures, which are deterministic and re-measured), and throughput-sensitive work can fetch the fp32 grc-joint-v2 asset instead. The quantized model needs onnxruntime >= 1.23 (the 8-bit MatMulNBits CPU kernel), so the [neural] extra floor was raised from 1.17 to 1.23; the fp32 model stays available at the grc-joint-v2 release for reproducibility.

Both models run torch-free at inference, on numpy + onnxruntime, loaded only on activation.

5. Data provenance, licensing, and reproducibility

The full accounting of where every byte comes from is on Data & Provenance; this section surfaces the parts that bear on trusting a result.

Bundled vs fetched, and why the wheel stays small

Code and tiny text JSON are bundled and work offline with zero third-party dependencies. Large or licence-restricted assets are never bundled: they are fetched on demand into a local, sha256-verified cache. The wheel ships only code + tiny JSON, and CI's scripts/check_footprint.py enforces exactly that (plus an instant, heavy-dependency-free import). A fetched dataset is permanent until you delete it: the "cache" is a permanent local store, not an evicting one.

The licence split that keeps training clean

The provenance rules and the leakage rules are the same rules seen from two sides:

  • Trained on (permit it): the AGDT (CC BY-SA 3.0), Gorman (CC BY-SA 4.0), and Pedalion (CC BY-SA 4.0) treebanks. The derived artifacts are republished under the same ShareAlike terms, clearly labeled, and fetched to the cache, never bundled.
  • Evaluation only, never trained on: the UD Ancient Greek treebanks (UD-Perseus CC BY-NC-SA 2.5; UD-PROIEL CC BY-NC-SA 3.0) and the PROIEL treebank (CC BY-NC-SA 3.0), plus the CoNLL-2018 evaluator (MPL-2.0). These are fetched for evaluate_on_ud() / evaluate_on_proiel() only, and their NonCommercial + ShareAlike obligations pass through to you.

Model cards make the base-vs-derivative licence explicit: the neural pipeline's base encoder is bowphs/GreBerta (Apache-2.0), and the released grc-joint bundle is CC BY-SA 4.0, fetched to the cache, never bundled, so the wheel itself stays Apache-2.0. The bundled Aegean sign data is from the Unicode Character Database (Unicode licence); the Linear A corpus JSON is GORILA via mwenge/lineara.xyz (Apache-2.0), with the facsimile imagery referenced, never redistributed. The per-source rights table, including the NonCommercial DAMOS / SigLA corpora and the CC0 Nestle 1904 NT, is on Data & Provenance.

Pinning an analysis for reproducibility

Every dataset pyaegean can touch is versioned and hashable. data.versions() returns a manifest with package, bundled (each JSON file hashed straight from the installed wheel), and fetched (each remote asset's pinned URL, pinned sha256, licence, and cached state). Matching sha256s mean byte-identical data.

import json, aegean
from aegean import data
with open("data-versions.json", "w", encoding="utf-8") as f:
    json.dump({"package": aegean.__version__, "data": data.versions()}, f, indent=2)
aegean data versions --json > data-versions.json

To pin an analysis for a paper, record aegean.__version__ and dump this manifest alongside your results. Loaded literary works additionally record the exact upstream commit as Provenance.data_version, and every Corpus carries a Provenance that stamps exports and produces a citation of the exact subset you used. Cite the underlying edition, not pyaegean's wrapper: the mechanics are on For Specialists and Data & Provenance.

See also

  • Greek NLP: the pipeline, the evaluation call sites, and the tier switch (baseline → treebank → neural)
  • Benchmarks: the full protocol, the field's published numbers, and citations
  • Data & Provenance: every dataset, licence, and the cache layout
  • For Specialists: auditing, citing, and correcting a result
  • Limitations: the candid register of what the toolkit can and cannot claim

pyaegean

Start here

Aegean scripts

Greek

Capabilities

Evaluation & methodology

Reference

Clone this wiki locally