-
Notifications
You must be signed in to change notification settings - Fork 0
Methodology
This page gathers, in one findable place, how pyaegean knows what it claims to know: how the Greek NLP models are evaluated, how they are trained and quantized, where every dataset comes from and under what licence, how the training data is kept clean of the evaluation folds, and how the toolkit marks the line between settled scholarship and machine-generated hypothesis.
None of this is new material: it is the methodology already recorded across the project, pulled together here for a reviewer who wants the whole picture at once. Each section names the primary source and links onward:
- the evaluation protocol, leakage controls, and metric definitions: Benchmarks and Greek NLP
- the model training and int8 quantization discipline:
training/README.mdwith the evidence files intraining/results/ - data sources, licences, and the reproducibility manifest: Data & Provenance
- the established / measured / exploratory framework and how to audit a result: For Specialists and Limitations
This is a documentation page, not an academic paper, and it does not claim to be one. It describes what the code and data actually do, with the commands to reproduce every number yourself.
| Section | The question it answers | Primary source |
|---|---|---|
| Evidence-tier framework | Is this a fact, a measurement, or a hypothesis? | For Specialists |
| Evaluation methodology | How are the Greek NLP numbers measured, and what do they mean? | Benchmarks |
| Leakage control | How is the test set kept out of training? | Benchmarks |
| Training and quantization | How is the shipped model built and shrunk without losing accuracy? | training/README.md |
| Provenance, licensing, reproducibility | Where does every byte come from, and may I redistribute it? | Data & Provenance |
Everything pyaegean reports falls into one of three registers, and each result is marked so you always know which one you are looking at. The register also tells you what a wrong result is: a correction, a challenge to a number, or a refutation of a hypothesis.
| Register | What it covers | How it is marked | If it is wrong, it is a… |
|---|---|---|---|
| Established | Facts carried from editions, lexica, and the Unicode standard: Linear B / Cypriot sign values, the Greek lexicon and morphology (Perseus AGDT, LSJ), bundled transliterations, the find-site gazetteer. | Each cites its source (info / cite, Data & Provenance, NOTICE). |
correction |
| Measured | Model accuracies reported leakage-free on held-out data (the Greek lemmatizer / tagger / parser and the neural pipeline). | A number with a reproducible protocol (this page, Greek NLP, docs/benchmarks.md). |
reproduce or challenge the number |
| Exploratory | Anything decipherment-adjacent over the undeciphered Linear A material (cross-linguistic distances, morphological clusters, structure heuristics, metrological guesses) and all AI-layer output. | An explicit [EXPLORATORY …] tag, an exploratory=True flag, a red badge in Jupyter, and an auditable trace(). |
validation (confirm or refute) |
Every token carries an editorial certainty following Leiden / EpiDoc conventions, so an edition's apparatus survives into the data model and back out through the EpiDoc and JSON round-trips. The four states are exhaustive:
ReadingStatus |
Meaning | EpiDoc / Leiden |
|---|---|---|
certain |
securely read (the default) | — |
unclear |
damaged but read |
<unclear> / underdot |
restored |
editorially supplied |
<supplied> / [ ]
|
lost |
not preserved / lacuna |
<gap> / [---]
|
The bundled corpora are normalized transcriptions, mostly certain with a real
fraction damaged. In the bundled Linear A corpus 552 tokens load as LOST and
120 as UNCLEAR (together touching 366 of the 1,721 documents), so any analysis
can choose to trust, weight, or exclude them rather than have the damage hidden.
pyaegean covers two undeciphered scripts, and treats them honestly:
-
Linear A (
lineara): of 342 signs in the bundled inventory, 50 carry an empirical sound value, drawn from the 81 signs shared with the Linear B grid and each stamped with aconfidence; the rest have no agreed reading. The phonetic transcription uses Linear B sound values only as a working convention. There is deliberately no Greek-reading bridge for Linear A; anything in that direction lives in the exploratory AI layer, labeled as such. -
Cypro-Minoan (
cyprominoan): of its 99 catalogued signs, none carries a settled sound value, so pyaegean offers no transliteration or lexicon for it, only the sign inventory and sign-sequence tokenization.
The two deciphered syllabaries (Linear B, Cypriot) do carry a Greek-reading bridge, because those scripts write Greek; that bridge is established, not exploratory.
Because exploratory output is only as good as the evidence under it, the AI layer
makes that evidence visible. Every generative result returns an
ExploratoryResult you can audit: the [EXPLORATORY …] label travels with the
text, trace() groups the exact local facts the model was given by source and
ref, and a trace that reads grounding: none (ungrounded generation …) tells you
the answer rested on the model's parametric knowledge alone. Grounding fidelity
itself is measured like an accuracy number (ai.run_eval, aegean ai eval),
scoring groundedness and fabrication rate rather than authority. The full
walkthrough is on For Specialists.
The Greek NLP accuracy numbers are the project's measured claims. They are scored on the standard Ancient Greek benchmarks under a fixed, reproducible protocol. The full protocol, the cross-tool comparison tables, and their citations are in Benchmarks; this section summarizes it and reports pyaegean's own numbers (per the wiki convention, cross-tool comparisons stay in the benchmarks doc).
-
Test sets. The Universal Dependencies Ancient Greek test folds:
UD_Ancient_Greek-Perseus(commit331ddef, CC BY-NC-SA 2.5) andUD_Ancient_Greek-PROIEL(commita4ab8d4, CC BY-NC-SA 3.0). Both are fetched to the cache for evaluation only: never bundled, never trained on. -
Scorer. The official CoNLL 2018 shared-task evaluator (
conll18_ud_eval.py, MPL 2.0), fetched sha256-pinned and imported from the cache. Reported figures are the evaluator's F1 per metric. -
Gold tokenization. pyaegean runs over each fold's gold
FORMcolumn, so the scores measure tagging / lemma / parsing quality rather than tokenizer agreement. The neural pipeline is also measured end-to-end from raw text through pyaegean's own tokenizer (tokens F1 99.97), and the scores track the gold-tokenization figures closely, so tokenization is not a bottleneck on this fold. -
No tagset reconciliation on the UD folds. UPOS and lemmas are scored exactly
as emitted; convention gaps count against pyaegean here, unlike
greek.evaluate_on_proiel, which reconciles tagsets to isolate real errors. -
Lemma scoring. On the UD folds, lemmas use the evaluator's exact string
match with no added normalization (the UD gold is already NFC and carries no
homograph-index digits). Convention differences (principal-part choice,
movable-nu, proper-noun citation form) therefore count as errors rather than
being normalized away. The native-corpus checks (
evaluate_on_nt/evaluate_on_proiel) apply a light clean-up (NFC plus homograph-digit stripping) because those golds are not pre-normalized. - Train / dev / test discipline. Training is the leakage-clean corpus (below). The dev fold is used for early stopping, checkpoint selection, light schedule tuning, and the quantization gate; the test folds are scored once on the finished model and never used for any selection.
All are percentages against a human-annotated gold standard; higher is better.
| Metric | In plain terms |
|---|---|
| UPOS | Universal part of speech: the basic word class from UD's 17-tag set. |
| XPOS | The language-specific (treebank's own) finer-grained tag; not comparable across treebanks with different tagsets, so sometimes marked n/a. |
| UFeats | The full morphology (case, number, gender, tense, mood, voice, person). A word counts only if every feature is right: the strictest word-level tag. |
| Lemma | The dictionary / citation form (λέγει → λέγω). |
| UAS | Unlabeled Attachment Score: the fraction of words hooked to the correct syntactic parent. |
| LAS | Labeled Attachment Score: UAS, but the dependency link must also carry the right relation label. The usual headline parsing number. |
Two supporting terms: the scorer reports F1 (the balance of precision and
recall) per metric, and a bootstrap confidence interval (e.g. [89.6, 90.9])
is the range a score would plausibly fall in on similar data, estimated by
re-sampling the fold's sentences. A narrow interval means the number is stable,
not a lucky fold.
The shipped joint model (grc-joint-v3, activated by
greek.use_neural_pipeline(), the [neural] extra) is one GreBerta-encoder
checkpoint serving UPOS, XPOS, UD FEATS, dependency trees (single-root
Chu-Liu/Edmonds MST decoding, so non-projectivity is handled natively), and
lemmas. Measured through the package's own inference code:
| Test fold | Lemma | UAS | LAS | UPOS | UFeats | XPOS |
|---|---|---|---|---|---|---|
| UD Perseus (in family) | 94.27 | 90.24 | 85.65 | 97.02 | 96.04 | 93.48 |
| UD PROIEL (out of domain) | 90.51 | 82.48 | 63.50 | 86.69 | 59.43 | n/a |
The shipped checkpoint is one of five seed replicates of the recipe; across
those seeds the UD Perseus test mean ± standard deviation is LAS 85.58 ± 0.10,
UAS 90.15 ± 0.12, UPOS 97.00 ± 0.06, UFeats 96.06 ± 0.04, lemma 94.30 ± 0.02,
XPOS 93.52 ± 0.05 (PROIEL LAS 63.50 ± 0.04), so the headline figures are
representative, not a lucky seed. Within-fold 95% bootstrap confidence intervals
(percentile bootstrap over the fold's sentences, greek.bootstrap_ud's default of
999 resamples) accompany every headline number in docs/benchmarks.md.
Out of domain is always reported alongside in-family. PROIEL is a treebank none of pyaegean's models train on, so its numbers are the honest generalization figure. The remaining PROIEL LAS and UFeats gaps are largely convention divergence between the two treebanks' UD conversions (PROIEL annotates feature types the Perseus scheme lacks, and its XPOS is a different tagset entirely), a measurement boundary rather than a model defect.
A second out-of-domain check scores the pipeline against the Nestle 1904 Greek
New Testament's own gold (greek.evaluate_on_nt()): lemma 87.96, UPOS
(reconciled) 86.75 over 137,303 tokens. The model never trains on the NT.
The zero-dependency stack (use_treebank() + use_tagger() + use_lemmatizer() + use_parser()) is the offline, no-heavy-deps path and reads like a baseline
(Perseus test UPOS 86.73, PROIEL test UPOS 78.83). Its tagger, lemmatizer, and
parser are built from the full AGDT, which contains the UD-Perseus test
sentences, so its Perseus-fold scores are an in-training upper bound reported
for orientation; the PROIEL fold is its honest number. The neural pipeline, not
the baseline, carries the accuracy claims.
from aegean import greek
greek.use_neural_pipeline()
greek.evaluate_on_ud("perseus", "test") # in-family
greek.evaluate_on_ud("proiel", "test") # out of domain
greek.evaluate_on_nt() # Koine / NT, out of domainaegean greek eval nt # the NT row from the shellThe published offline-stack rows are re-measured against the claims registry by
scripts/check_benchmarks.py; every published number lives in
training/results/published-claims.json, and tests/test_benchmark_claims.py
pins the docs to the registry so a documented number cannot drift silently. Full
tables, the field's published numbers, and citations:
Benchmarks.
The evaluation call sites are documented on
Greek NLP → Standard-benchmark evaluation
and Greek NLP → Neutral evaluation.
UD Perseus is converted from the AGDT, the treebank pyaegean's Greek backends are built from, so a naïve evaluation would leak the test set into training. Two controls keep the neural pipeline's numbers honest:
-
The UD-Perseus exclusion manifest.
greek.agdt_ud_overlap()resolves every UD-Perseus dev and test sentence to its AGDT source and verifies the reference by NFC form-sequence comparison: 2,443 sentences across 5 AGDT files, all form-identical. The neural model's training split excludes all of them (cached atud-grc/agdt-ud-exclusion.json). -
PROIEL is held out entirely. No pyaegean model trains on PROIEL, so it is a
genuine out-of-domain fold. The combined-corpus model adds the Gorman and
Pedalion treebanks (both CC BY-SA 4.0); the overlap audit excluded 1,591 Gorman
and 155 Pedalion sentences matching either evaluation fold, and Gorman's
Herodotus files (the same work as PROIEL's
hdt.xml) are excluded at source.
One caveat applies to the pure-Python baseline only, not the neural pipeline: it is built from the full AGDT (which contains the UD-Perseus test sentences), so its Perseus-fold scores are an in-training upper bound, and the PROIEL fold is its honest number. This is stated in place wherever the baseline is reported.
The licence split behind these controls is not incidental: the models train only on the CC BY-SA treebanks (AGDT, Gorman, Pedalion), which permit it, while the CC BY-NC-SA evaluation treebanks (UD Perseus, UD PROIEL, PROIEL) are fetched for evaluation only and never trained on, satisfying both the leakage discipline and the NonCommercial obligation at once (see §5).
The training code lives in
training/ and
nothing in it ships in the wheel; trained artifacts are published as GitHub
release assets and fetched to the cache, never bundled. The evidence for every
claim below is in
training/results/.
Training data is leakage-clean against the evaluation folds (§3):
-
train = AGDT + Gorman + Pedalion (about 1.41 M tokens), minus every sentence
in the UD-Perseus dev+test exclusion manifest (
greek.agdt_ud_overlap). - dev = the AGDT sentences behind the UD-Perseus dev fold (early stopping, checkpoint selection, light schedule tuning, and the quantization gate).
-
test = in neither train nor dev; final numbers come only from
greek.evaluate_on_ud("perseus", "test")against a finished model.
training/data/ and training/out/ are gitignored; the datasets rebuild
deterministically from the cache.
One GreBerta encoder serves every task from a single forward pass. It is built in five stages, each with a dataset builder and a training script:
- Stage A — encoder selection. Fine-tune UPOS on each candidate encoder under an identical budget; pick on dev accuracy, generalization to unseen forms, size, and licence. GreBerta (a RoBERTa for Ancient Greek, Apache-2.0) is the shipped encoder.
- Stage B — joint tagger. UPOS + XPOS + UD FEATS on the GreBerta encoder, trained on UD-convention labels from the AGDT→UD converter.
- Stage C — biaffine parser. Dozat–Manning arc and relation scorers on the shared encoder, with single-root Chu-Liu/Edmonds MST decoding at evaluation (non-projective trees handled natively).
- Stage D — lemmas. A word-level edit-script classifier (Chrupała edit trees) plus a train-only lookup, on the same checkpoint.
-
Stage E — export and quantize. Export to ONNX (fp32, torch-free at
inference: the reproducibility
grc-joint-v2asset), then quantize weight-only to produce the shippedgrc-joint-v3asset.
The shipped model is quantized at about 173 MB (tar.gz; 182 MB uncompressed
model.onnx), roughly 3× smaller than the fp32 build (518 MB tar.gz / 556 MB
uncompressed) and lossless on accuracy: the UD Perseus test scores are
unchanged within ±0.02 (UPOS 97.0 / UFeats 96.0 / lemma 94.3 / UAS 90.2 /
LAS 85.6). The recipe is weight-only int8 + fp16, activations kept fp32:
onnxruntime MatMulNBits (block 128, symmetric) on the MatMul weights, fp16 on
everything else (crucially the ~160 MB word-embedding table), activations fp32 by
design.
This is the recipe that works because the obvious one does not. Full int8
(quantized activations) collapses the GreBerta encoder: its activation outliers do
not survive 8-bit quantization, so every dynamic or static int8-activation recipe
tried dropped UPOS from 97 to 16–32 and LAS from 86 to 1–13. That recipe was gated
on a ≤0.3-point drop and rejected; keeping activations fp32 and quantizing only
the weights ships the size win at no accuracy cost. The rejected recipe is recorded
in training/results/gate-report.json, and the shipped sizes plus the lossless
comparison in training/results/v3-quantize-report.json.
The trade-off the quantization does carry is CPU throughput, not accuracy:
the int8 MatMulNBits kernels run several times slower than fp32 MatMul on this
workload (roughly 20–70 words/s quantized versus roughly 300 words/s fp32 on the
development machine), so the quantized default optimizes download size and disk,
not speed. Throughput is hardware-dependent and illustrative, not a pinned
benchmark (unlike the accuracy figures, which are deterministic and
re-measured), and throughput-sensitive work can fetch the fp32 grc-joint-v2
asset instead. The quantized model needs onnxruntime >= 1.23 (the 8-bit
MatMulNBits CPU kernel), so the [neural] extra floor was raised from 1.17 to
1.23; the fp32 model stays available at the grc-joint-v2 release for
reproducibility.
Both models run torch-free at inference, on numpy + onnxruntime, loaded only on activation.
The full accounting of where every byte comes from is on Data & Provenance; this section surfaces the parts that bear on trusting a result.
Code and tiny text JSON are bundled and work offline with zero third-party
dependencies. Large or licence-restricted assets are never bundled: they are
fetched on demand into a local, sha256-verified cache. The wheel ships only code +
tiny JSON, and CI's scripts/check_footprint.py enforces exactly that (plus an
instant, heavy-dependency-free import). A fetched dataset is permanent until you
delete it: the "cache" is a permanent local store, not an evicting one.
The provenance rules and the leakage rules are the same rules seen from two sides:
- Trained on (permit it): the AGDT (CC BY-SA 3.0), Gorman (CC BY-SA 4.0), and Pedalion (CC BY-SA 4.0) treebanks. The derived artifacts are republished under the same ShareAlike terms, clearly labeled, and fetched to the cache, never bundled.
-
Evaluation only, never trained on: the UD Ancient Greek treebanks
(UD-Perseus CC BY-NC-SA 2.5; UD-PROIEL CC BY-NC-SA 3.0) and the PROIEL treebank
(CC BY-NC-SA 3.0), plus the CoNLL-2018 evaluator (MPL-2.0). These are fetched for
evaluate_on_ud()/evaluate_on_proiel()only, and their NonCommercial + ShareAlike obligations pass through to you.
Model cards make the base-vs-derivative licence explicit: the neural pipeline's
base encoder is bowphs/GreBerta (Apache-2.0), and the released grc-joint
bundle is CC BY-SA 4.0, fetched to the cache, never bundled, so the wheel
itself stays Apache-2.0. The bundled Aegean sign data is from the Unicode
Character Database (Unicode licence); the Linear A corpus JSON is GORILA via
mwenge/lineara.xyz (Apache-2.0), with the facsimile imagery referenced, never
redistributed. The per-source rights table, including the NonCommercial DAMOS /
SigLA corpora and the CC0 Nestle 1904 NT, is on
Data & Provenance.
Every dataset pyaegean can touch is versioned and hashable. data.versions()
returns a manifest with package, bundled (each JSON file hashed straight from
the installed wheel), and fetched (each remote asset's pinned URL, pinned
sha256, licence, and cached state). Matching sha256s mean byte-identical data.
import json, aegean
from aegean import data
with open("data-versions.json", "w", encoding="utf-8") as f:
json.dump({"package": aegean.__version__, "data": data.versions()}, f, indent=2)aegean data versions --json > data-versions.jsonTo pin an analysis for a paper, record aegean.__version__ and dump this manifest
alongside your results. Loaded literary works additionally record the exact
upstream commit as Provenance.data_version, and every Corpus carries a
Provenance that stamps exports and produces a citation of the exact subset
you used. Cite the underlying edition, not pyaegean's wrapper: the mechanics are
on For Specialists and Data & Provenance.
- Greek NLP: the pipeline, the evaluation call sites, and the tier switch (baseline → treebank → neural)
- Benchmarks: the full protocol, the field's published numbers, and citations
- Data & Provenance: every dataset, licence, and the cache layout
- For Specialists: auditing, citing, and correcting a result
- Limitations: the candid register of what the toolkit can and cannot claim
Start here
Aegean scripts
Greek
Capabilities
Evaluation & methodology
Reference