API-CLI alignment of all processing cases

v1.6.0-beta: the command line and the service, case by case

What changed (atrium-project#72)

• Both entry points read one list of licence components per method, so a KeyBERT run's paradata names the KeyBERT model's licence from the command line too.
• In a batch, a record that fails to write costs only its own document; before, it dropped the rest of the batch.
• The service's default method and keyword count come from kw_config.txt, as the command line's; DEFAULT_KW_METHOD still overrides the method. The shipped values are unchanged (KeyBERT, 20).
• A command-line run started outside the checkout records the tool version and licences (it recorded unknown).
• docs/usage.md has one table of what still differs between the two entry points, and why; the flag table and the spec's descriptions are corrected.

Compatibility

• The API is unchanged apart from the version.

Upgrade notes

• Pin 1.6.0-beta.

Images this release certifies (images.json, atrium-project#81):

• ghcr.io/ufal/atrium-keyword-extract:1.6.0-beta = sha256:05f445fad41ad9974507098cfb7053ff4648b9d932df1d80839009f865e67936
• ghcr.io/ufal/atrium-keyword-extract-api:1.6.0-beta = sha256:df7d902f5e2f3630e6e6fd5385b7da2f0d1d3672d076ea1463edc4d15d81467c
• ghcr.io/ufal/atrium-keyword-extract-llm:1.6.0-beta = sha256:cf79afce3c21102064f39a43db683de05674f9d0da5acfcc660243b4461c3ca9

Full Changelog: v1.5.1-beta...v1.6.0-beta

API-CLI alignment and citation updated

v1.5.1-beta: the release v1.5.0-beta did not get

What changed

• The v1.5.0-beta tag pushed its images, but the release refused the vocabulary because its sidecars were stamped v1.4.0-beta. They are rebuilt (same 4,711 terms), and tests/test_vocab_release_stamp.py checks the stamp on every push. This release carries the vocabulary asset atrium-vocabulary-v1.5.1-beta.zip.
• The licences page now says that the paradata's license is a summary of the components used, not the licence of the keywords (atrium-project#6).
• The legacy method is described as what it is: a lemma-frequency count named after ÚFAL's KER, cited by its LINDAT record.
• YAKE stays declared AGPL-3.0, following the LICENSE file the package ships.
• Everything from v1.5.0-beta is included:
  • one input module (record, text, CoNLL-U, TEITOK, line CSV) for the command line, the service and the batch clients;
  • --kind, per-page keywords and record output on the command line;
  • the additive file part on /extract_keywords.

Compatibility

• The API is unchanged apart from the version. The vocabulary content is unchanged.

Upgrade notes

• Pin 1.5.1-beta. The 1.5.0-beta images exist but have no release or vocabulary asset.

Images this release certifies (images.json, atrium-project#81):

• ghcr.io/ufal/atrium-keyword-extract:1.5.1-beta = sha256:9e59448f8dfcde4a13a89b28d768aa869215800420ca75190ea9e9c95748f395
• ghcr.io/ufal/atrium-keyword-extract-api:1.5.1-beta = sha256:dee8139dd3aa9d1a87e4231cb5496e3b1e2c2ae91083edbf264c078b764f3283
• ghcr.io/ufal/atrium-keyword-extract-llm:1.5.1-beta = sha256:79111547a77c7035b0d67878f1fc26cd5a428086e6d583df79bbf9a8f2d0e645

Full Changelog: v1.4.0-beta...v1.5.1-beta

Minor changes for LLM-based KE and vocabulary

What's new

• Fewer forced labels in the controlled kind (#2). Three prompt rules, each behind a flag and on by default: copy the category from the list, pick the most specific term the line supports, and how to score confidence. A category given as one half of a composite term (jímka for jáma odpadní/jímka) is recorded as the term, from the reviewed data_samples/vocab/composite_halves.csv (ACCEPT_COMPOSITE_HALVES). /info names the rules in force and counts the halves.
• score_categories.py: missing lines, forced labels, confidence spread and agreement per run, and a re-run against its earlier run.
• Research default MODEL_KEY = qwen-3.8-27b-it: the 19 October re-runs (data_samples/KW_PER_DOC_LLM_R3_*) leave Qwen3.8-27B with no forced label.
• Vocabulary: Markdown at the edge of a TEATER label is dropped (**suborál), so atrium-vocabulary-v1.4.0-beta.zip differs from v1.3.0-beta's; atrium-translator picks it up when it next builds from a release.
• service/README.md is an operator runbook (ufal/atrium-project#77), with known limits: forced labels, uncalibrated confidence_score, research output.

Compatibility

Additive spec. The controlled kind's output is still research output: there are no gold labels yet.

Full Changelog: v1.3.0-beta...v1.4.0-beta

Minor API and LLM run changes + new wave of results shared

What's new

• Record schema declares enrichment.items[].teater_category_ids (ufal/atrium-project#51). These are the AMČR/TEATER record ids the controlled kind already writes. service/openapi.json carries the new schema and its sha256.
• The RO-Crate names vocabulary concepts by their source URIs. Each id in teater_category_ids becomes a DefinedTerm at https://api.aiscr.cz/id/<id> or https://teater.aiscr.cz/id/<id>, in the amcr-harvest / teater-harvest sets. teater_category is no longer minted as w3id.org/atrium/theme/<label>: it is a source label, not an ATRIUM theme.
• Controlled kind, remote line path (#2). Before, a reply that started the category with the other case of a label's first letter (Keramika) failed validation and the line was dropped. It now maps to the vocabulary spelling. The schema sent to OpenRouter and Ollama is unchanged.
• Research GPU path (llm_run.py, the llm image; #2). It now enriches documents on both backends:
  • vLLM's structured-outputs API; the call signatures fixed;
  • the prompt fitted to the window the model is served with, with the target line kept and the cut recorded in the paradata;
  • the shared prompt prefilled once (REUSE_PROMPT_CACHE) and in chunks (PREFILL_CHUNK_SIZE);
  • lm-format-enforcer's memory freed after each line; the label enum indexed by prefix (35× faster);
  • one BOS token; YaRN for the 32k Qwen models; AWQ on vLLM; VLLM_BATCH_SIZE;
  • --download to fill the Hugging Face cache for offline nodes;
  • the category enum accepts either case of a label's first letter;
  • paradata records config.prompt: the prompt's sha256, terms offered, window, category_case_folding.
• New models for 2× 24 GB: Gemma 4 12B / 31B (+ QAT), Qwen 3.8 27B (+ FP8), Qwen 3.5 / 3.6 MoE (GPTQ, FP8), Granite 4.1 8B / 30B. requirements_llm.txt adds accelerate and flash-linear-attention.
• Vocabulary: the AMČR country list's sub-header reads země (was zeme). The terms are unchanged (4,711). Shipped as atrium-vocabulary-v1.3.0-beta.zip with its .sha256.
• Sample runs: 17 models over 19 documents in data_samples/KW_PER_DOC_LLM_<model>/. They are not a score: they predate the case fix, ten carry forced labels, and the gold set and scorer (D2) are still open.
• Shared modules of 13 October. api_util/bbox_scale.py re-vendored from atrium-nlp-enrich (PDF points).

Compatibility

The API changes are additive: one new record field, and a 502 description that now says when to retry. Crate consumers that read teater_category as an ATRIUM theme URI now get the AMČR/TEATER concept URIs instead. The controlled kind's output is still research output.

Full Changelog: v1.2.0-beta...v1.3.0-beta

Added statistical keywords extraction functional

What's new

• Statistical keywords in the record (ufal/atrium-project#73). POST /extract_keywords writes the keywords block: the document's and each page's keywords, each with method, score and rank. With kind=both, keywords and enrichment go into one record and stay separate.
• Vocabulary asset (ufal/atrium-project#72). atrium-vocabulary-v1.2.0-beta.zip (CC0) and its .sha256 hold the nested facets, flat harvests, union.skos.ttl, provenance sidecars and vocabulary.csv.
• Vocabulary layout (#2): the A1 facet layout and the malta split; 4,711 terms.
• Shared modules of 9 October.

Compatibility

Additive. A record sent with the statistical kind now comes back as document_json; before, that happened only when the controlled kind ran.

Full Changelog: v1.1.0-beta...v1.2.0-beta

LLM-based keywords extraction controlled by vocabulary functionality reproduced

Summary

v1.1.0-beta moves the controlled kind into the service. POST /extract_keywords and POST /extract_keywords_text now run a language model over the AMČR and TEATER vocabularies and write the result into the record's enrichment block. The model runs in an inference service, OpenRouter or a local Ollama, chosen by LLM_BACKEND. The statistical kind (KeyBERT, YAKE, KER) is unchanged.

In v1.0.0-beta the controlled kind was a research target, and the service reported it skipped (kind=both) or refused it with 501 (kind=controlled).

This release merges the LLM engine of atrium-llm-enrich (atrium-digital-convert 31534d5, merged 2026-10-07) into this repository. The engine and the vocabulary code now exist once (atrium-keyword-extract#1, #2; atrium-project#72).

Note

Pre-release. The controlled kind's quality is not evaluated yet: the evaluation rubric (D1) of atrium-keyword-extract#2 is open. Treat its output as research output.

Full changelog: v1.0.0-beta...v1.1.0-beta

---

Highlights

• Controlled keywords in the service. Per line: the vocabulary term that describes it, the Czech and English keywords found in it, a confidence score, and the AMČR/TEATER record ids behind the term.
• Record write-back. The response's document_json is the record sent, with the enrichment block and entities[].pid written and every other block as it came.
• Two inference backends. OpenRouter (default) and Ollama, behind one shared engine (llm_client_shared.py) that needs no GPU stack.
• One prompt, three senders. The service, the batch clients and the research GPU run (llm_run.py) all render prompts/system_prompt.txt under the same llm_config.txt. A test holds the untruncated prompt byte-identical to prompt_full.txt.
• Live-backend CI lane. controlled-live.yml runs the controlled kind against a real OpenRouter model.
• Production image carries the controlled kind and no model weights.

---

Service

The controlled kind

• Each line that passes the quality filter is sent to the model on its own. The filter drops lines labelled Trash, Garbage, Inverted or Empty, and lines too short to read.
• A line is marked inside its neighbours on the same page, with the document's first lines and the current heading as context.
• The model answers in a JSON schema whose teater_category is an enum of the vocabulary's terms (plus a Nerelevantní (meta-text) marker for lines that are not archaeology), so it cannot invent a term.
• A homonym the vocabulary build qualified (zámek (sídlo elity)) comes back as its plain label. Its teater_category_ids tell the senses apart.
• entities[] rows of the record whose lemma or surface is a vocabulary term get a pid with the term's AMČR URI and its AAT exact match.
• /extract_keywords_text with the controlled kind reads the text's non-empty lines as one page.

Field	Meaning
teater_category	the vocabulary term describing the line
teater_category_ids	every AMČR/TEATER record the term stands for ({source, id}); omitted with EMIT_CATEGORY_IDS=false in llm_config.txt
extracted_keywords_cs / _en	the terms found in the line, in Czech and their English translations, in the same order
confidence_score	the model's confidence in teater_category, 0–1
page, line, citation	where the line came from

Response and API changes

All changes to openapi.json are additive. No path, schema or property was removed.

• kinds[] has one entry per kind asked for: ok, skipped (not available here) or failed (the LLM backend failed every call; the statistical keywords still come back).
• New response properties: enrichment, controlled (backend, model, outcome, per-run stats), document_json and document_json_schema_error.
• controlled.outcome separates contributed, empty (the model was asked and found nothing; an empty enrichment block is still written) and not-asked (no line passed the filter; nothing written).
• New 502 response on both extract endpoints: the LLM backend failed every call.
• GET /info gains a controlled block: ready or why not, backend, model, vocabulary, prompt.
• GET /ready stays 503 until the controlled kind's vocabulary and prompt have loaded, when it is configured.
• New schemas: AtriumDocument (and its bbox, entity_ref and line_ref parts), CategoryId, ControlledInfo, ControlledPrompt, ControlledRun, ControlledVocabulary, Enrichment, EnrichmentItem.
• New limits_applied notes:
  • vocab_prompt_budget_tokens (trimmed): vocabulary terms left out of the prompt.
  • llm_max_new_tokens (skipped): a reply cut at the token limit.
  • llm_max_consecutive_errors (stopped): a document given up after too many failed lines.
• The service refuses to start its controlled kind when the configured guardrail wording contradicts the vocabulary build (PROMPT_GEO_GUARDRAIL against taxonomy_config.json). /info controlled.detail says why.
• The line filter for the statistical kind now also leaves out Garbage and Inverted lines.

Configuration

Variable	Default	Meaning
LLM_BACKEND	openrouter	openrouter or ollama
LLM_CONFIG	llm_config.txt	config for what the environment does not set (PROMPT_* flags, VOCAB_PATH, EMIT_CATEGORY_IDS, the line filter)
OPENROUTER_API_KEY	none	secret; required for the controlled kind with openrouter
OPENROUTER_MODEL	none	required with the key, e.g. openai/gpt-4o-mini
OPENROUTER_SITE_URL, OPENROUTER_APP_NAME	none, atrium-keyword-extract	optional attribution headers
OLLAMA_HOST	http://localhost:11434	the Ollama server; compose supplies http://host.docker.internal:11434
OLLAMA_MODEL	none	required with ollama, e.g. qwen2.5:14b
VOCAB_PATH	data_samples/vocab/union_nested.json	the built vocabulary; flat files beside it resolve entities[].pid
LLM_TIMEOUT	300	seconds per LLM call; a timeout is retried
LLM_MAX_RETRIES	3	attempts per call; only timeouts, connection errors, 429 and 5xx are retried
LLM_MAX_NEW_TOKENS	2048	reply token limit; a reply cut at it is never used
LLM_MAX_CONSECUTIVE_ERRORS	10	failed lines in a row after which the document is given up
LLM_CONTEXT_WINDOW	128000 (openrouter), 32000 (ollama)	sizes the vocabulary prompt; sent to Ollama as num_ctx

Without these variables the service still starts and serves the statistical kind. kind=controlled then answers 501, kind=both reports it skipped, and /info controlled.detail names what is missing.

---

Batch clients and research path

• New: openrouter_client.py and ollama_client.py, over the shared llm_client_shared.py.
  • They read records, CSV or TEITOK (per line) and .md/.txt (per document).
  • They write <doc_id>_enriched.json, and the record itself with --document-json-out.
  • Dependencies are in the new requirements_remote.txt (pydantic, requests, tqdm, jsonschema).
• llm_run.py (research GPU path, llm Docker target):
  • Paradata now names the program keyword-extract, not nlp-enrich.
  • Secrets in the config are redacted from published paradata.
  • A prompt cut to the model's input budget is recorded as a llm_context_window trimmed limit note.
  • --help works.
• requirements_llm.txt no longer pins torch. vLLM pins torch exactly, so the old torch==2.7.0 made the file unsatisfiable next to recent vLLM releases.

---

Docker

• The api image (ghcr.io/ufal/atrium-keyword-extract-api, the production image) now includes the controlled kind's code and the built vocabulary files. It carries no model weights.
• .github/production-image.json and .dockerignore were updated to match.
• docker-compose.yaml maps host.docker.internal and defaults OLLAMA_HOST to the host's Ollama, so the Ollama backend works from a container.
• Images are built by docker.yml on the tag push.

---

Shared document record

atrium_document.py and atrium_document.schema.json were synced from the hub for the digital-convert route (atrium-project#71, atrium-digital-convert#4):

• New page-level field text_layer: the converter's verdict on the embedded text layer, one of digital, garbled, ocr, none, blank.
• Scoring co-contribution: ocr-postprocess may add line-quality fields to a plane another tool originated, without being treated as a second originator.
• OCR hand-off: pages flagged needs_ocr can have their lines replaced by the OCR run (replace_page_rows(), ocr_handoff_pages()), per page.

---

CI and tests

• New workflow controlled-live.yml: runs tests/test_controlled_live.py against OpenRouter. Triggers:
  • weekly, Mondays 05:00 UTC;
  • on demand, with an optional model input;
  • on pushes to test that touch what the model is sent or how its reply is read.
  • It uses the repository's OPENROUTER_KEY secret. The default model is openai/gpt-4o-mini. A run is two calls of about 50k prompt tokens each.
• The test never runs by default: it skips unless ATRIUM_LIVE_BACKEND=1.
• New test modules for the shared engine, both clients, the offload budget, the record write path (/extract_keywords returning document_json) and the live lane. The API contract, limits and env-contract tests are extended to the controlled kind.
• tqdm added to requirements-test.txt, so the contract lane that installs only test dependencies runs the controlled-kind tests.
• agent_dev_logs digests and plans refreshed; issues re-indexed.

---

Upgrade notes

• **Callers that omit `k...

Keyword extraction split from nlp-enrich repo - cleaning done

The first release of ufal/atrium-keyword-extract: keyword extraction moved here from atrium-nlp-enrich (keywords.py) and atrium-llm-enrich (the LLM and vocabulary code), as agreed on 30 September (#1, atrium-project#72). The history of the moved files is kept.

The service

• POST /extract_keywords takes the ATRIUM document record (document_json, after nlp-enrich); POST /extract_keywords_text takes text in a JSON body.
• Statistical keywords: KeyBERT (default), YAKE, and KER (legacy, needs lines[].lemma/upos in the record). Every keyword carries its method, score and rank.
• Keywords for the document and, by default, for each of its pages.
• kind=statistical|controlled|both (default both). The controlled kind (LLM over AMČR and TEATER) is not in the service yet: kind=both reports it as skipped, kind=controlled answers 501.
• Every success carries the run's CreateAction as paradata (atrium-project#71). No record block is written yet: the keywords block is atrium-project#73.
• /info, /health, /ready (it turns ready once the KeyBERT model is loaded), the shared error body, and these limits: MAX_UPLOAD_MB 10, MAX_DOCUMENT_WORDS 200000 (413), MAX_KEYWORDS 100 (422), MAX_CONCURRENT_REQUESTS 2 (429 busy), KEYBERT_CHUNK_WORDS / KEYBERT_CHUNK_OVERLAP.
• Service id atrium-keyword-extract; the spec declares x-atrium-service-previous: atrium-nlp-enrich. Program id keyword-extract, which the shared contract accepts as the successor of llm-enrich.

Images

• ghcr.io/ufal/atrium-keyword-extract-api: the production service.
• ghcr.io/ufal/atrium-keyword-extract: the batch CLI, keywords.py.
• ghcr.io/ufal/atrium-keyword-extract-llm: the research target for the controlled kind (llm_run.py, vocab_*.py, prompts/). Not on the release line.

Licences

KeyBERT (MIT) with sentence-transformers (Apache-2.0); YAKE is AGPL-3.0 and is recorded only on runs that use it; the AMČR and TEATER vocabularies are CC0.

Assets

openapi.json and openapi.json.sha256.

Not in this release

The controlled kind in the service, the keywords record block (atrium-project#73), and the vocabulary as a release asset. The LLM code still has a second copy in atrium-digital-convert until that repository drops it.

Full Changelog: https://github.com/ufal/atrium-keyword-extract/commits/v1.0.0-beta