Generation Pipelines

Generation Pipelines

Three pipelines under tools/ turn the raw OMG sources in sources/<tag>/ into the knowledge base under knowledge/<tag>/. The .NET pipelines exist only to build what the plugin reads ahead of time; tools/spec-extract is the exception – it ships with an installed plugin (a marketplace install is a full tag-pinned checkout) and now actually runs there too, via uv (see below). All are driven by [[the hypha CLI|Hypha-CLI]], never by the test runner: no test writes into knowledge/, so dotnet test / pytest always leave the working tree clean.

Generation order {#order}

tools/knowledge-gen defines a small abstraction, IKnowledgeGenerator (Artifact, Order, GenerateAsync(tag)), registered as a collection so a full run is just a loop over Order:

Order	Artifact	Generator
10	metamodel	tools/metamodel-gen
20	grammar-references	GrammarReferenceGenerator
30	textual-notation	TextualNotationGenerator
35	model-library	ModelLibraryGenerator
37	spec	SpecGenerator (needs the OMG PDFs and provisions uv on demand – see [[below
40	cross-references	CrossReferenceGenerator (last, it reads the index and generated examples of everything before it, including spec's catalog when present)

A generator whose inputs are absent returns Skipped with a reason, so CI stays green without the copyrighted/optional inputs (spec PDFs, in particular) – spec degrades the same way when uv itself can't be provisioned (offline the first time, or an unsupported platform). A generator whose inputs are present but would yield wrong output throws instead of guessing, a missing metamodel index, a clause title reaching the cross-references, a slug collision.

tools/metamodel-gen

Reads the combined KerML/SysML v2 XMI and emits knowledge/<tag>/metamodel/, markdown per element plus a JSON sidecar (metamodel.json, index.json; see The Knowledge Base). It is a class library with no entry point of its own; it registers a MetamodelGenerator into the shared IKnowledgeGenerator collection, and hypha generate metamodel runs it.

• Projects: Hypha.MetamodelGen (XMI reading + markdown/JSON generation) and Hypha.MetamodelGen.Tests (NUnit).
• Key dependency: uml4net.xmi (Starion Group, Apache-2.0), which reads UML 2.5.1 XMI via XmiReaderBuilder.Create().Build().
• Test fixtures: Hypha.MetamodelGen.Tests/Fixtures/xmi/ holds one real release's XMI, captured from TestModel.FixtureTag, so the tests have something real to exercise the generators against without a network fetch. It is committed as a test fixture, not a distributed one – it lives under tools/, which is never shipped with the plugin, so committing it does not reintroduce a committed knowledge floor; it only means that one fixture release is no longer regression-tested against every other release.
• Golden files: Expected/ is the one "matches a known-good baseline" check left in the repository, re-blessed by hand after an intentional format change, via the [Explicit] Bless_expected_files tests.
• "Interesting" metaclasses are discovered via ModelInspector, never hardcoded.

tools/knowledge-gen

The larger of the two class libraries (namespace Hypha.Knowledge), covering grammar references, textual notation, the model library and cross-references, plus release discovery/fetching/installation that backs the hypha verbs. Source is organized by concern:

Module	Covers
CrossReferences/	CrossReferenceBuilder, the edge types (ClauseEdge, FeatureEdge, GrammarEdge, ExampleEdge), Provenance, and KnowledgeReader – which reads already-generated knowledge back in to build the cross-references
Generation/	The IKnowledgeGenerator abstraction itself, and the four concrete generators (CrossReferenceGenerator, GrammarReferenceGenerator, ModelLibraryGenerator, TextualNotationGenerator)
Grammar/	Parses the release's BNF (ClauseNumberComparer, GrammarParser, GrammarLinks, Production) – every grammar-to-metamodel link is stated by the grammar itself
ModelLibrary/	DeclarationScanner + ModelLibraryRenderer – scans the standard-library .sysml/.kerml sources and renders knowledge/<tag>/model-library/
Releases/	Release discovery, fetching, installing and removing (ReleaseCatalog, ReleaseFetcher, ReleaseInstaller, ReleaseRemover, CommitResolver) and the ReleaseWindowEvictor used only by hypha move-window, never by hypha check
Layout/	KnowledgeLayout – the knowledge/<tag>/... path conventions

Hypha.Knowledge.Tests covers all of the above with NUnit, including a [Explicit], TestCategory=Live fixture (LiveReleaseFetch) that really talks to GitHub, opt-in only, per Developers Experience.

tools/spec-extract {#tools-spec-extract}

The one Python component in the repository (Python ≥ 3.12), scoped purely to turning the OMG specification PDFs into markdown clauses. It reads sources/<tag>/specs/*.pdf and writes knowledge/<tag>/spec/{kerml,sysml2}/, both git-ignored, since the OMG license forbids redistributing the PDFs or their extracted text (see Sources and Licensing).

Two ways to run it, both calling the same spec_extract.pipeline functions:

• hypha generate spec --tag <tag> – what an installed plugin runs, and what makes spec-citation quoting work without a maintainer source checkout. SpecGenerator (Hypha.Knowledge/Generation/SpecGenerator.cs) fetches and caches uv from its own GitHub releases – mirroring how the SessionStart hook self-fetches the hypha CLI (see Skills and Agents#the-sessionstart-hook) – then runs uv run --project tools/spec-extract python -m spec_extract --repo-root <root> --tag <tag>. uv resolves or fetches a matching Python itself and installs this project's dependencies into a managed venv on demand, so neither a pre-existing Python nor a provisioned .venv is needed.
• pytest – still the maintainer-facing way to regenerate every installed release at once. Because its only output is git-ignored, this half stays deliberately test-driven: a test writing it can never dirty the repository. test_generate.py is what actually produces knowledge/<tag>/spec/ when the PDFs are present; tests skip cleanly when they are not.

Each layer is a pure function, unit-tested on synthetic data, with one skip-if-absent test exercising the real PDF:

Layer	Responsibility
pdf_reader	Accesses pdfplumber's positioned chars/words – page_count, extract_pages, extract_text, extract_words
layout	Assembles positioned words into reading-ordered lines: multi-column ordering, running header/footer/page-number stripping
clauses	Detects headings via a successor-numbering rule (skipping TOC entries and stray numbers), grouping body text with page ranges
normative	Splits normative prose from informative NOTE/EXAMPLE blocks, flagging shall/must clauses, losslessly
markdown	Renders one clause to deterministic markdown – front matter plus body, e.g. 07.04.02-concrete-syntax.md
pipeline	Wires the above: extract_document(pdf, DocMeta) → clauses, write_clauses(clauses, out_dir) → files

pdfplumber (MIT) was chosen over PyMuPDF (AGPL-3.0/commercial) for a clean Apache-2.0 fit, and because it exposes the positioned words/chars the clause-detection layer needs.

Running it

cd tools/spec-extract
python -m venv .venv && . .venv/Scripts/activate   # or source .venv/bin/activate
pip install -e .[dev]
ruff check .
pytest
pytest tests/test_clauses.py::test_detects_clauses_and_groups_body   # one test

Or, without a local Python install, via uv:

uv run --python 3.12 --extra dev pytest

python -m spec_extract --repo-root <path> --tag <tag> [--out-root <path>] is the non-pytest entry point SpecGenerator drives (--out-root defaults to --repo-root; a caller redirecting output, e.g. hypha generate --output, passes it explicitly so sources/<tag>/specs/ is still read from --repo-root while only the write target moves).

Freezing this environment into a distributable binary was considered and rejected: it would pull in ~104 MB / 2,910 files of native wheels (Pillow, cryptography, pdfminer), and PyInstaller/Nuitka are least reliable at exactly that weight. Shipping the interpreter fetch instead – uv resolves or fetches a matching Python itself, rather than a frozen build carrying one – sidesteps that weight entirely, and is what let spec-citation stop being maintainer-only.