Releases: pbisson44/ddigraph
Release list
v0.5.0
Makes the RDF story real. The package advertised five graph backends and
shipped one, and the RDF surface that did exist was spread across four
mutually inconsistent namespaces and three predicate conventions, so
nothing it produced could be joined to anyone else's data. This release
settles the vocabulary, gives every DDI flavor one graph shape, and
removes the CLI verbs deprecated in 0.4.0rc1.
This entry grows as the release lands; it is not yet dated or published.
Added
-
Backend-neutral graph view (
ddigraph.graph.view).iter_graph()
streams any DDI file asGraphChunkvalues built from the existing
NodeandRelationshipdataclasses, so exporters, previewers and
validators target one shape instead of three. Previously only
DDI-Codebook had such a projection, which is why everydemo/load_*.py
script works on DDI-L alone. -
DDI-CDI reaches the graph tier for the first time. It was
parse-only:api.aloadraisesNotImplementedErrorfor it, no adapter
writes it, and the CLI has nocdiformat choice, so a parsed CDI file
had nowhere to go. It now projects to the same nodes and relationships
as the other two flavors. -
A defined RDF vocabulary (
ddigraph.rdf.vocabulary). One project
namespace, versioned independently of the package, aligned to the DDI
Alliance's own published RDF work: DISCO for
Study/Variable/Question/Universe/DataFile, SKOS for code lists and
categories, XKOS for classification levels. Neither vocabulary was
referenced anywhere in this repo before. Every node carries two
rdf:typetriples -- the standard class for interoperability and the
project class for identity -- because the standard alignment is
many-to-one and could not otherwise be reversed. -
ddigraph export, the first command that writes a file rather than
loading a database. Emits Turtle, N-Triples, JSON-LD, RDF/XML, JSON or
CSV, needs no Neo4j connection, and works for all three DDI flavors:ddigraph export survey.xml --format turtle -o out.ttlRDF formats need the
[rdf]extra; JSON and CSV work on a base
install. Also available asddigraph.export()from Python, alongside
ddigraph.iter_graph()for building your own consumer. -
RDF as an input format (
ddigraph.rdf.read_graph). Turtle,
N-Triples, JSON-LD and RDF/XML parse back into the sameGraphChunk
stream the XML parsers produce, so everything built on the graph view
consumes them unchanged. The round trip is lossless at triple level:
exporting a fixture, reading it back and re-exporting reproduces the
original graph exactly, for all four fixtures and all four
serialisations.Two things make that possible, and neither is an accident. Every node
already carried a project-namespacerdf:typebeside its standard
class. Relationships needed the same treatment: three published
predicates are reached by more than one relationship type
(disco:question,skos:inScheme,dcterms:isPartOf-- nine of 369
types), andskos:inSchemealso reverses the graph's edge direction,
so those now carry a project-namespace companion triple. The other
published predicates are one-to-one and get none, keeping the extra
triples to the cases that need them.The reader skips subjects with no project type rather than guessing, so
pointing it at unrelated RDF yields nothing instead of nonsense, and a
mixed graph still yields the part it understands. -
ddigraph loadaccepts RDF, closing the loop:ddigraph export survey.xml --format turtle -o out.ttl ddigraph load out.ttlThis needed a Neo4j writer over
GraphChunk
(ddigraph.graph.writer.GraphChunkWriter).Neo4jGraphAdaptertakes a
DDIIngestGraph, which only the codebook parser produces, so RDF and
DDI-CDI had no way into a database. The new writer groups a chunk by
label and identity shape and issues oneUNWINDper group, so it needs
no knowledge of the schema -- and it gives DDI-CDI its first write
path. Labels and relationship types are validated before being
interpolated into Cypher, because Neo4j cannot parameterise them and an
RDF input's labels come from a file someone else wrote. -
Code lists and categories are emitted as SKOS. A
CodeListbecomes
askos:ConceptSchemeand aCategoryaskos:Concept, with
skos:prefLabel,skos:notationandskos:definition. Membership is
emitted from the member's side asskos:inScheme, inverting the graph
edge, becauseskos:memberbelongs toskos:Collectionrather than
skos:ConceptScheme.external_referencesbecomesskos:exactMatch,
which is the hook for joining a code list to EuroVoc, DBpedia or any
other published vocabulary. -
ddigraph shapes, writing SHACL derived fromDDISchema-- the same
table that generates the Neo4j constraints, so the shapes cannot drift
from the data:ddigraph shapes -o shapes.ttl --flavor lifecycle
Every exported fixture is validated against them with
pyshaclin the
test suite, which holds the vocabulary and the writer to the contract
consumers are asked to validate against.--flavoris recommended for
real data: 21 labels appear in more than one DDI flavor with different
identity fields, and constraints the flavors disagree on are dropped
rather than guessed at. -
ddigraph preview, answering "what is actually in this file?"
without a database and without an optional extra:ddigraph preview survey.xml --format html -o preview.html
Until now the only way to see what a load had produced was to open
Neo4j Browser and start writing Cypher;ddigraph loadreports
nodes=1247, relationships=3891and nothing about what any of them
are. Preview reports the shape -- counts per node type and per
type -[EDGE]-> type-- because the demo corpus runs to 65 MB and a
box per node is unreadable.--limit Nadds example identities when
the counts alone do not tell you whether the right thing was parsed.Three renderers:
textfor the terminal,mermaidto paste into the
docs or a GitHub comment, andhtmlas one self-contained page with an
inline SVG chart -- no CDN, no external stylesheet, no JavaScript, so
it works offline and survives being emailed. -
ddigraph validate, checking a file against the official DDI XSD:ddigraph validate survey.xml || exit 1
The package has shipped the official schemas all along -- 154 XSD files
across Codebook 2.6, Lifecycle 3.1/3.2/3.3 and CDI 1.0 -- and only the
build-time codegen ever read them. Nothing let a user ask the question a
data archivist asks first. It picks the schema from the flavor and, for
DDI-L, from the version the document declares in its own namespace, and
exits non-zero on a violation.loadandexporttake--validateto
run the same check as a pre-flight. No new dependency:lxmlwas already
required and covers XSD 1.0, which is what the DDI schemas are.It is opt-in, and the reason matters. Every XML fixture in this
repository fails validation -- they are synthetic, and the Codebook one
is a bare<codeBook>with no namespace at all -- as does a good deal of
published DDI. Those files parse and load correctly. Validating by
default would refuse work that currently succeeds.It also required working around a defect in the DDI Alliance's own
Codebook 2.6 schema, which is not itself valid XSD: in 55 places an
xs:attributeholds itsxs:annotationafter itsxs:simpleType, while
the specification requires(annotation?, simpleType?). Every conforming
parser rejects it, so without intervention Codebook could not be
validated at all. The file's checksum matchesschemas/manifest.json, so
this is upstream rather than a vendoring accident. ddigraph reorders the
annotations in the in-memory tree and leaves the file byte-identical, and
a test asserts the repair changes no element and no name. -
--include-cdionddigraph bootstrap, for pre-provisioning the
DDI-CDI schema when CDI data is written by something other than
ddigraph. -
The vocabulary namespace resolves.
vocabulary.pydescribed the
namespace IRI as dereferenceable, and it was not: nothing was served at
it, so every IRI in every exported file pointed at a 404. There is now a
reference page athttps://pbisson44.github.io/ddigraph/ns/1.0/and a
vocabulary.ttlbeside it, defining 249 classes, 369 object properties
and 85 datatype properties.It is generated from
DDISchema-- the same table behind the Neo4j
constraints and the SHACL shapes -- so it cannot describe terms the
exporter does not emit, and a test fails if the committed copy drifts.
A hand-maintained mapping table that no code implemented is what this
release started out fixing; a hand-maintained vocabulary document would
have been the same bug one level up.Terms with a published equivalent are declared against it rather than
redefined (rdfs:subClassOf disco:Question, notowl:equivalentClass,
because the alignment is many-to-one). The three predicates whose graph
direction is opposite to the published one are declaredowl:inverseOf
rather thanrdfs:subPropertyOf, since calling them subproperties would
tell a reasoner the scheme is in the concept. The document asserts
nothing about DISCO, SKOS or XKOS terms themselves. -
shaclextra (pip install "ddigraph[shacl]"), pullingrdflib
andpyshacl. -
tests/fixtures/cdi_sample.xml, a small materialised DDI-CDI file.
The demo corpus lives in Git LFS and is not materialised in CI, so the
suite needed its own.
Changed
xmlschemais no longer a runtime dependency. It appears nowhere
undersrc/; its only use isscripts/generate_schema_definitions.py,
the XSD codegen, which is not shipped in the wheel. Every base install
was pulling it, and `...