Skip to content

Releases: pbisson44/ddigraph

Release list

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 16 Aug 01:54
Immutable release. Only release title and notes can be modified.
5226909

Makes the RDF story real. The package advertised five graph backends and
shipped one, and the RDF surface that did exist was spread across four
mutually inconsistent namespaces and three predicate conventions, so
nothing it produced could be joined to anyone else's data. This release
settles the vocabulary, gives every DDI flavor one graph shape, and
removes the CLI verbs deprecated in 0.4.0rc1.

This entry grows as the release lands; it is not yet dated or published.

Added

  • Backend-neutral graph view (ddigraph.graph.view). iter_graph()
    streams any DDI file as GraphChunk values built from the existing
    Node and Relationship dataclasses, so exporters, previewers and
    validators target one shape instead of three. Previously only
    DDI-Codebook had such a projection, which is why every demo/load_*.py
    script works on DDI-L alone.

  • DDI-CDI reaches the graph tier for the first time. It was
    parse-only: api.aload raises NotImplementedError for it, no adapter
    writes it, and the CLI has no cdi format choice, so a parsed CDI file
    had nowhere to go. It now projects to the same nodes and relationships
    as the other two flavors.

  • A defined RDF vocabulary (ddigraph.rdf.vocabulary). One project
    namespace, versioned independently of the package, aligned to the DDI
    Alliance's own published RDF work: DISCO for
    Study/Variable/Question/Universe/DataFile, SKOS for code lists and
    categories, XKOS for classification levels. Neither vocabulary was
    referenced anywhere in this repo before. Every node carries two
    rdf:type triples -- the standard class for interoperability and the
    project class for identity -- because the standard alignment is
    many-to-one and could not otherwise be reversed.

  • ddigraph export, the first command that writes a file rather than
    loading a database. Emits Turtle, N-Triples, JSON-LD, RDF/XML, JSON or
    CSV, needs no Neo4j connection, and works for all three DDI flavors:

    ddigraph export survey.xml --format turtle -o out.ttl

    RDF formats need the [rdf] extra; JSON and CSV work on a base
    install. Also available as ddigraph.export() from Python, alongside
    ddigraph.iter_graph() for building your own consumer.

  • RDF as an input format (ddigraph.rdf.read_graph). Turtle,
    N-Triples, JSON-LD and RDF/XML parse back into the same GraphChunk
    stream the XML parsers produce, so everything built on the graph view
    consumes them unchanged. The round trip is lossless at triple level:
    exporting a fixture, reading it back and re-exporting reproduces the
    original graph exactly, for all four fixtures and all four
    serialisations.

    Two things make that possible, and neither is an accident. Every node
    already carried a project-namespace rdf:type beside its standard
    class. Relationships needed the same treatment: three published
    predicates are reached by more than one relationship type
    (disco:question, skos:inScheme, dcterms:isPartOf -- nine of 369
    types), and skos:inScheme also reverses the graph's edge direction,
    so those now carry a project-namespace companion triple. The other
    published predicates are one-to-one and get none, keeping the extra
    triples to the cases that need them.

    The reader skips subjects with no project type rather than guessing, so
    pointing it at unrelated RDF yields nothing instead of nonsense, and a
    mixed graph still yields the part it understands.

  • ddigraph load accepts RDF, closing the loop:

    ddigraph export survey.xml --format turtle -o out.ttl
    ddigraph load out.ttl

    This needed a Neo4j writer over GraphChunk
    (ddigraph.graph.writer.GraphChunkWriter). Neo4jGraphAdapter takes a
    DDIIngestGraph, which only the codebook parser produces, so RDF and
    DDI-CDI had no way into a database. The new writer groups a chunk by
    label and identity shape and issues one UNWIND per group, so it needs
    no knowledge of the schema -- and it gives DDI-CDI its first write
    path
    . Labels and relationship types are validated before being
    interpolated into Cypher, because Neo4j cannot parameterise them and an
    RDF input's labels come from a file someone else wrote.

  • Code lists and categories are emitted as SKOS. A CodeList becomes
    a skos:ConceptScheme and a Category a skos:Concept, with
    skos:prefLabel, skos:notation and skos:definition. Membership is
    emitted from the member's side as skos:inScheme, inverting the graph
    edge, because skos:member belongs to skos:Collection rather than
    skos:ConceptScheme. external_references becomes skos:exactMatch,
    which is the hook for joining a code list to EuroVoc, DBpedia or any
    other published vocabulary.

  • ddigraph shapes, writing SHACL derived from DDISchema -- the same
    table that generates the Neo4j constraints, so the shapes cannot drift
    from the data:

    ddigraph shapes -o shapes.ttl --flavor lifecycle

    Every exported fixture is validated against them with pyshacl in the
    test suite, which holds the vocabulary and the writer to the contract
    consumers are asked to validate against. --flavor is recommended for
    real data: 21 labels appear in more than one DDI flavor with different
    identity fields, and constraints the flavors disagree on are dropped
    rather than guessed at.

  • ddigraph preview, answering "what is actually in this file?"
    without a database and without an optional extra:

    ddigraph preview survey.xml --format html -o preview.html

    Until now the only way to see what a load had produced was to open
    Neo4j Browser and start writing Cypher; ddigraph load reports
    nodes=1247, relationships=3891 and nothing about what any of them
    are. Preview reports the shape -- counts per node type and per
    type -[EDGE]-> type -- because the demo corpus runs to 65 MB and a
    box per node is unreadable. --limit N adds example identities when
    the counts alone do not tell you whether the right thing was parsed.

    Three renderers: text for the terminal, mermaid to paste into the
    docs or a GitHub comment, and html as one self-contained page with an
    inline SVG chart -- no CDN, no external stylesheet, no JavaScript, so
    it works offline and survives being emailed.

  • ddigraph validate, checking a file against the official DDI XSD:

    ddigraph validate survey.xml || exit 1

    The package has shipped the official schemas all along -- 154 XSD files
    across Codebook 2.6, Lifecycle 3.1/3.2/3.3 and CDI 1.0 -- and only the
    build-time codegen ever read them. Nothing let a user ask the question a
    data archivist asks first. It picks the schema from the flavor and, for
    DDI-L, from the version the document declares in its own namespace, and
    exits non-zero on a violation. load and export take --validate to
    run the same check as a pre-flight. No new dependency: lxml was already
    required and covers XSD 1.0, which is what the DDI schemas are.

    It is opt-in, and the reason matters. Every XML fixture in this
    repository fails validation -- they are synthetic, and the Codebook one
    is a bare <codeBook> with no namespace at all -- as does a good deal of
    published DDI. Those files parse and load correctly. Validating by
    default would refuse work that currently succeeds.

    It also required working around a defect in the DDI Alliance's own
    Codebook 2.6 schema, which is not itself valid XSD: in 55 places an
    xs:attribute holds its xs:annotation after its xs:simpleType, while
    the specification requires (annotation?, simpleType?). Every conforming
    parser rejects it, so without intervention Codebook could not be
    validated at all. The file's checksum matches schemas/manifest.json, so
    this is upstream rather than a vendoring accident. ddigraph reorders the
    annotations in the in-memory tree and leaves the file byte-identical, and
    a test asserts the repair changes no element and no name.

  • --include-cdi on ddigraph bootstrap, for pre-provisioning the
    DDI-CDI schema when CDI data is written by something other than
    ddigraph.

  • The vocabulary namespace resolves. vocabulary.py described the
    namespace IRI as dereferenceable, and it was not: nothing was served at
    it, so every IRI in every exported file pointed at a 404. There is now a
    reference page at https://pbisson44.github.io/ddigraph/ns/1.0/ and a
    vocabulary.ttl beside it, defining 249 classes, 369 object properties
    and 85 datatype properties.

    It is generated from DDISchema -- the same table behind the Neo4j
    constraints and the SHACL shapes -- so it cannot describe terms the
    exporter does not emit, and a test fails if the committed copy drifts.
    A hand-maintained mapping table that no code implemented is what this
    release started out fixing; a hand-maintained vocabulary document would
    have been the same bug one level up.

    Terms with a published equivalent are declared against it rather than
    redefined (rdfs:subClassOf disco:Question, not owl:equivalentClass,
    because the alignment is many-to-one). The three predicates whose graph
    direction is opposite to the published one are declared owl:inverseOf
    rather than rdfs:subPropertyOf, since calling them subproperties would
    tell a reasoner the scheme is in the concept. The document asserts
    nothing about DISCO, SKOS or XKOS terms themselves.

  • shacl extra (pip install "ddigraph[shacl]"), pulling rdflib
    and pyshacl.

  • tests/fixtures/cdi_sample.xml, a small materialised DDI-CDI file.
    The demo corpus lives in Git LFS and is not materialised in CI, so the
    suite needed its own.

Changed

  • xmlschema is no longer a runtime dependency. It appears nowhere
    under src/; its only use is scripts/generate_schema_definitions.py,
    the XSD codegen, which is not shipped in the wheel. Every base install
    was pulling it, and `...
Read more