v3.0.0

VCF-RDFizer v3.0.0

This is the largest release since the project began. It retargets the conversion to a new vocabulary, adds a semantic validation suite that proves the RDF still reproduces its source VCF, adds a data-linking subsystem, and makes the conversion aware of VCF versions 4.1 through 4.5.

The major version bump reflects a breaking change to the emitted graph: every IRI moves to a new namespace.

Breaking: the emitted graph moved to the VCF Core vocabulary

Every emitted IRI moves from vcfr: (https://w3id.org/vcf-rdfizer/vocab#) to vcfc: (https://w3id.org/vcf-core/vocab#). The vocabulary was renamed VCF Core and moved, because it is a semantic target any conversion system can adopt and its name should not carry the name of one converter.

The old namespace is not redirected. It now serves a deprecation document linking every former term to its successor. Graphs produced by v2.1.0 and earlier will not match queries written against v3.0.0 output, and vice versa.

One term was renamed rather than re-namespaced: vcfr:DenseRepresentation is now vcfc:ExpandedRepresentation.

VCF Core models considerably more of VCF 4.5 than its predecessor, so the mapping rules and the wrapper's emitters were rewritten to cover it rather than merely re-prefixed. No version IRI is pinned anywhere — the converter targets the namespace, and VCF Core's release line is independent of this one's.

Added

Semantic validation (--mode validation, --validate). Compare a source VCF against its RDF and get an answer you can defend in a paper. Thirteen SPARQL queries plus preflight checks cover record density, variant shapes, Ti/Tv, FILTER distribution, file metadata, header census, predicate and class censuses, and record/INFO/FORMAT identity digests. Coverage is measured, not asserted: a mutation-testing harness proves that a named corruption would actually be caught, and known gaps are assertions that fail when closed rather than comments.

Multi-engine validation and benchmarking. --validation-engine accepts comunica, qlever, hdt, cottas, all, or a comma-separated list. Several engines answer the whole query set in one run, are cross-checked against each other for agreement on the real graph, and are timed into benchmark.csv. hdt and cottas query the compressed artifact in place — no decompression round-trip.

Version-aware conversion for VCF 4.1 through 4.5. --vcf-version {auto,4.1,…,4.5}, defaulting to auto, which reads each input's ##fileformat line per input — a directory of mixed-version VCFs is handled correctly. Version-sensitive semantics follow suit: CIPOS/CIEND tuple arity, EVENT, and the Number-code and LA/M families. Families a version does not define are no longer invented, and VCF 4.0 is correctly reported as unrecognized.

Data linking. Three runnable plug-in tiers — rsid-dbsnp declarative token links, gene-demo with a digest-pinned synthetic GFF3 bundle, and a batched rsid-ensembl live resolver. Available as --link in full mode, as a Docker-free --mode link --rdf, and through vcf-rdfizer-link list|keys|init|check|dry-run|run for authoring. Includes Turtle manifests, disk-backed key deduplication, interval joins, assembly refusal, atomic side-graphs, provenance and per-linker metrics, plus cached HTTPS sessions with per-host pacing, request ceilings, Retry-After handling, offline replay and contact headers.

Condensed sample representation (--sample-representation) for large multi-sample cohorts, with a streaming COTTAS merge that writes the final file incrementally rather than building a graph-wide hash table.

--mode index regenerates an artifact's query index in place, for HDT and COTTAS.

--strict-conformance, --shacl-shapes for conformance against VCF Core's published per-version SHACL overlays, --quiet and --no-progress, and --validation-time-budget.

A documentation set. Fourteen documents under docs/ covering architecture, conversion, representations, validation and its methodology, VCF coverage, custom RML mappings, data linking, sample representations, limitations and roadmap — plus a design proposal for granular, machine-readable privacy policies over the VCF graph (ODRL profile with graph selectors, three enforcement tiers, GA4GH DUO consent codes). The privacy work is a proposal, not implemented.

Changed

• --rdf-storage-mode now defaults to space-optimized instead of being required in full mode. Paired benchmarking showed it reaches the same triple count at a much smaller peak workspace footprint, with no practically important change in wall time, CPU time, or peak memory. plain remains available and unchanged.
• --hdt-strategy single is documented as a verification path, not a faster alternative. There is no size regime where it wins; its purpose is producing a one-shot HDT so the chunk-and-merge result can be checked against it.
• Ordinals serialize as xsd:integer, not xsd:positiveInteger, so they satisfy the SHACL shapes' exact datatype comparison. Affects vcfc:sampleIndex among others.
• --header-representation basic no longer produces a conformant graph — structured is what the SHACL profile requires.
• QUAL accepts the full VCF Float lexical space.
• rdflib is imported where it is used, so the tool runs — including --help — without the data-linking dependency installed.
• rdflib is now a runtime dependency for Turtle/N-Triples parsing; rich remains optional.
• The PyPI source archive no longer ships a partial test suite. setuptools' default walk collected test/*.py without the support modules and fixtures they import, so the tests in the published sdist could not run at all. The suite runs from a git checkout, which is what the workflows assume.
• changelog.md has been retired. Per-version notes now live on the releases page.

Fixed

• A ##META allowed-value list kept only its first member. ##META=<ID=Assay,Type=String,Number=.,Values=[WholeGenome, Exome]> emitted a single vcfc:metaAllowedValue, whose object was "[WholeGenome" — opening bracket included, Exome lost. The structured-header parser split attributes on every unquoted comma, including the ones inside the bracketed list; the fragments it produced carry no =, so they were discarded rather than reported. This was silent: the emitter and the validator's oracle call the same parser, so the expectation was wrong in exactly the same way the graph was, and validation agreed with itself. The parser now tracks bracket depth and splits only at depth zero. Single-value lists were always correct, and the vcfc:HeaderAttribute count is unchanged, so the census and its ordinals are unaffected.
• A failed decode leaked its stdout pipe. The compression validator's decoder ran outside a context manager, so a decode that raised while counting left a file descriptor open.
• --hdt-strategy single was silently ignored when COTTAS was selected. --representations hdt,cottas --hdt-strategy single ran the partitioned path for both while reporting a single label — benchmark runs using that combination were measuring the wrong thing. It is now refused with a message naming the remedy.
• Every declared INFO and FORMAT definition was emitted twice.
• QUAL was extracted from every VCF and never mapped into RDF; ##fileDate was never mapped either.
• Flattened tuple keys were never decomposed, and confidence-interval bounds were emitted as plain strings.
• A sites-only VCF got no representation profile.
• A bracketed CHROM now links to its assembly contig.
• A timed-out validation query no longer costs one timeout for every remaining query, nor orphans its engine process. An up-front warning fires when an unindexed engine meets a large graph.
• Comunica could not query an HDT file by path; @comunica/query-sparql-hdt is now in the image.
• The parser oracle was reading htslib's header rather than the file's.
• COTTAS: large-input merge reliability, SIGKILL/OOM handling, condensed index generation, a partition bug with large compressed intermediates, and a raised memory ceiling.
• CI never installed the project's declared dependencies, so six unit-test jobs failed.
• Temporary Docker volumes from the current run are now removed on interrupt.
• QLEVER tag for ARM architecture docker build

Upgrading

If you have graphs from v2.1.0 or earlier, reconvert them — do not mix pre-v3.0.0 and v3.0.0 output in one store. Rewrite queries from vcfr: to vcfc:, and replace vcfr:DenseRepresentation with vcfc:ExpandedRepresentation. Every former term's successor is listed in the deprecation document at the old namespace.

One further difference affects VCFs carrying a multi-value ##META declaration: v3.0.0 emits one vcfc:metaAllowedValue per declared member, where earlier versions emitted a single truncated one. A graph converted before this release under-reports them, so its triple count will not match a v3.0.0 reconversion of the same input.

pip install --upgrade vcf-rdfizer==3.0.0
conda install -c conda-forge vcf-rdfizer=3.0.0
docker pull ecrum19/vcf-rdfizer:3.0.0