Repository navigation
New tool: automated dataset-header validation & auto-fixing #337
benceszirbik
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
We now have a reusable tool for checking — and auto-fixing — the DCAT dataset headers on our instance files, merged into
cgmes-3.0_ncp-2.5_tc-2.0(and also to cgmes-3.0_ncp-2.4_tc-1.1`). It encodes the header rules from the recent review (#322) as a single source of truth, so the validator and the fixer always agree (rather than maintaining two separate things that drift).Why: the header standardization we did by hand doesn't carry across version branches, and hand-fixing every branch doesn't scale. This re-applies the deterministic fixes on any branch in seconds.
Two tiers of rules
type,accessRights,license,rights,rightsHolder),xml:langtags, forbidden properties, disallowed namespaces (eumd/euvoc), UUID + UTC-Zdatetime formats,TSO-/RCC-publisher naming.conformsTo/isVersionOf/spatial/wasGeneratedBy/ publisher-party checked against theInstance/referenceDataschemes. Reported but not auto-fixed — they depend on the reference schemes being extended first.How to use it
The fixer is idempotent and formatting-preserving (minimal diffs) and never invents Tier-B values, real dates, titles or spatial refs. README is under
buildScripts/dataset_header/;tests/test_dataset_header.pyruns it in CI (report-only for now).Already applied: #334 ran it across all 68 NetworkCode cimxml headers on ncp-2.5 (auto-fixable Tier-A 329 → 0). A few genuine gaps remain that need real data / decisions (e.g. missing
issued/spatialon the Jotunheim IAM files, a couple of non-TSO/RCC publishers) — the validator lists them.Roadmap / where help is welcome
pyshaclas an independent cross-check of the Python rules (pyshaclis already a dependency). The Python rules stay the fast path; SHACL becomes the correctness cross-check.referenceDataschemes (parties, frames, models, activities, profiles) so Tier-B can flip from report-only to enforced, and the fixer can normalize those values too.eumd) fail the build.Feedback and contributions welcome!
All reactions