Skip to content

🚀 [Feature]: Query Norwegian law offline and parse the archives into objects - #20

Merged
Marius Storhaug (MariusStorhaug) merged 20 commits into
build-lovdata-modulefrom
lovdata-archive-object-model
Aug 2, 2026
Merged

🚀 [Feature]: Query Norwegian law offline and parse the archives into objects#20
Marius Storhaug (MariusStorhaug) merged 20 commits into
build-lovdata-modulefrom
lovdata-archive-object-model

Conversation

@MariusStorhaug

Copy link
Copy Markdown
Member

Norwegian acts and central regulations are now reachable as PowerShell objects, offline and with no account. The module ships a bundled index of every openly published document's metadata, refreshed daily, and can parse a downloaded archive into a full object hierarchy, preserving chapter and paragraph structure, clauses, lists, footnotes, and change notes, with nothing dropped.

New: Search the whole corpus offline

Find-LovdataDocument searches the bundled index by title, short title, identifier, ministry, or legal area, and Get-LovdataDocument returns a document from the index with no network access. Get-LovdataLegalArea returns the legal-area taxonomy derived from the index.

New: Full-fidelity documents from an archive

Expand-LovdataPublicDataset unpacks a downloaded .tar.bz2 package, and Get-LovdataDocument -Path parses the extracted XML into a LovdataDocument with its complete body. A date field carrying two dates or a prose qualifier is exposed in full rather than truncated, and a legal area keeps its hierarchical path.

Technical Details

  • Object model under src/classes/public/ derived from a scan of the live corpus.
  • Class-keyed metadata parser, recursive body parser, and a date-field parser under src/functions/private/Parsing/.
  • Deterministic per-source index JSON under src/, generated by the module's own parser, refreshed by scripts/Update-LovdataData.ps1 and a daily workflow.
  • Implementation plan progress: tracks Section 3 of Represent the open archive corpus as PowerShell objects with a bundled, daily-updated index #18.

Note: the daily updater needs two GitHub App secrets provisioned: LOVDATA_UPDATER_BOT_CLIENT_ID and LOVDATA_UPDATER_BOT_PRIVATE_KEY.

Related issues

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@MariusStorhaug Marius Storhaug (MariusStorhaug) added the minor New feature or enhancement, version 0.x.0 increase label Aug 2, 2026
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

Super-linter summary

Language Validation result
CHECKOV Pass ✅
GITHUB_ACTIONS Pass ✅
GITLEAKS Pass ✅
GIT_MERGE_CONFLICT_MARKERS Pass ✅
JSON Pass ✅
MARKDOWN Pass ✅
NATURAL_LANGUAGE Pass ✅
POWERSHELL Pass ✅
PRE_COMMIT Pass ✅
SPELL_CODESPELL Pass ✅
TRIVY Pass ✅
XML Pass ✅
YAML Pass ✅

All files and directories linted successfully

For more information, see the GitHub Actions workflow run

Powered by Super-linter

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…pell check

The metadata parser now normalises irregular whitespace read from the archive so the bundled index carries no non-breaking spaces, which the strict JSON linter rejects. The spell checker skips the generated index files and the Norwegian test fixtures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…nology linter

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Read each LastChangedBy reference date from that reference's own RefID rather
than from the field as a whole, and add InForceFrom for the per-reference
'fra <date>' in-force qualifier. Previously one date was computed for the whole
field and stamped on every reference, mislabelling every reference after the
first and dropping the in-force date entirely.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Assert the real two-reference shape from a small NLOD fixture: each reference
keeps its own date, and the first reference's 'fra' in-force date does not leak
onto the second reference that has none. This is the exact failure mode that
slipped through before, because no test covered more than one reference.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Rebuild the index so LastChangedBy references carry their own correct dates and
the new InForceFrom field. The committed data previously shipped a neighbour's
date on every reference after the first.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Note that the skip silences the Norwegian legal data on purpose and that the
trade-off is a real typo in a future fixture going uncaught.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
codespell reads the bare local 'inForce' as a misspelling of 'enforce'. Rename
it to inForceFrom to match the property; the index output is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@MariusStorhaug

Copy link
Copy Markdown
Member Author

Verified independently before merge, against the committed index rather than the parser tests.

Corpus-wide invariant across the bundled data: for every reference whose RefID embeds a date, does Date match that reference's own RefID? 1351 references checked, 0 mismatches. That is the exact property the defect violated, asserted over the whole shipped dataset rather than one fixture.

Independent spot-check against published text: veglova's record is RefID lov/2024-12-06-74, Date 2024-12-06, InForceFrom 2025-01-01. A separately hand-written rendering of that law, produced by a person reading Lovdata, states Sist endret | LOV-2024-12-06-74 fra 01.01.2025. They agree, and that source is genuinely independent of this parser. InForceFrom is populated on 479 documents, so the new field carries real data rather than sitting null.

ChangesToDocuments shares the same reference parser and is covered by the same check. It never showed the visible symptom, because those fields are bare RefID lists with no free-text date for the previous code to latch onto, so the field-wide date it computed usually coincided with the correct answer. Same latent defect, masked by the data; fixed by the same change and now proven rather than assumed. eeaReferences and appliesTo use the plain-text path and were never affected.

The regression test asserts per reference including the negative case, so it fails against the previous parser rather than merely passing against the new one.

First-query cost is deferred to #21 as a measured, non-blocking characteristic.

@MariusStorhaug
Marius Storhaug (MariusStorhaug) marked this pull request as ready for review August 2, 2026 16:50
@MariusStorhaug
Marius Storhaug (MariusStorhaug) merged commit 7b2ee34 into build-lovdata-module Aug 2, 2026
51 checks passed
@MariusStorhaug
Marius Storhaug (MariusStorhaug) deleted the lovdata-archive-object-model branch August 2, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

minor New feature or enhancement, version 0.x.0 increase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant