Reduce first-query cost of the bundled Lovdata index

[MariusStorhaug]

Consumers query Norwegian legal metadata offline through Get-LovdataDocument, Find-LovdataDocument, and Get-LovdataLegalArea, all of which read the bundled index. Bundling the whole corpus rather than a small list was the main design risk, and the lazy-load design handles it well: nothing is read at import, so a caller using only Get-LovdataPublicDataset or Test-LovdataConnection never pays for the corpus, and the packaged artifact is trivial.

Request

User impact

The first index query — the moment a caller touches any of the three document commands — costs roughly 2.3 seconds and a one-time jump of about 254 MB resident memory, regardless of what the caller asked for. Import-LovdataIndexData reads every source file and materialises all 4,185 records as LovdataDocument instances up front, then caches them for the session. Subsequent queries are fast (~22 ms). A quarter of a gigabyte of live objects is material in CI runners, containers, and constrained hosts, and a caller querying only acts (nl) still pays for sf, del, ins, and stv.

Measurements taken by building the module locally and copying the index to the module root as Build-PSModuleBase does:

Metric	Value
Index JSON on disk	9.94 MB across 5 files
Built module folder	10.03 MB
Packaged (zip/nupkg)	1.21 MB
Import-Module cold	50 ms
Memory after import	91 MB
First index query	2,319 ms
Memory after first query	345 MB
Subsequent query	22 ms
Find-LovdataDocument -Name 'veg'	89 ms, 112 hits

The nl source is 1.87 MB of the 9.94 MB total, so a caller who only needs acts currently loads roughly five times more data and memory than the query requires. Constructing 4,185 full class instances is what dominates the memory: a search returning 112 hits does not need 4,185 constructed objects to exist first.

Acceptance criteria

• A caller can restrict a query to the sources it needs, and doing so loads and constructs only those sources.
• The default behaviour is unchanged for callers that pass no source, so nothing regresses.
• First-query time and peak memory drop measurably for source-scoped queries, and ideally for the whole-index case as well.
• No change to the shipped index files or to the object shape callers already receive.

Regression

Not a regression. This is the behaviour since the index was introduced in #18. Deferred from the review of #20 to keep that correctness fix (per-reference LastChangedBy dates) landing clean on its own.

---

Technical decisions

Two independent improvements, either deliverable on its own.

1 — Load only the sources needed. Import-LovdataIndexData (in src/functions/private/Parsing/) takes no source parameter today; it globs LovdataIndex.*.json and reads all of them into one session cache. Add an optional Source parameter that selects which per-source files to read, defaulting to all so existing callers are unaffected. Thread an equivalent selector through the three public commands so a caller can scope a query. The cache becomes per-source rather than a single all-or-nothing blob. Open decision: whether the source selector is surfaced on the public commands as a parameter with a ValidateSet of the known bases (nl, sf, del, ins, stv) or discovered from the shipped files at runtime — discovery avoids hardcoding but loses tab-completion.

2 — Defer LovdataDocument construction. The loader currently calls [LovdataDocument]::new(...) for every record eagerly. Keep the deserialised records and construct a LovdataDocument only for the records a query actually returns. This preserves the [LovdataDocument] output contract the commands and tests assert while removing the up-front construction of thousands of objects that are never returned.

Breaking changes: None. Both changes are additive; default output type and default behaviour are preserved.

Test approach: Assert that a source-scoped query does not read or construct other sources (spy on file reads / count constructed instances), that the default path still returns every document, and that the output remains [LovdataDocument].

---

Implementation plan

Core changes

• Add an optional Source parameter to Import-LovdataIndexData in src/functions/private/Parsing/, defaulting to all sources, with a per-source cache
• Surface a source selector on Get-LovdataDocument, Find-LovdataDocument, and Get-LovdataLegalArea
• Defer LovdataDocument construction to the records actually returned rather than constructing the whole index up front

Tests

• Add a test asserting a source-scoped query reads and constructs only that source
• Add a test asserting the default (no source) path still returns every document with type LovdataDocument
• Add a test asserting deferred construction does not change the returned object shape

Documentation

• Update the affected command help and Documents.md with the source parameter
• Add an example showing a source-scoped offline query

[MariusStorhaug]

Measurements

Taken on a locally built module with the index copied to the module root as Build-PSModuleBase does. Recording them here so they are not re-measured or guessed at later.

	Value
Index JSON on disk	9.94 MB across 5 files
Records	5 879 documents (nl 759, sf 3 426, plus del, ins, stv)
Built module folder	10.03 MB
Packaged (nupkg)	1.21 MB
Import-Module cold	50 ms
Memory after import	91 MB
First index query	2 319 ms
Memory after first query	345 MB (+254 MB)
Subsequent query	22 ms
Find-LovdataDocument -Name 'veg'	89 ms, 112 hits

Packaged size and import time are both fine and need no work. The index is genuinely lazy, so anyone using only the download or service commands never pays for it. This issue is only about the first query.

This is the common path, not a tail case

Framing it as a "first-use cost" understates it. For this module, querying the corpus is the primary reason to install it, so a user of the archive surface pays this once per session, every session. It is the common case rather than a rare one.

Where the cost actually is

The obvious assumption is that constructing 5 879 LovdataDocument instances is the whole problem. That was worth testing before committing to a refactor built on it, so the two stages were measured separately in a clean process:

Stage	Memory	Time
ConvertFrom-Json only, no class construction	+93.5 MB	~440 ms
LovdataDocument construction on top	+130.8 MB	1 878 ms
Total	+224.3 MB	2 319 ms

Class construction is the larger share — 58% of the memory and 81% of the time. So deferring it is worth doing.

But deserialization alone is still 93.5 MB, which is 42% of the footprint. That is the floor: deferring class construction cannot take the module below roughly 93 MB and 440 ms without also changing the on-disk or in-memory shape. Anyone planning that refactor should size it against that ceiling rather than expecting the cost to disappear.

Proposed order

1. Source-scoped loading — cheap, do this first. Import-LovdataIndexData reads every source file regardless of what the caller asked for. nl is 1.87 MB of the 9.94 MB, so someone querying only acts currently pays roughly 5x what their query needs. Adding a source parameter, with the present behaviour as the default, is small, backwards-compatible, and independently useful.

2. Defer class construction — the bigger win, but a real refactor. Keep the deserialized records and build a LovdataDocument only for what is returned. Find-LovdataDocument returning 112 hits does not need 5 879 constructed objects to exist. Worth keeping separate from item 1 so the cheap change can land on its own.

Item 2 is bounded by the 93.5 MB deserialization floor above. If that is still too much, the answer is a leaner on-disk shape or streaming, not a deeper object-construction change.

[MariusStorhaug]

Resolving the open decision on how Source should be constrained: an argument completer, not a ValidateSet.

The choice was framed as tab-completion versus not hardcoding, but those are not actually in tension. ValidateSet is what couples the two — it is the only one of the options that both completes and validates. An [ArgumentCompleter] gives completion from the shipped files at runtime, with no baked-in list.

This is the established pattern in the ecosystem rather than a novel suggestion: PSModule/Context and PSModule/GitHub both use completers.ps1 files registering Register-ArgumentCompleter for exactly this shape of value — GitHub has sixteen of them. The convention is a completers.ps1 alongside the commands in each public function group.

Two reasons ValidateSet is the wrong tool here specifically:

• The set is data, not code. The bases come from whatever Lovdata ships in the archives. A ValidateSet would have to be edited whenever Lovdata adds or renames one, and the daily updater would then be capable of producing an index the module refuses to read. The same reasoning already applies elsewhere in this module: the archive reader was deliberately written to enumerate the extracted folders rather than assume nl, because the regulations package uses different ones.
• A wrong value is not worth a hard failure. ValidateSet rejects at bind time with a generic message. An unknown source is better handled by the loader returning nothing for it, or by naming the sources actually present, which tells the caller something the validator cannot.

So: complete from the index files present on disk, and let the loader handle an unrecognised value with a message that names what is available.