Testing Strategy

Testing Strategy

Status: proposed. This is planned as part of development from M1 onwards, not added afterwards.

Why this page exists

A tool whose whole purpose is to give a correct answer about the live Peppol network has an unusual testing problem: the most valuable behaviour depends on remote systems that change without notice, must not be written to casually, and cannot be part of an ordinary CI run. The strategy below separates what must always be true from what can only be checked against the real network.

Two properties are the ones users will actually rely on, so they get the most attention:

1. The verdict is right. can-receive, cert verify and validate must not say yes when the answer is no.
2. The exit code and JSON are right. A CI pipeline reads those, not the prose. A wrong exit code is a silent failure in someone else's build.

Test tiers

Tier	Scope	Network	Runs	Gate
T1 — Unit	Pure logic: identifier parsing and validation, code list lookup, hash computation, document detection, config resolution, output rendering, exit code mapping.	none	every commit	must pass
T2 — Component	One area end to end against a stubbed remote: SMP client against canned XML, Phorm client against a stub HTTP server, DNS resolution against a stub resolver.	none	every commit	must pass
T3 — CLI integration	The real main(), with real argument parsing, config files on a temp directory, and stubbed remotes. Asserts stdout, stderr and exit code.	none	every commit	must pass
T4 — Live read-only	Real queries against the test network: DNS resolution, SMP lookups, Directory search, certificate verification.	yes	nightly + pre-release	reported, does not block a PR
T5 — Live write	SML and SMP write operations against test-network resources that the project controls.	yes	manual / pre-release	manual approval
T6 — Packaging	The built artefacts actually run: fat JAR, distribution archive, launcher scripts, completion scripts, on each supported OS.	none	per release	must pass

The split at T3/T4 is the important one. Everything up to T3 is hermetic and fast, so it can gate every pull request; T4 and T5 tell the truth about the network but cannot be allowed to fail a build for reasons outside the project's control.

T1 — Unit

Straightforward, and most of the correctness lives here because most of the logic is local.

• Seat ID extraction. SPIDHelper.getSeatIDFromCertificate is covered upstream, so the CLI only needs a test that a certificate from the local test PKI with CN=POP000001 yields that Seat ID and Main ID 000001, and that a certificate whose CN is not a Seat ID yields a clear "not a Peppol service provider certificate" message rather than an empty field.
• Check digits. Verified vectors to use directly: 4035600210708 and 5790000435975 are checksum-valid GLNs; 7315458756324 is not (GS1 yields 8, the value carries 4) despite being a commonly-circulated Peppol test identifier. That last case is the important test — it must produce a warning at exit 0 by default, and exit 4 only under --strict-check-digits.
• Identifiers. Table-driven tests over valid and invalid values per scheme: length limits, character sets, the NNNN: prefix rule, case folding, both document type schemes including the wildcard form, and strict versus lax factory behaviour. Every rejection asserts which rule rejected it, not merely that it failed.
• The DNS hash. An authoritative vector is now available: SML 1.3.0 §3.1.1 gives 0010:5798000000001 → XUKHFQABQZIKI3YKVR2FHR4SNFA3PF5VPQ6K4TONV3LMVSY5ARVQ, and it has been independently reproduced (SHA-256 of the lowercased ID, Base32, trailing = stripped). This becomes the anchor test vector, since it comes from the spec rather than from the code under test. Add cases for a mixed-case input (to prove lowercasing happens) and for an identifier whose digest Base32-encodes with padding (to prove stripping happens). > TODO(verify): whether PFUOI 4.4.0 POLICY 7 supplies further vectors worth adding.
• U-NAPTR URI rules. Table-driven: a URI with http, with userinfo, with a query and with a fragment must each be rejected with the specific rule cited; the four example URLs the spec calls valid must each be accepted.
• Endpoint validity windows. ServiceActivationDate in the future and ServiceExpirationDate in the past must both make an endpoint unusable, with a missing date meaning unbounded. Always tested at a fixed evaluation date, never at "now" — which --at makes straightforward, and which is also the test for --at itself: the same fixture must answer "can receive" at one date and "cannot" at another, and the answer must state which date it used.
• Transport profile selection. A participant publishing endpoints for a profile other than the requested one must produce the distinct answer — registered, document type supported, but not over that profile, listing what it does publish — rather than a bare "cannot receive". Also assert the default is the active Peppol AS4 profile when --transport-profile is omitted.
• Dry run must be proven, not assumed. Two fixtures matter more than the happy path: an importResult whose settings echoes dryRun="true" (report a preview), and one where the attribute is absent — simulating a pre-8.6.1 server that ignored the parameter — which must report that a real import was performed. Getting that second case wrong would print a reassuring preview of changes that have already been made.
• SMP import results render per participant. An importResult fixture mixing success, warning and error actions must produce a per-row table keyed by participantID, and the exit code must come from the highest error level present rather than from the HTTP status. Both formats need a fixture, since 8.6.1 negotiates the response and anything older always answers XML: the identical action list in XML and in JSON must produce the identical table and exit code. The dispatch assertion is the one that would otherwise be missed — given a request for JSON and an XML response, as a pre-8.6.1 server returns, the CLI must parse it rather than fail, which is only true if it keys off the response Content-Type instead of what it asked for.
• Batch SML failures leave state unchanged. A DeleteList fixture containing one unknown identifier must report that nothing was deleted — not "partially applied" and not a per-row table — and must reproduce the fault message verbatim. Assert the CLI does not parse that message for identifiers, since it is implementation text: the test should still pass if the wording changes.
• Exactly one valid endpoint. Three fixtures: one matching endpoint (the answer), zero (cannot receive), and two with overlapping validity windows for the same transport profile — which must be an error listing both, never a silent pick. The last one is the important case, because the library returns the first match without complaint, so a CLI that trusts it would report a confident answer about a broken registration. Also assert the duplicate process ID and duplicate transport profile cases are reported as findings rather than only logged.
• One selection path, asserted across both commands. smp can-receive and cert of-endpoint must agree on every fixture: same endpoint chosen, same --at and --transport-profile behaviour, same distinct answers when selection yields nothing. Run the selection fixtures through both and compare, so a divergence fails a test rather than surfacing as two commands contradicting each other in the field.
• Code lists. Counts and spot checks per kind, and the three-valued state (active/deprecated/removed) surfacing correctly. Do not assert that the per-kind CODE_LIST_VERSION values match each other — each code list is versioned independently, so they are legitimately allowed to differ and such a test would fail for a correct library. Assert instead that each version is reported, and reported against the right kind.
• Alias resolution. A slugified display name resolves to the expected full identifier, and the resolved full identifier appears in the output. The ambiguity case is the important one and needs asserting per command family — a lookup command and a codelist action — because it is the case that would otherwise silently answer about the wrong document type: one of the 16 duplicated display names in v9.7 must exit 2, emit "alias is not unique", list both candidates in full, and in JSON carry ALIAS_NOT_UNIQUE with the candidates as data. Assert explicitly that it does not resolve to the first match.
• Transport profiles: assert Peppol validity is judged against the Peppol code list, and explicitly do not assert that EPredefinedTransportProfileIdentifier and ESMPTransportProfile agree — the latter is a wider collection, so they legitimately differ and an equality test would fail against a correct library. The valuable test is the opposite one: a transport profile that exists only in the wider enum must be rejected as invalid for Peppol.
• Document detection. Sample documents → expected syntax, Customization ID and derived VESID, including SBDH- and XHE-wrapped inputs. peppol-testfiles is the obvious source of sample documents.
• Config resolution. The full precedence chain asserted layer by layer: command line over system property over environment variable over explicit file over discovered file over shipped default. Plus: unknown key handling, a missing file, an unreadable file, and that redaction actually redacts.

T2 — Component, against recorded responses

The rule: no test above T3 may touch the network, so recorded fixtures carry the weight.

• SMP. A fixture corpus of real SMP responses, captured once from the test network and committed: signed and unsigned, with redirects, with wildcard document types, with expired endpoints, and deliberately malformed. Each fixture is a file plus an expected-result file.

• Signature verification needs both a positive and, importantly, negative fixtures: a tampered body, a signature from an untrusted CA, an expired signing certificate. A signature check that never fails in tests is not a tested signature check.

• Redirect certificate check. Three fixtures: a redirect whose destination signing certificate's subject DN matches CertificateUID (pass), one that genuinely differs (reported mismatch, showing both values), and one where the two DNs are the same name in different textual renderings — different attribute order or spacing. The last must pass, which is what proves the comparison is canonical rather than string equality.

• Signed versus unsigned resources. Only SignedServiceMetadata is signed; the ServiceGroup listing is not. So assert that verify-signature requires both a participant and a document type, and that commands reading the service group report their output as unauthenticated rather than implying a verified signature. One further test guards the forward-compatibility note: a ServiceGroup fixture that does carry a ds:Signature must be verified, proving the code path keys off the signature's presence rather than off which resource was requested.

• Phorm. The protocol belongs to phorm-client, so these tests cover the CLI's use of it and its mapping of PhormValidationResult and PhormClientException onto output and exit codes. Fixture-driven PhormValidationResult values must include: a clean pass, a document with findings (→ exit 4), an interrupted validation (isInterrupted ()), and a server-side error (getServerException () populated). Assert that a report with findings yields exit 4 while a transport or configuration failure yields 5, 6 or 7 — the client library removes the need to reason about status codes, but the exit code mapping is still ours to get right and is still the easiest thing to get wrong. Also assert the JSON envelope embeds getAsJson () unchanged, so the contract test fails if someone starts reshaping Phorm's report by hand. > TODO(decide): is a live smoke test against a real Phorm instance wanted in T4, since phorm-client's own tests cover the protocol?

• DNS. A stub resolver returning prepared NAPTR results, plus the failure shapes: NXDOMAIN, a NAPTR record with no usable service, a timeout, and a malformed record.

• DNSSEC. All four EDnsSecValidationStatus values need a case, because they map to different behaviour: NOT_VALIDATED (only reachable via --no-dnssec, since validation is on by default), SECURE (the expected result, since both SML zones are signed), INSECURE → reported as an anomaly at exit 0, because a provably-unsigned answer from a known-signed zone means something is interfering, and BOGUS → a prominent security finding at exit 4. Two further cases: a validated NXDOMAIN must be reported as provably not registered (exit 3) rather than as a failed lookup; and the resolver that strips DNSSEC records must produce output blaming the resolver and offering --dns-server and --no-dnssec, not output reporting every participant as unregistered — this is now the first-run experience for affected users, so it deserves a test of its own. Also assert there is no automatic fallback: a validation failure must not be retried without DNSSEC.

• Business card versions. One fixture per version — v1 (…/businesscard/20160112/), v2 (…/20161123/) and v3 (…/20180621/). All three must parse; v1 and v2 must additionally produce the outdated-version warning at exit 0, and v3 must produce none. This test is what guards the custom marshaller customizer: with the library's default customizer the older versions parse silently, so a regression here would be invisible in ordinary use.

• Directory. The protocol layer belongs to peppol-directory-searchclient, so these tests cover the CLI's use of it, not the HTTP itself: inject a stub HttpClientSettings (the PDSearchClient (URI, HttpClientSettings) constructor exists for exactly this) or stub the endpoint, serving recorded result lists validated against the shipped directory-search-result-list-v1.xsd. Paging assertions should be driven by the response's own metadata, with the traversal itself being searchAllPages's job rather than ours: given total-result-count = 347 with result-page-count = 20, --all must render every item and report 347, and --max-items must make the page handler return EContinue.BREAK so the traversal stops early rather than fetching pages it will discard. A fixture with total-result-count above 1000 must produce an explicit "truncated by the server" statement rather than a silent stop. Required failure cases: an empty query (HTTP 400 → exit 2), paging past the 1000-result cap (HTTP 400), and HTTP 429 with Retry-After: 2 — the client surfaces this as PDSearchRateLimitException with getRetryAfterSeconds (), so the test must prove the CLI backs off on that value rather than retrying immediately, because a bulk run that ignores it will get the user rate-limited in production.

• NSS list staleness. With the list's last-update date injected as a fixed value, assert a date five months old produces no warning and seven months old does, that the warning appears only on the TLS check and not on unrelated commands, and that the exit code is unchanged either way. Tested against an injected date, never against the real bundled one, or the test would start failing six months after it was written.

• NSS trust list resolution. The version gate decides which roots the tool trusts, so it needs its own tests rather than being exercised incidentally. With a temporary cache directory holding synthetic lists named mozilla-nss-root-certs-<version>-<date>.p12: a list newer than the bundle is used; equal and older ones are ignored in favour of the bundle; and the reason is reported in each case. Because discovery scans the directory, a case with several lists present must select the highest version rather than the first or the newest by file timestamp. The numeric comparison needs an explicit case — a local 2.100 against a bundled 2.90 must be selected, which is precisely the pair a string comparison gets wrong and the one a test written only with single-digit minors would never catch.

  Then the two rejection classes, which must not be conflated. A sha256 mismatch stops the command with exit 6 and no fallback — assert that specifically, since the natural implementation is to treat it as one more unusable file. Everything else falls back to the bundle with a warning and an unchanged exit code: a filename that does not parse, an absent .sha256, a PKCS#12 that will not open with the expected password, an empty trust store, and a version at or below the bundle.

  Finally the instruction path, which is where the rule is easiest to get wrong: a file named with --nss-truststore is held to the same version gate and must be refused, not honoured, when it is older or equal — and a named file whose name carries no readable version must exit 2 rather than being used unchecked. Two assertions guard the reporting: the output names the source it used in every one of these cases, and no test asserts against the real bundled version, or the suite will need editing on each ph-nss upgrade.

• SSL Labs link, never a call. Assert the emitted URL carries the correct host extracted from the endpoint URL and hideResults=on — a missing suppression parameter would publish a customer's configuration, so it is worth an explicit assertion rather than trusting the string template. Also assert that no request is made to ssllabs.com in any mode, and that --open in a headless environment falls back to printing instead of failing. Also assert an endpoint on a non-443 port reports that SSL Labs cannot test it, rather than emitting a link to a different service.

• TLS policy checks. The three checkable policies need fixtures: a self-signed server certificate (POLICY 2 violation), a certificate from a root outside the Mozilla NSS list (POLICY 1, verified against MozillaNSSTrustStore rather than the JDK trust store), and a server offering only TLS 1.0/1.1 (a grade-A blocker). Each must be reported as the specific policy it breaks, not as a generic handshake failure. Two negative assertions matter as much as the positive ones: the output must not claim an SSL Labs grade, and the test must prove no request is made to SSL Labs — the URL is printed, not called.

• Revocation modes. One case per ERevocationCheckMode value, asserting which sources were actually consulted — ocsp must not fall back to CRL, crl must not consult OCSP, and none must make no network request at all. Two further assertions matter more than the happy paths: a soft-failed pass must be reported as soft-failed rather than as a pass, and --no-cache must bypass the 6-hour result cache, since a stale "not revoked" is the worst answer this command can give.

• Certificates. A small, committed test PKI — self-signed root, intermediate, leaf — so chain validation, expiry and revocation handling can be tested deterministically. Includes an already-expired leaf and a not-yet-valid leaf, generated at fixed dates rather than relative to "now". > TODO(decide): generate the test PKI once and commit it, or generate it at build time? Committed fixtures with fixed dates eventually expire; build-time generation makes failures reproducible but adds a build step. Committed with far-future dates plus a date-override parameter is probably the pragmatic answer.

Fixture discipline, because this is where such suites usually rot:

• Every fixture records where and when it was captured, in a sidecar file.
• A documented, scripted way to re-capture a fixture, so refreshing is a command rather than an archaeology exercise.
• A nightly T4 job that re-fetches the live equivalents and reports when a fixture no longer matches reality — the fixture is not updated automatically, but somebody is told.

T3 — CLI integration

The tier that tests the contract users depend on. Runs the real command tree in-process, with HOME and the config directory pointed at a temporary location.

For every command, assert all three of: stdout, stderr, exit code.

Case	Expectation
Success	Exit 0, result on stdout, nothing required on stderr.
Not found	Exit 3, and --output json still valid JSON with success: false.
Check failed	Exit 4, clearly distinguished from not-found. A test asserting that 3 and 4 are not interchangeable is worth writing explicitly.
Network failure	Exit 5, with a message naming the host that failed.
Bad config	Exit 6, naming the file and the key.
Unknown option / missing argument	Exit 2, with usage on stderr and nothing on stdout.
Auth rejected	Exit 7.
--output json	stdout parses as exactly one JSON document, in every case above, including every failure.
--quiet	Result only.
--verbose / --debug	Extra output on stderr only, and no secret ever appears in it — asserted by putting a known token in the config and grepping the whole output for it.

JSON contract tests are golden-file based: one committed expected document per command, per scenario. A change to the envelope then shows up as a failing diff, which is the mechanism that makes the "stable contract" claim in CLI Conventions enforceable rather than aspirational.

Commands that support --output jsonl need a golden file for both serialisations, and one assertion that the golden files alone will not give: every envelope field present in the json form must appear in the jsonl header or trailer, since the two are one contract and the easy mistake is to add a field to the document form and forget the stream. Two more are worth asserting outright — that each line is independently parseable, so a consumer reading line by line never needs the next line to make sense of this one, and that a stream cut off mid-run has no trailer, which is how a consumer distinguishes a crashed run from a clean one. > TODO(decide): also generate and publish a JSON Schema per command and validate the golden files against it? It would catch drift the golden files alone would not.

Write commands are tested here for their safety rails, with a stub SMP: that --dry-run performs no request at all, that a destructive command without confirmation exits non-zero having done nothing, and that --yes bypasses only the prompt.

T4 — Live read-only

Against the test network only. Nightly plus pre-release, reported rather than blocking, because a failure here is as likely to be someone else's outage as a regression.

• Resolve a known test participant end to end and assert the whole chain: DNS → SMP URL → service group → metadata → endpoint → certificate.
• can-receive for a known-good and a known-bad combination.
• Verify a live SMP signature against the real G3 trust store.
• Directory search for a known entry.
• Certificate verification against the live CA chain, including a real revocation check — the one thing a local test PKI cannot exercise.
• Validation of a known-good and known-bad document against a real Phorm instance.

> TODO(verify): which test-network participants, document types and SMPs are stable enough to assert against. Hard-coding a third party's registration into a test suite is fragile and impolite; a participant the project controls on the test network would be much better. Does one exist, or should one be registered for this purpose?

Failures here are triaged as our regression versus their outage — which is why the assertions should report which step failed, exactly as the dns resolve command does.

T5 — Live write

The highest-risk tier, so it is manual and narrow.

• Only ever against the test network, and only against SMP and SML resources the project itself owns.
• A test participant identifier reserved for this purpose, registered and deleted by the test itself.
• Every test cleans up after itself, and the suite starts by cleaning up leftovers from a previous failed run.
• Never run automatically on a pull request, and never against production. A guard that refuses to run when the resolved network is prod — asserted by its own test.
• Includes the SML propagation delay explicitly: register, then poll until the DNS record appears, which also exercises the dns wait waiter.

> TODO(decide): is a project-owned test-network SMP and participant available for this? Without one, T5 cannot exist and the SML/SMP write features of M3 ship on manual testing alone — which should then be stated openly rather than left implied.

T6 — Packaging

Cheap to automate and catches a class of failure no unit test will.

• The fat JAR runs: java -jar peppol-cli.jar id validate <known-good-id> produces the expected output. This also catches the shading SPI problem — peppol-commons registers identifier validators and transport profiles through META-INF/services, and if the shade plugin is not merging those, id check-value will silently lose its validators. A packaging test is the only place that surfaces it.
• The distribution archive extracts and its launcher works on Linux, macOS and Windows (subject to Open Questions #8).
• The launcher's Java version check produces a clear message on an older JRE rather than UnsupportedClassVersionError.
• Generated completion scripts are syntactically valid in their target shells.
• peppol version reports the expected version — using CPeppolCommonsVersion / CSMPClientVersion to report library versions too, which makes bug reports far more useful.

Practices

• Every bug fix gets a regression test, at the lowest tier that can reproduce it. A network-dependent bug becomes a T2 fixture, not a T4 test.
• Negative tests are not optional. For each of signature verification, certificate validation and document validation there must be a test that fails correctly. The failure path is the one users depend on.
• No test may depend on the current date, other than deliberate expiry tests, which pass an explicit evaluation date. isEndpointValidAt and setValidationDateTime exist precisely so this is possible.
• No secrets in the repository. Live-tier credentials come from CI secrets; the suite skips with a clear message when they are absent rather than failing obscurely.
• Coverage is a diagnostic, not a target. The exit code and JSON contract tests matter more than a percentage.
• The stubs are shared. The stub SMP, stub DNS resolver and stub Phorm server live in one test-support module, so tests stay short enough that people write them.

> TODO(decide): should the test-support module be published, so that downstream users of peppol-cli-core get the stubs too? Probably not for a first release.