Testing Strategy

Testing Strategy

Status: proposed. This is planned as part of development from M1 onwards, not added afterwards.

Why this page exists

A tool whose whole purpose is to give a correct answer about the live Peppol network has an unusual testing problem: the most valuable behaviour depends on remote systems that change without notice, must not be written to casually, and cannot be part of an ordinary CI run. The strategy below separates what must always be true from what can only be checked against the real network.

Two properties are the ones users will actually rely on, so they get the most attention:

1. The verdict is right. can-receive, cert verify and validate must not say yes when the answer is no.
2. The exit code and JSON are right. A CI pipeline reads those, not the prose. A wrong exit code is a silent failure in someone else's build.

Test tiers

Tier	Scope	Network	Runs	Gate
T1 — Unit	Pure logic: identifier parsing and validation, code list lookup, hash computation, document detection, config resolution, output rendering, exit code mapping.	none	every commit	must pass
T2 — Component	One area end to end against a stubbed remote: SMP client against canned XML, Phorm client against a stub HTTP server, DNS resolution against a stub resolver.	none	every commit	must pass
T3 — CLI integration	The real main(), with real argument parsing, config files on a temp directory, and stubbed remotes. Asserts stdout, stderr and exit code.	none	every commit	must pass
T4 — Live read-only	Real queries against the test network: DNS resolution, SMP lookups, Directory search, certificate verification.	yes	nightly + pre-release	reported, does not block a PR
T5 — Live write	SML and SMP write operations against test-network resources that the project controls.	yes	manual / pre-release	manual approval
T6 — Packaging	The built artefacts actually run: fat JAR, distribution archive, launcher scripts, completion scripts, on each supported OS.	none	per release	must pass

The split at T3/T4 is the important one. Everything up to T3 is hermetic and fast, so it can gate every pull request; T4 and T5 tell the truth about the network but cannot be allowed to fail a build for reasons outside the project's control.

T1 — Unit

Straightforward, and most of the correctness lives here because most of the logic is local.

• Identifiers. Table-driven tests over valid and invalid values per scheme: length limits, character sets, the NNNN: prefix rule, case folding, both document type schemes including the wildcard form, and strict versus lax factory behaviour. Every rejection asserts which rule rejected it, not merely that it failed.
• The DNS hash. Known participant identifier → known DNS label, as fixed vectors. This is exactly the kind of value that a refactor can silently change, and where a wrong answer is invisible until someone's lookup fails. > TODO(verify): obtain authoritative hash test vectors — from the Peppol Policy for use of Identifiers or from peppol-commons' own tests — rather than generating expected values with the same code under test, which would prove nothing.
• Code lists. Counts and spot checks per kind, deprecation flags, and an assertion that every generated enum reports the same CODE_LIST_VERSION (currently "9.7") — that catches a partial regeneration.
• A consistency test between the two transport profile enums, EPredefinedTransportProfileIdentifier and ESMPTransportProfile, since nothing else keeps them in agreement (see Library Gaps #10).
• Document detection. Sample documents → expected syntax, Customization ID and derived VESID, including SBDH- and XHE-wrapped inputs. peppol-testfiles is the obvious source of sample documents.
• Config resolution. The full precedence chain asserted layer by layer: command line over system property over environment variable over explicit file over discovered file over shipped default. Plus: unknown key handling, a missing file, an unreadable file, and that redaction actually redacts.

T2 — Component, against recorded responses

The rule: no test above T3 may touch the network, so recorded fixtures carry the weight.

• SMP. A fixture corpus of real SMP responses, captured once from the test network and committed: signed and unsigned, Peppol flavour and BDXR 1.0/2.0, with redirects, with wildcard document types, with expired endpoints, and deliberately malformed. Each fixture is a file plus an expected-result file.
• Signature verification needs both a positive and, importantly, negative fixtures: a tampered body, a signature from an untrusted CA, an expired signing certificate. A signature check that never fails in tests is not a tested signature check.
• Phorm. A stub HTTP server asserting the request the CLI sends — X-Token header present, Content-Type: application/xml, Accept: application/json, correct path and VESID — and returning recorded responses. Response fixtures must include the awkward cases: HTTP 400 carrying a valid report (an invalid document), HTTP 400 carrying a text/plain error (an unparseable body), HTTP 200 carrying a failed report (the server configured with phorm.api.response.onfailure.http400 = false), HTTP 403, and HTTP 500. Those map to four different exit codes and are the single most likely place to get the mapping wrong.
• DNS. A stub resolver returning prepared NAPTR results, plus the failure shapes: NXDOMAIN, a NAPTR record with no usable service, a timeout, and a malformed record.
• Certificates. A small, committed test PKI — self-signed root, intermediate, leaf — so chain validation, expiry and revocation handling can be tested deterministically. Includes an already-expired leaf and a not-yet-valid leaf, generated at fixed dates rather than relative to "now". > TODO(decide): generate the test PKI once and commit it, or generate it at build time? Committed fixtures with fixed dates eventually expire; build-time generation makes failures reproducible but adds a build step. Committed with far-future dates plus a date-override parameter is probably the pragmatic answer.

Fixture discipline, because this is where such suites usually rot:

• Every fixture records where and when it was captured, in a sidecar file.
• A documented, scripted way to re-capture a fixture, so refreshing is a command rather than an archaeology exercise.
• A nightly T4 job that re-fetches the live equivalents and reports when a fixture no longer matches reality — the fixture is not updated automatically, but somebody is told.

T3 — CLI integration

The tier that tests the contract users depend on. Runs the real command tree in-process, with HOME and the config directory pointed at a temporary location.

For every command, assert all three of: stdout, stderr, exit code.

Case	Expectation
Success	Exit 0, result on stdout, nothing required on stderr.
Not found	Exit 3, and --output json still valid JSON with success: false.
Check failed	Exit 4, clearly distinguished from not-found. A test asserting that 3 and 4 are not interchangeable is worth writing explicitly.
Network failure	Exit 5, with a message naming the host that failed.
Bad config	Exit 6, naming the file and the key.
Unknown option / missing argument	Exit 2, with usage on stderr and nothing on stdout.
Auth rejected	Exit 7.
--output json	stdout parses as exactly one JSON document, in every case above, including every failure.
--quiet	Result only.
--verbose / --debug	Extra output on stderr only, and no secret ever appears in it — asserted by putting a known token in the config and grepping the whole output for it.

JSON contract tests are golden-file based: one committed expected document per command, per scenario. A change to the envelope then shows up as a failing diff, which is the mechanism that makes the "stable contract" claim in CLI Conventions enforceable rather than aspirational. > TODO(decide): also generate and publish a JSON Schema per command and validate the golden files against it? It would catch drift the golden files alone would not.

Write commands are tested here for their safety rails, with a stub SMP: that --dry-run performs no request at all, that a destructive command without confirmation exits non-zero having done nothing, and that --yes bypasses only the prompt.

T4 — Live read-only

Against the test network only. Nightly plus pre-release, reported rather than blocking, because a failure here is as likely to be someone else's outage as a regression.

• Resolve a known test participant end to end and assert the whole chain: DNS → SMP URL → service group → metadata → endpoint → certificate.
• can-receive for a known-good and a known-bad combination.
• Verify a live SMP signature against the real G3 trust store.
• Directory search for a known entry.
• Certificate verification against the live CA chain, including a real revocation check — the one thing a local test PKI cannot exercise.
• Validation of a known-good and known-bad document against a real Phorm instance.

> TODO(verify): which test-network participants, document types and SMPs are stable enough to assert against. Hard-coding a third party's registration into a test suite is fragile and impolite; a participant the project controls on the test network would be much better. Does one exist, or should one be registered for this purpose?

Failures here are triaged as our regression versus their outage — which is why the assertions should report which step failed, exactly as the dns resolve command does.

T5 — Live write

The highest-risk tier, so it is manual and narrow.

• Only ever against the test network, and only against SMP and SML resources the project itself owns.
• A test participant identifier reserved for this purpose, registered and deleted by the test itself.
• Every test cleans up after itself, and the suite starts by cleaning up leftovers from a previous failed run.
• Never run automatically on a pull request, and never against production. A guard that refuses to run when the resolved network is prod — asserted by its own test.
• Includes the SML propagation delay explicitly: register, then poll until the DNS record appears, which also exercises the dns wait waiter.

> TODO(decide): is a project-owned test-network SMP and participant available for this? Without one, T5 cannot exist and the SML/SMP write features of M3 ship on manual testing alone — which should then be stated openly rather than left implied.

T6 — Packaging

Cheap to automate and catches a class of failure no unit test will.

• The fat JAR runs: java -jar peppol-cli.jar id validate <known-good-id> produces the expected output. This also catches the shading SPI problem — peppol-commons registers identifier validators and transport profiles through META-INF/services, and if the shade plugin is not merging those, id check-value will silently lose its validators. A packaging test is the only place that surfaces it.
• The distribution archive extracts and its launcher works on Linux, macOS and Windows (subject to Open Questions #8).
• The launcher's Java version check produces a clear message on an older JRE rather than UnsupportedClassVersionError.
• Generated completion scripts are syntactically valid in their target shells.
• peppol version reports the expected version — using CPeppolCommonsVersion / CSMPClientVersion to report library versions too, which makes bug reports far more useful.

Practices

• Every bug fix gets a regression test, at the lowest tier that can reproduce it. A network-dependent bug becomes a T2 fixture, not a T4 test.
• Negative tests are not optional. For each of signature verification, certificate validation and document validation there must be a test that fails correctly. The failure path is the one users depend on.
• No test may depend on the current date, other than deliberate expiry tests, which pass an explicit evaluation date. isEndpointValidAt and setValidationDateTime exist precisely so this is possible.
• No secrets in the repository. Live-tier credentials come from CI secrets; the suite skips with a clear message when they are absent rather than failing obscurely.
• Coverage is a diagnostic, not a target. The exit code and JSON contract tests matter more than a percentage.
• The stubs are shared. The stub SMP, stub DNS resolver and stub Phorm server live in one test-support module, so tests stay short enough that people write them.

> TODO(decide): should the test-support module be published, so that downstream users of peppol-cli-core get the stubs too? Probably not for a first release.