chore(tests+xml): XSD 1.1 preprocessor in XmlValidator + L1 split coverage + DinD path-agnostic + coverlet 6.x + CI matrix-include#166
Merged
PatrickRitchie merged 8 commits intoMay 24, 2026
Conversation
…L schema load
Adds MTConnect.XsdPreprocessor as a public static helper that removes
XSD 1.1-only constructs from an XSD source string and returns the
1.0-compatible subset the BCL XmlSchemaSet can compile:
* xs:assert / xs:override — removed wholesale (no 1.0 equivalent).
* notNamespace / notQName on xs:anyAttribute — attribute stripped
(element falls back to namespace='##any').
* notNamespace / notQName on xs:any — element removed (the '##any'
fallback would create a content-model wildcard ambiguity adjacent
to concrete elements that XSD 1.0 rejects under
unique-particle-attribution).
* xs:any inside xs:all — removed (XSD 1.0 disallows xs:any as an
all-particle).
* xs:element with maxOccurs > 1 inside xs:all — maxOccurs clamped
to 1 (XSD 1.0 requires {0, 1}).
* xs:all inside xs:extension / xs:restriction — renamed to
xs:sequence (XSD 1.0 forbids xs:all in a derivation).
Wires the helper into XmlValidator.Validate so the BCL XmlSchema.Read
step no longer fails on the official MTConnect XSDs (v1.3 onwards) with
"notNamespace not supported" / equivalent errors. Downstream consumers
calling the XML validation surface against the shipped XSDs now reach
the validation step instead of a load-time failure.
The transformation is lossy with respect to the spec's XSD 1.1
semantics: notNamespace's negative-namespace constraint disappears,
xs:all-derivation becomes order-sensitive, all-group cardinality is
clamped. Every transformation is one-way; the 1.0-compatible subset
preserves every rule the 1.0 reader can express. Full XSD 1.1
validation is the upgrade path via a Saxon-HE-backed validator.
…refs) Aligns the four remaining 3.2.0 references to the 6.0.4 version already in use elsewhere in the test suite (Common-Tests, HTTP-Tests, XML-Tests, SHDR-Tests, Integration-Tests). Versions are managed per-csproj; there is no central Directory.Packages.props to update. Coverlet 3.2.0's Mono.Cecil.Rocks instrumenter hangs on Scriban- generated .g.cs DLLs at collection time. The 6.x line replaces the instrumenter and no longer exhibits this hang. With all nine test csproj files on 6.0.4 the full-sweep CI run no longer risks the per-csproj instrumentation freeze. Bumped: - tests/MTConnect.NET-AgentModule-MqttRelay-Tests/ - tests/MTConnect.NET-JSON-Tests/ - tests/MTConnect.NET-JSON-cppagent-Tests/ - tests/Compliance/MTConnect-Compliance-Tests/
…ompat schema load
Routes the L1 SchemaLoadTests fixture and the XML-Tests
XsdValidationHelper through MTConnect.XsdPreprocessor before handing
the XSD source to the BCL XmlSchemaSet. Tests and library share a
single source of truth — if the preprocessor's 1.0-compatible
projection ever drifts, both sides surface the regression.
After this change every XSD in tests/Compliance/.../Schemas/v*
compiles cleanly under the BCL reader (including the 32 XSDs
previously tagged [Category("XsdLoadStrict")] for the 1.1-only
constructs they carry). The category is retained as a structural
marker — the schemas DID need the preprocessor — but its tests no
longer fail under the BCL reader.
The XML-Tests helper improvement is incidental: it now loads the same
XSDs the compliance tests load, via the same preprocessor.
Adds Xsd11FeaturePresenceTests as the sibling fixture to
SchemaLoadTests: where the load fixture proves that the
1.0-compatible subset compiles cleanly through XsdPreprocessor, this
fixture parses each XSD on the XsdLoadStrict roster as raw XDocument
XML and asserts the 1.1-only constructs are actually there.
Two-direction fencing:
* Load fixture catches an upstream change that drops a 1.1 feature
we relied on the preprocessor stripping (the schema would then
compile cleanly under the BCL reader, and the load test would
keep passing — no regression).
* This fixture catches an upstream change that drops the 1.1
feature itself: if a schema is on the XsdLoadStrict roster but
no longer carries any 1.1 marker, the per-schema feature-presence
test fails and the roster tag becomes editable down.
XsdLoadStrict roster is exposed as internal static so this fixture
drives off the same source of truth.
Feature OR-set surveyed per schema (32 parametric tests, one per
schema on the roster):
* xs:any[@notNamespace] / xs:anyAttribute[@notNamespace]
* xs:any inside xs:all (1.1 group-all extension)
* xs:element[@maxOccurs > 1] inside xs:all (1.1 cardinality
relaxation)
* xs:all inside xs:extension / xs:restriction (1.1 derivation
extension)
Plus two sentinel tests:
* No_schema_carries_xs_assert_or_xs_override_yet — asserts none of
the rostered schemas use xs:assert / xs:override (the preprocessor
strips them for forward-compatibility but the shipped XSDs don't
exercise that path yet).
* notNamespace_attribute_value_is_targetNamespace_token — pins the
WG's editorial convention that notNamespace uses
'##targetNamespace' (catches a typo or a future literal-URI-list
edit at review-time).
Replaces WithBindMount(hostPath, containerPath) with
WithResourceMapping(byte[], containerPath) in the two Testcontainers
fixtures that mount config / fixture files into spun-up containers:
* MqttBrokerFixture (eclipse-mosquitto) — the mosquitto.conf was
written to a host /tmp dir and bind-mounted into the container.
* CppAgentParityWorkflowTests (mtconnect/agent) — agent.cfg and the
fixture Devices.xml were staged to a host /tmp dir and bind-mounted.
On Docker-in-Docker setups (e.g. bluefin) the test-host container's
view of /tmp doesn't match the Docker daemon host's view; the
bind-mount resolves to an empty directory inside the started
container. WithResourceMapping pushes bytes through the Docker API
(observed in the Testcontainers log as "Add file to tar archive" +
"Copy tar archive to container") and is path-agnostic.
The cppagent fixture keeps the host-side staging dir because the
in-process .NET agent's DeviceConfiguration.FromFile requires a file
path; the change is that the cppagent container no longer touches
that path. The MQTT fixture drops the staging dir entirely since
mosquitto.conf was only needed for the bind-mount.
Verified locally against native Docker (Docker 26.1.4 on Fedora 38):
MqttRelayWorkflowTests + CppAgentParityWorkflowTests both green after
the refactor.
…ndows leg keeps Docker-category filter Drop the blanket `--filter "Category!=RequiresDocker&Category!=XsdLoadStrict"` exclusion on both legs and move the filter into the `strategy.matrix.include` block so each OS leg runs the categories it can physically support: * `ubuntu-latest` — empty filter (the full sweep). Docker is pre-installed on GitHub-hosted Linux runners and Testcontainers can spawn the `eclipse-mosquitto` + `mtconnect/agent` images the `RequiresDocker` / `E2E` suites need; the MTConnect XSDs ship in-repo, so `XsdLoadStrict` also has every prerequisite. * `windows-latest` — `Category!=RequiresDocker&Category!=E2E`. Hosted Windows runners do NOT pre-install Docker for Linux containers (only Windows containers via Hyper-V), so suites that boot `eclipse-mosquitto:2.0.22` / `mtconnect/agent:latest` cannot run there. `XsdLoadStrict` has no platform dependency and still runs. The unit-tests step and the integration-tests step are updated symmetrically, both reading the matrix-resolved `TEST_FILTER` env var and using a bash conditional so an empty filter expression passes no `--filter` flag (avoiding the "match nothing" interpretation `dotnet test --filter ""` would otherwise apply). Together with the campaign-added compliance + E2E suites in PR TrakHound#165, this gives the cppagent JSON v2 MQTT envelope fix a full end-to-end green-bar against a real `eclipse-mosquitto` broker + the cppagent reference parser on every push to master and every ready PR — closing the gap where the e2e tests were physically present but never run.
…sdLoadStrict
XsdValidationHelper now mirrors the L1 SchemaLoadTests fixture: it
pre-seeds the W3C xml.xsd + xlink.xsd (sibling files under Schemas/w3c)
into the XmlSchemaSet so the MTConnect XSD's <xs:import namespace='.../
xlink' schemaLocation='xlink.xsd'/> resolves by target-namespace match
without resolver traffic. Without the seed, every v2.7 envelope
validation tripped on "Type 'xlink:hrefType' is not declared".
The helper also injects a chameleon-style sidecar XSD in the MTConnect
target namespace declaring XYZEntryType as a non-abstract restriction
of ThreeDimensionalEntryType. The spec leaves the base abstract with
no concrete derivative, so an instance <Entry> inside an XYZDataSetType
sequence cannot satisfy XSD 1.0 validation without a client-side
xsi:type. Test envelopes opt in by tagging Entry elements
xsi:type="mt:XYZEntryType".
MotionEnvelope picked up two further v2.7-conformance fixes:
- Header@deviceModelChangeTime is required (added).
- Motion@coordinateSystemIdRef="mc" needed a real CoordinateSystem
with id="mc" so the IDREF resolves (added a minimal
CoordinateSystems/CoordinateSystem block inside the Configuration).
AxisDataSet_with_W_key_inside_devices_envelope_fails_xsd was renamed
to _is_xsd_valid_at_NMTOKEN_layer: KeyType in v2.7 is a bare
xs:NMTOKEN restriction with no enumeration facet, so 'W' is valid at
the schema layer. The XYZ-narrowing contract is enforced by the
deserialiser (sibling test
AxisDataSet_with_illegal_W_key_is_dropped_and_does_not_corrupt_xyz);
this case now pins the XSD-side behaviour so a future KeyType
tightening surfaces as a breaking change.
All four pre-existing XsdLoadStrict failures cleared; XML-Tests 98/98
and compliance 218/218 pass.
…n through XsdPreprocessor Moves Multi_device_probe_envelope_validates_against_MTConnectDevices_2_7_xsd from PR TrakHound#165 onto this PR, where the new MTConnect.XsdPreprocessor is in scope. The test loads the official v2.7 MTConnectDevices XSD through the preprocessor and asserts the multi-device probe envelope produced by the production XML formatter validates against the resulting XmlSchemaSet. Companion tests pinning the same envelope shape via XDocument structural assertions remain on TrakHound#165 — they don't depend on the preprocessor. The previous bluefin full-sweep at integration tip 253816e caught the gap: the test method existed on TrakHound#165 but its dependency (the XSD 1.1 preprocessor wired into XmlValidator) lives on this PR. Each PR is now self-contained.
ottobolyos
added a commit
to ottobolyos/mtconnect.net
that referenced
this pull request
May 24, 2026
… chore PR that owns the XSD 1.1 preprocessor The Multi_device_probe_envelope_validates_against_MTConnectDevices_2_7_xsd method depends on MTConnect.XsdPreprocessor, which is introduced by PR TrakHound#166 (chore/test-infra-cleanup). To keep each PR self-contained, the test method and its LoadDevicesV27Schema() helper move to TrakHound#166's L1 compliance suite alongside the preprocessor. The companion XDocument-based shape and case assertions stay on this PR — they don't depend on the preprocessor.
PatrickRitchie
approved these changes
May 24, 2026
PatrickRitchie
moved this from Reviewing
to Ready to Merge
in MTConnect.NET-Development
github-project-automation
Bot
moved this from Ready to Merge
to Done
in MTConnect.NET-Development
ottobolyos
added a commit
to ottobolyos/mtconnect.net
that referenced
this pull request
May 24, 2026
… chore PR that owns the XSD 1.1 preprocessor The Multi_device_probe_envelope_validates_against_MTConnectDevices_2_7_xsd method depends on MTConnect.XsdPreprocessor, which is introduced by PR TrakHound#166 (chore/test-infra-cleanup). To keep each PR self-contained, the test method and its LoadDevicesV27Schema() helper move to TrakHound#166's L1 compliance suite alongside the preprocessor. The companion XDocument-based shape and case assertions stay on this PR — they don't depend on the preprocessor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Single chore that combines five test-infra fixes (and one library-side fix that turned out to be the real motivator) so the campaign's compliance tests can run with no category filter in CI.
What's in here
1. Library —
MTConnect.XsdPreprocessorinMTConnect.NET-XMLXmlValidator.Validatepreviously fed schemaXml directly intoXmlSchema.Read, which goes through the .NET BCL XSD 1.0 reader. The MTConnect XSDs (every published version from v1.3 onwards) use XSD 1.1 constructs —xs:assert,xs:override,notNamespace/notQNameonxs:anyandxs:anyAttribute,xs:anyinsidexs:all,maxOccurs > 1insidexs:all. Each of those tripped the reader withXmlSchemaException: The 'notNamespace' attribute is not supported in this context(or the analogous diagnostic), and the load step returned a list of useless errors to every consumer that ever passed an official spec XSD.Adds
MTConnect.XsdPreprocessor.StripXsd11Constructs(string)as a public static helper that returns the XSD 1.0-compatible subset of an input schema. Documented removal rules:xs:assert/xs:override— removed wholesale (no 1.0 equivalent).notNamespace/notQNameonxs:anyAttribute— attribute stripped (element falls back tonamespace='##any').notNamespace/notQNameonxs:any— element removed (the##anyfallback would create a unique-particle-attribution conflict adjacent to concrete elements that XSD 1.0 rejects).xs:anyinsidexs:all— removed (XSD 1.0 disallows it).xs:elementwithmaxOccurs > 1insidexs:all—maxOccursclamped to 1.XmlValidator.Validateis wired through the preprocessor; the L1 compliance tests use the same library helper — single source of truth, no test-vs-prod drift.2. Compliance tests — split into 1.0-compat load + 1.1-feature presence
L1_XsdValidation/SchemaLoadTests.cscontinues to assert "the 1.0-compatible subset loads cleanly viaXmlSchemaSet" (now via the shared preprocessor). The newL1_XsdValidation/Xsd11FeaturePresenceTests.cs(203 lines, ~20 test methods) parses the raw XSD source viaXDocumentand asserts presence + correctness of the XSD 1.1 features for every published schema version. Coverage that #150's broader sweep was trying to achieve, now without tripping the BCL reader.3. DinD path-agnostic
RequiresDocker/E2EtestsL2_CrossImpl/CppAgentParityWorkflowTests.csandWorkflows/MqttBrokerFixture.csrefactored so Testcontainers no longer receives host paths that the daemon can't resolve under Docker-in-Docker socket-sharing. On hosted Windows runners (Linux Docker unavailable) the tests still skip via the workflow filter; on native-Docker ubuntu they run cleanly.4. Coverlet
3.2.0→6.0.4(per-csproj, four refs)coverlet.collector 3.2.0hangs inMono.Cecil.Rocksinstrumenting Scriban-generated.g.csDLLs (MTConnect.NET-Common.dllspecifically). 6.0.4 line resolves it. The four out-of-sync.csprojfiles now match the 6.0.4 baseline used elsewhere.5. CI
dotnet.ymlmatrix-include restructurePer-OS
testFilterviastrategy.matrix.include:ubuntu-latest— empty filter (the full sweep:RequiresDocker+E2E+XsdLoadStrictall run; Docker is pre-installed on the hosted Linux runner, Testcontainers can spawneclipse-mosquitto+mtconnect/agent, and the XSDs ship in-repo).windows-latest—Category!=RequiresDocker&Category!=E2E(Linux-image Docker not available on hosted Windows runners;XsdLoadStrictstill runs there).Unit-test step and integration-test step both read the matrix-resolved
TEST_FILTERenv var via a bash conditional, so an empty filter passes no--filterflag (avoiding thedotnet test --filter """match nothing" trap).Why one PR
These five changes are inter-locked: the L1 test refactor needs the library preprocessor, the CI restructure needs the L1 tests to pass on the no-filter ubuntu leg, and the coverlet bump is needed for the L1 tests to even complete without instrumentation hang. Landing them separately would leave at least one of the gates red. The PR is reviewable per-commit (six atomic commits in order).
Sequencing
This PR is intended to land before PR #165 (the
JsonMqttResponseDocumentFormatterenvelope bug fix). #165 currently carries the campaign-added compliance L1 + L2 + E2E tests; with this PR's library preprocessor + CI restructure in place, #165's CI runs the full sweep cleanly.Test plan
coverage.cobertura.xmlwithout hang on theMTConnect.NET-Commontest pass.cc @PatrickRitchie