DRAFT Add experimental data encodings - #4007
Draft
marcschier wants to merge 144 commits into
Draft
Conversation
Flip the 'managedPool' parameter default from false to true so Windows/Linux jobs route to the netstandard Managed DevOps Pool via ImageOverride demands. Operators can still override to Microsoft-hosted agents per queue via the pipeline UI.
Squash-merges the Protobuf (gRPC) DataEncoding port into the stack: codec in Stack/Opc.Ua.Types/Encoders/Protobuf, gRPC wrappers in Stack/Opc.Ua.Core/Encoders/Protobuf. Includes decoder hardening (nesting-depth guard, Proto.Parse bounds -> BadDecodingError), enum array/matrix encode fix, and union/optional discriminator carried on the wire. net8+ only; excluded on legacy TFMs.
Squash-merges the Avro DataEncoding port into the stack (Stack/Opc.Ua.Types/Encoders/Avro + shared SchemaId/SchemaExchange; Libraries/Opc.Ua.PubSub/Encoding/Avro + schema-exchange/cache infra). Includes decoder hardening (nesting-depth guard, Max*Length limits) and pooled-buffer release in the schema-exchange codecs. net8+ only; excluded on legacy TFMs.
Squash-merges the Apache Arrow DataEncoding port into the stack (Stack/Opc.Ua.Types/Encoders/Arrow + SchemaExchange/Arrow*; Libraries/Opc.Ua.PubSub/Encoding/Arrow; re-adds Arrow branches to the shared SchemaCache/SchemaExchangeMessages). Adds Apache.Arrow 18.1.0 referenced conditionally on net8+. Includes decoder hardening (header-column guard, catch filter, list-offset bounds) and a thread-safe schema cache. net8+ only; excluded on legacy TFMs.
Add EncoderCompat (Unsafe.As bit-casts + hex) and span-based Stream/Encoding polyfills so the Avro and Protobuf Part 6 encoders compile and run on net472/net48/netstandard2.0/netstandard2.1. Arrow stays net8.0+ (Apache.Arrow). Modern .NET fast paths are unchanged. Validated: all TFMs build; 126 round-trip tests pass on net10.0 and 53 on net48.
Enable the Avro Part 14 network message encoder/decoder, the shared schema-exchange support and PubSubMessageEncoding on net472/net48/netstandard2.1. Arrow-specific members in the shared SchemaCache and SchemaExchangeMessages are gated with #if NET8_0_OR_GREATER; Convert.ToHexString/FromHexString are replaced with the cross-TFM Utils equivalents. Arrow PubSub stays net8.0+ (Apache.Arrow). Validated: all PubSub TFMs build; Avro/SchemaExchange tests pass on net10.0.
Apply the Experimental(UA_NETStandard_1) attribute to the public Part 6 encoders/decoders, SchemaId, schema-exchange records, the Protobuf gRPC service-message wrappers, and the Part 14 PubSub network/dataset message types, network message encoders/decoders, SchemaCache and ISchemaResolver. The repo .editorconfig sets UA_NETStandard_1 severity=none so the in-repo build stays clean while downstream consumers get the evaluation-only signal.
Delete the hand-rolled Protobuf codec (Opc.Ua.Types/Encoders/Protobuf), the Protobuf gRPC service-message wrappers (Opc.Ua.Core/Encoders/Protobuf) and ProtobufRoundTripTests. Remove the EncodingType.Protobuf enum member and every reference (EncoderCommon encoder/decoder cases and datapoint groups, the EncodeableTests validation switch). No external Protobuf dependency existed and nothing else referenced the codec. Builds clean on net10.0.
…ions) AvroDecoder.ReadVariantValue passed dimensions=null to ReadVariantBody, whose array-vs-matrix branch keyed off that dimensions array, so every matrix Variant was mis-decoded as an array and ran past the end of the stream. Route on ValueRank (0/1 => array, >1 => matrix), mirroring TypeInfo.IsArray/IsMatrix and the encoder's WriteVariantBody dispatch. Avro now passes the full Part 6 shared matrix (EncodeMatrixInArray + all encodeable single/array/matrix).
Arrow does not yet reconstruct IEncodeable/ExtensionObject bodies (the decoder returns the raw binary body) nor Enumeration Variants. Guard those specific scenarios with Assume.That so the shared matrix stays green while Avro is fully integrated. Encoder suite: 3743 passed, 0 failed on net10.0.
Verify default registration keeps classic and async factory paths separate and the Reference Server helper flags remain reusable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the current ISA95 JobControl NodeSet2 input as a focused source-generator regression without changing generator or codec behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add TranscodeEncoding.Avro and its transport-profile mapping/classification, project transcoded messages to AvroNetworkMessage/AvroDataSetMessage in the NetworkMessageProfileProjector (ToAvro + same-encoding options + DataSetClassId), and register the Avro INetworkMessageEncoder/Decoder in DI so routes can transcode to and from Avro alongside UADP and JSON. The Avro encoder already generates and announces per-DataSet schemas progressively (announce-once via SchemaCache) during EncodeAsync.
Add SchemaCache.Reset to clear cached schemas and per-destination announcement state, and register the Avro NetworkMessage encoder/decoder as transient so each transcoding bridge owns its progressive-schema state. A route reload recreates the bridge (fresh schema state = reset), a DataSet MetaData version change re-announces automatically (the descriptor embeds the version), and Reset offers an explicit programmatic reset.
Cover the Avro transport-profile classification, projection of UADP/Avro to the Avro mapping, announce-once progressive schema generation, re-announcement on a DataSet MetaData version change, SchemaCache.Reset re-announcement, and an Avro network-message encode/decode round-trip. 8 tests pass on net10.0.
Note the experimental Avro NetworkMessage mapping as a transcoding target alongside UADP and JSON, and describe the progressive schema generation and reset behaviour (announce-once, auto re-announce on MetaData version change, SchemaCache.Reset, and per-bridge reset on route reload).
Replace the incomplete resource with the unmodified official ISA95 JobControl 2.0.0 NodeSet and assert its model, node coverage, and local references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Verify cancellation preserves the caller token and stops the connect attempt started by ManagedSession.CreateAsync. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ex helper - Add end-to-end coverage: a UADP->Avro transcode through PubSubTranscoder (frame produced via ResolveEncoder + encoder-map keying, then decoded), Avro->UADP and Avro->JSON projections, and a real-DI test asserting AddPubSub registers the Avro INetworkMessageEncoder/Decoder keyed by the Avro transport profile URI. Extend the shared Encoders() helper and add DecodeAvroAsync. - Remove the unused EncoderCompat.ToLowerHexString (orphaned by the Protobuf removal; duplicated CoreUtils.ToHexString) and its now-unused using. All 12 affected tests pass on net10.0; Types builds clean on net10.0 + netstandard2.0.
Keep the caller-visible cancellation and primary disposal path covered while excluding only the defensive secondary disposal-failure classifier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…it tests - Remove the 4 EncoderCompat bit-cast helpers orphaned by the Protobuf removal (SingleToUInt32Bits/UInt32BitsToSingle/DoubleToUInt64Bits/UInt64BitsToDouble had zero uses); keep the 4 signed helpers used by the Avro reader/writer. - Add EncoderCompatTests (round-trips the kept bit-casts over NaN/+-0/min/max/inf). - Add EncodingStreamPolyfillTests exercising Stream.ReadExactly/Read(Span)/Write(Span) and Encoding.GetString/GetBytes span overloads incl. empty-input branches; these are TFM-agnostic so the net48 leg covers the legacy polyfill implementations. - Add a projector test for the Avro same-encoding rebuild path (FieldContentMask preserved, field encoding retargeted). 33 Types.Tests pass on net10.0 and net48.
…/Enumeration) Arrow now runs the full shared Part 6 encoder matrix ungated: - Enumeration Variant (scalar/array/matrix) encoded as Int32 columns. - ExtensionObject/IEncodeable bodies are binary-serialized on encode and reconstructed to the concrete IEncodeable on decode via the message-context factory (raw binary body retained when the type is unregistered). - Implemented the previously-throwing WriteEncodeable*/ReadEncodeable* surface (scalar/array/matrix + AsExtensionObject) using a binary-body representation. - Fixed top-level composite array writers (NodeId/ExpandedNodeId/QualifiedName/ LocalizedText/ExtensionObject) to use the multi-element *ManySlot builders. Removed the 5 Arrow Assume.That skip-gates; added Arrow round-trip tests (encoder 90.7%, decoder 95.8%, A.cs 98.4% line coverage). Documented the experimental Avro/Arrow encodings + Avro transcoder. Validated on net10 (Encoders 3904 pass, PubSub 1345 pass) and net48 (Encoders 3361 pass).
A Part 14 DataSetMessage may be sparse (it does not carry a value for every declared key). Mirror the opcua-drafts nullable-keys design so a sparse subset reuses the SAME schema as the full key frame instead of announcing a new one; an absent key is missing (null:null / null column cell), distinct from a present null-valued field and from a delta frame's 'absent = unchanged'. Arrow: FindField now resolves a column by explicit DataSet FieldIndex, then by field name, and only falls back to positional placement when the message carries the full (dense) field set; a key absent from a sparse message yields a null column cell (missing). The Arrow IPC schema is already metadata-driven, so full and sparse frames serialize byte-identical schemas. Descriptor/announce SchemaId: BuildSchemaDescriptor now derives the field list from DataSetMetaData (all declared keys, metadata types) when available, so a sparse Avro or Arrow frame produces the identical descriptor and SchemaId as the full key frame and does not trigger a re-announcement. Falls back to present fields when no metadata is resolvable. Tests: add Avro + Arrow sparse round-trip tests asserting a subset frame shares the full-frame schema/SchemaId and decodes present keys with absent keys missing/null.
- SeedDocument and FederatedDocument are ByteString instead of byte[], per the repo rule preferring ByteString in public API. This also drops a copy on the seed publish path, and the null checks move to .IsNull as INullable requires. - WriteDocumentAsync no longer allocates a byte[] per chunk. ByteString.From(ReadOnlyMemory) copies, so the fix is the ByteString(ReadOnlyMemory) constructor, which interns the slice without copying. Added a test asserting the streamed chunks reassemble into the original document. - Added XRegistryClient.RegistryNodeId. The documented GetOrCreateGroupAsync snippet omitted the registry root argument and would not have compiled, and there was no supported way to locate the root without Browsing. - Rewrote all three NuGet READMEs. The base one still claimed the NodeSet was embedded and that XRegistryWellKnown held resource/method NodeIds; the client and server ones still described the retired CreateResource/Write/Close lifecycle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
- Renamed InMemoryXRegistryResourceStore to InMemoryResourceStore and FileSystemXRegistryResourceStore to FileSystemResourceStore. The XRegistry prefix was redundant inside the Opc.Ua.XRegistry.Server namespace. - Replaced the value tuples returned by the client convenience API with ResourceRegistrationResult and GroupRegistrationResult readonly record structs, matching the result type convention used across the stack. All three tuple returning methods were converted rather than only the one flagged, so the convenience layer stays consistent. RegisterResourceAsync now reports Created as well; it drives the strict CreateResource, so it only returns when it created the version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
The base branch replaced the client's value tuples with named record structs, added a chunk-size parameter to the resolve path, renamed the resource store implementations, and moved the seed/federation documents to ByteString. - SchemaRegistryClient returns GroupRegistrationResult and ResourceRegistrationResult instead of tuples, and forwards the new maxByteStringLength argument on ResolveSchemaAsync. - SchemaRegistryOptions constructs InMemoryResourceStore under its new name and holds the seed and federated schema documents as ByteString, matching the ByteString-over-byte[] preference the base now follows. - The Schema Registry integration tests read those documents through .Span. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
Four defects found in review, all around the shared content-addressed fast-path node and the write handle: - The fast-path node is shared by every resource whose document has the same bytes, but its lifetime was tied to whichever resource happened to be deleted first, which silently broke resolution for the others. Its lifetime is now reference counted: publishing takes a reference and the node is only unpublished once the last resource that resolves to those bytes lets it go. - Re-writing a resource with different content left the superseded content id published forever. Closing a write handle now releases the previous content id before taking a reference on the new one, and skips the churn entirely when the bytes did not change. - Committing a document wrote at offset 0 without truncating, so replacing a version with a shorter one left the tail of the previous version behind. The stored document is now removed before the new one is written, which is what the store contract prescribes for a wholesale replacement. - GetOrCreateResource handed out a write handle on the existing resource path without checking MaxConcurrentUploads, and the client never closed that handle when it had not created the version, so every idempotent re-registration leaked one handle. The bound now covers every path that hands out a handle and the client always releases it. Also widened the source generator value decoding factory to register every encodeable type exported by the built-in assembly rather than only Argument, so a NodeSet carrying any other standard structure decodes too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
OnFileReadAsync took the handle's Position outside the lock and only advanced it after awaiting the store, so two overlapping Reads on one handle both started at the same offset: both returned the same slice and the cursor then skipped one. StoreKey and Writing are immutable for the life of the handle, but Position is not. The cursor is now taken and the range reserved inside the same lock that validates the handle, and the optimistic reservation is corrected down afterwards so a short read at the end of the document leaves the cursor at the real end rather than past it. Added a gated resource store that holds both readers inside ReadAsync until each has taken its offset, which makes the overlap deterministic instead of timing dependent. The test fails against the old code with both reads starting at offset 0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
The spec defines ExpectedEpoch as optimistic concurrency where a non-zero value that does not match the entity's epoch fails Bad_InvalidState and 0 disables the check. The implementation compared for equality unconditionally, and since Epoch starts at 1 and only increments, passing 0 could never succeed: the documented force path was unreachable on Delete for resources and groups and on AddAttribute and RemoveAttribute. Also syncs the compiled NodeSet2 byte for byte with the regenerated spec model so the spec stays the single source of truth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
Review found that the idempotent-registration fix landed earlier was destructive. GetOrCreateResource hands out a write handle even when it returned an existing version, and the client releases it without writing; Close treated every write handle as a completed upload, so it erased the stored document, re-fingerprinted the resource as empty and dropped the fast-path node. A handle now tracks whether anything was written and Close only commits when it was. File handles were also server-wide, sequential and unvalidated: a caller could drive another resource's document through its own resource's Methods, and Close would store one resource's bytes while marking a different one. A handle now records the resource it was opened on and is rejected elsewhere. Close was gated unconditionally on the write policy, so a read handle opened on a channel the read policy allows could never be closed and leaked, permanently consuming the upload budget. Close is now gated on what the handle actually does. Handles open on a deleted resource were never released, and a repeated or racing Delete double-released the shared fast-path reference and drifted the registration count. Removal is now idempotent and drops the resource's handles. Close could also publish a fast-path node for a resource deleted while the store call was in flight; it now re-validates before touching fast-path state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
Open reduced the mode to a single bool. Per OPC 10000-5 the combinations with neither Read nor Write, with both, and with EraseExisting or Append but no Write are now rejected with BadInvalidArgument. More importantly EraseExisting and Append are honoured: a write-open that does not erase seeds the buffer from the stored document and overwrites at the cursor, so a partial rewrite no longer silently truncates the rest of it. The inherited Size and OpenCount Properties are kept current. Auto-assigned VersionId could collide with a version the caller created explicitly, which failed the create outright or made GetOrCreateResource return an unrelated version. The counter is now scoped to the owning group, advances past any existing version, and is pruned when a resource's last version goes so the map stays bounded. File handles are now bound to the session that opened them (a null session id means an in-process call) and released when that session closes, so an abandoned session no longer holds the upload budget forever. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
Repo guidance mandates ByteString over ReadOnlyMemory in public API, and the store surface was inconsistent with itself since ReadAsync already returned ByteString. Switched IXRegistryResourceStore.WriteAsync, both stores, WriteDocumentAsync and the two client registration methods. ByteString wraps ReadOnlyMemory with no copy, so the per-chunk allocation removed earlier stays removed. XRegistryServerOptions is now sealed. The three node managers stay unsealed on purpose: subclassing is the server-side extension seam a domain registry uses, mirroring how a domain client derives from XRegistryClient, and that is now stated in their docs. Added XRegistryServiceCollectionExtensions so the server pieces wire into DI, with direct construction still supported. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
Adds the coverage the plan called for but the previous commits omitted: version assignment (skipping an explicitly created version, per-group scoping, counter pruning on delete) and session ownership (a handle rejected from another session, and a closing session releasing its handles). Both suites were verified to fail against the pre-fix behaviour. Two existing security tests were opening a handle in-process and then operating on it from a real session, which the session check now correctly rejects. They are restructured to hold the session constant and vary only the channel, so they test the security policy rather than two rules at once. Also covers the new service collection extensions, taking the patch from 95.8 to 98.5 percent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 19cb13c4-ea7c-4f1c-b6a6-ab037e418c6b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
The base branch moved the client's document parameter from ReadOnlyMemory<byte> to ByteString, matching the repository's preference for ByteString over byte[] / ReadOnlyMemory<byte> in public API. SchemaRegistryClient takes ByteString on RegisterSchemaAsync and GetOrRegisterSchemaAsync, and SchemaRegistrySink now hands the notification's schema document straight to the client instead of copying it into a new array on every publish. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
marcschier
added a commit
that referenced
this pull request
Jul 28, 2026
# Description xRegistry is the *abstract registry base model*: a registry-agnostic, content-addressed resource registry. It knows nothing about what a resource contains — a concrete registry supplies its own companion namespace and a fingerprinting strategy and reuses everything else. The PubSub Schema Registry in #4007 is the first specialization (its resources are schema documents), and #4093 (WoT Binding) carries a byte-identical copy of the same base NodeSet, so a shared canonical home is worth having. ## Packages | Package | Depends on | Contains | | --- | --- | --- | | `OPCFoundation.NetStandard.Opc.Ua.XRegistry` | `Opc.Ua.Core` | Abstract base companion model (compiled by the model source generator), `XRegistryWellKnown`, `IResourceContentIdProvider` | | `OPCFoundation.NetStandard.Opc.Ua.XRegistry.Client` | + `Opc.Ua.Client` | `XRegistryClient` (abstract) / `GenericXRegistryClient` (sealed), built on the generated ObjectType proxies | | `OPCFoundation.NetStandard.Opc.Ua.XRegistry.Server` | + `Opc.Ua.Server` | Fast-path / registration / federation node managers, `XRegistryServerOptions`, `IXRegistryResourceStore` | `Opc.Ua.XRegistry` deliberately depends on neither SDK, so a codec or a shared contracts assembly can reference the identity abstraction without pulling in the client or the server. ## The model is compiled, not loaded The companion model is compiled into the assemblies by the OPC UA model source generator. `Opc.Ua.XRegistry.NodeSet2.xml` is an `AdditionalFiles` generator input only — there is no `EmbeddedResource`, no runtime NodeSet parsing and no hand-written NodeId tables for anything the model declares. All three node managers load it the same way: ```csharp protected override NodeStateCollection LoadPredefinedNodes(ISystemContext context) { return new NodeStateCollection().AddOpcUaXRegistry(context); } ``` `XRegistryWellKnown` is reduced to the namespace URI plus the identifiers of the *instances* a registry materializes at runtime, which live above the model's own 63000-63999 range so they can never collide with it. ## What it does - **Content-derived identity** — `IResourceContentIdProvider` maps a document + format to a fingerprint and names the (canonicalization, hash) algorithm. Identity is derived from the bytes, not assigned by the server, so it is stable across registries — which is what makes de-duplication and federation work. - **Opaque-NodeId fast path** — a registered resource is reachable at an Opaque `NodeId` whose Identifier is the raw content-id bytes, so a consumer that received the id on the wire resolves the document in a single `Read`: no Browse, no fingerprint recomputation. The client reads it through the `ReadBytesAsync` session extension, so a document larger than `MaxByteStringLength` is fetched with range-based reads rather than failing. Optionally pre-publishes a seed resource so a fresh server can resolve one resource before any registration. - **The model's own lifecycle** — the registry root is a `RegistryType` instance serving `CreateGroup` / `GetOrCreateGroup`; each group serves `CreateResource` / `GetOrCreateResource`; and because `ResourceType : FileType`, the document *is* the file — content is streamed through the inherited `Open` / `Read` / `Write` / `Close`. `Close` fingerprints the accumulated bytes and publishes the Opaque fast-path node at runtime. Identical bytes reuse the existing node, whose lifetime is reference counted so it survives until the last resource resolving to those bytes is gone and is retired when a resource is re-written with different content. - **Optimistic concurrency** — `Delete(ExpectedEpoch)` on both resources and groups, and the same epoch check on label mutations, so a caller working from a stale read is rejected rather than clobbering a concurrent change. - **Labels** — `RegistryType`, `GroupType` and `ResourceType` each expose an `AttributesType` Object with `AddAttribute` / `RemoveAttribute`; labels are published as addressable `String` Properties. - **Federation** — a resource hosted by another registry is published as a real `ResourceType` instance whose `ExternalReference` (`ExpandedNodeId` + `ServerIndex` into the local `ServerArray`) and `ResourceUrl` carry the link and whose `Xid` carries the content-id. Because it is an ordinary `ResourceType`, a generic client drives a federated resource through exactly the same proxy as a local one. ## Extensibility A domain registry subtypes the base model — the PubSub Schema Registry declares `SchemaFileType : ResourceType` — and the generator mirrors that hierarchy across assemblies, so the proxy chain follows automatically (`SchemaFileTypeClient : ResourceTypeClient : FileTypeClient`). ```text abstract XRegistryClient ├── sealed GenericXRegistryClient // any registry namespace ├── SchemaRegistryClient (domain) └── WotRegistryClient (domain) ``` * A **domain client** derives from `XRegistryClient` and inherits the whole lifecycle. Helpers written as extension methods over a base proxy (such as `WriteDocumentAsync` on `ResourceTypeClient`) are callable on the domain proxy with no inheritance in the client layer. * A **generic client** still drives a domain registry, because a domain instance *is* an instance of the base type. This is covered by a test that defines a domain subclass over the base. ## State Document bytes live behind an injectable `IXRegistryResourceStore`. Because a resource is a `ResourceType`, which *is* a `FileType`, the store mirrors the file access model: reads and writes are **offset and length based**, so a document never has to be materialized as a whole and a ranged read touches only the bytes it needs. The document is transferred with the inherited `FileType` Methods, whose mode bits are honoured: a write that does not set `EraseExisting` starts from the stored bytes and replaces only the range it writes rather than truncating the rest. A handle is valid only on the resource and the session that opened it, and a session's handles are released when it closes. Two implementations ship: `InMemoryResourceStore` (the default, in-process) and `FileSystemResourceStore`, built on the `IFileSystem` abstraction so documents outlive the process and a shared volume can back a cluster. `XRegistryResourceStoreContractTests` is a reusable fixture that validates any implementation against the contract. ## Transport security Registry **writes always require a `SignAndEncrypt` secure channel** — a document and its content-derived identity are integrity-critical, so every mutation is rejected with `BadSecurityModeInsufficient` on a channel that is only signed or unprotected. This is not configurable. Reads are allowed on any secure channel by default; `RequireEncryptionForReads` extends the requirement to them when the documents themselves are confidential. An in-process call carries no channel and is always allowed. ## Hardening The lifecycle Methods are remotely callable, so every unbounded dimension is bounded via `XRegistryServerOptions`; exceeding a bound fails the call rather than the server: | Option | Default | Enforced on | Status code | | --- | --- | --- | --- | | `MaxConcurrentUploads` | 64 | `CreateResource`, `Open` | `BadTooManyOperations` | | `MaxResourceBytes` | 16 MiB | `Write` | `BadRequestTooLarge` | | `MaxRegisteredResources` | 4096 | `CreateResource` | `BadTooManyOperations` | ## A source generator bug fixed along the way Building on the compiled model surfaced a defect in shared generator infrastructure. `NodeSetToModelDesign.CreateDecoder` decoded values with an empty `EncodeableFactory`, so a NodeSet2's method `InputArguments` / `OutputArguments` — encoded as a list of `Argument` ExtensionObjects — never decoded, while `HasArguments` was still set. **Every** NodeSet2-sourced model in the stack was silently generating argument-less `MethodState`s and ObjectType proxies. ModelDesign-sourced models (the core stack) were unaffected, which is why this went unnoticed. Fixed in `b87726af` with a regression test that fails without it. Two defects in the xRegistry NodeSet itself were fixed at the same time: `BrowseName="1:InputArguments"` → ns-0 `InputArguments` (published NodeSets never namespace-qualify these), and the `CreateResource` / `GetOrCreateResource` **output** `VersionId` renamed to `AssignedVersionId`, which collided with the input of the same name and produced a duplicate C# parameter. ## Other fixes applied while extracting - `XRegistryRegistrationNodeManager` uses `System.Threading.Lock` instead of an `object`-typed sync root, per the repository locking rules. - Dropped the `InternalsVisibleTo` entries granting internals to `Opc.Ua.PubSub.Server.Tests`. That project only consumes public `XRegistryWellKnown` API. - The `UA.slnx` edit in #4007 accidentally **replaced** the `tests/Opc.Ua.Redundancy.Kubernetes.Tests` entry instead of adding a line, silently dropping that project from the solution. The entry is preserved here and the change is purely additive. ## Tests `tests/Opc.Ua.XRegistry.Tests` (168) covers the group and resource lifecycles, the `FileType` transfer, delete and epoch conflicts, labels, federation, both resource stores, the fast path, transport security and the extensibility contract — positive and negative paths. Passing on `net10.0` and `net48`. Patch line coverage is **98.5%**. Also adds `tests/Directory.Build.props` / `.targets`: the .NET 10 SDK prunes `Microsoft.Extensions.Logging.Abstractions` out of the test output, and coverlet's Mono.Cecil resolver only searches the output directory, so it silently fails to instrument every module that references `ILogger` and reports 0% coverage for them. These force the package back into the restore graph and copy its implementation next to the test binary. ## Documentation New [`docs/XRegistry.md`](docs/XRegistry.md), linked from the documentation index.
#4097 was squash-merged into master, so master now carries the xRegistry libraries as a single commit with no shared history with the copies this branch carried while it was stacked on that PR. Resolve every xRegistry path in favour of master, which is now authoritative and also newer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4 # Conflicts: # docs/README.md # src/Opc.Ua.XRegistry.Server/XRegistryRegistrationNodeManager.cs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
Findings from a security review of the experimental Avro/Arrow encodings and the schema-exchange paths. All of these parse attacker-controlled bytes: PubSub NetworkMessages arrive over UDP/MQTT/Kafka, and the Schema Registry Close method fingerprints a document supplied by a remote session. Availability: - ArrowSchemaCanonicalForm walked the schema type tree recursively with no depth guard, so a schema nesting list/struct/union thousands of levels deep raised StackOverflowException. That cannot be caught in .NET, so it terminated the whole server process rather than failing the request. TypeCode now carries a depth and rejects beyond MaxSchemaNestingDepth. Because the Arrow reader itself recurses while materializing the schema, before this class ever sees it, ComputeSchemaIdFromIpc also bounds the document at MaxIpcSchemaLength first. - ArrowNetworkMessageDecoder pre-allocated a List from RecordBatch.Length, which is declared in the FlatBuffers metadata and is not cross-checked against the body. A few hundred bytes could therefore ask for an 800 MB array. The row count is now bounded by MaxArrayLength, matching the Avro decoder, and the capacity hint is gone. - AvroDecoder.ReadArray accumulated across an unlimited number of blocks without ever consulting MaxArrayLength, unlike the Binary and XML decoders. The running total is now checked. - A.ReadListAt sliced using raw list offsets, and the Read*Many helpers allocate eagerly from the resulting length, so bad offsets allocated before failing. The range is now validated against the values array, as ReadListVariant already did. Integrity: - SchemaCache.Add verified the announced SchemaId against the recomputed fingerprint and then stored last-writer-wins. A SchemaId is only 64 bits, so a peer able to reach Add could announce a document colliding with a cached id and replace the schema that id decodes with. The first document bound to an id is now kept; re-announcing the identical document remains a no-op and a differing one is rejected. - AvroSchemaIdProvider caught only FormatException and JsonException, but AvroParsingCanonicalForm also throws KeyNotFoundException and InvalidOperationException on malformed schema JSON, which escaped into the registry Close handler. It now reuses SchemaExchangePayload.IsMalformedPayload alongside the two it already handled. The content id stays the 64-bit wire SchemaId deliberately: it is the Opaque NodeId a consumer resolves a schema by, so widening it would break the fast path. The residual collision exposure is in the registry's first-writer-wins fast-path dedup, which lives in Opc.Ua.XRegistry.Server and is outside this change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 000aa394-d085-4bf5-b605-a7adbe7d27f4
The Avro schema announcement carried a bespoke JSON descriptor rather than an Avro schema. The SchemaId provider implements CRC-64-AVRO over the Parsing Canonical Form correctly, but it caught the resulting parse failure and fell back to hashing the raw bytes, so the descriptor still produced a well-formed looking SchemaId. The value was stable and self-consistent within this library while being unreproducible by any other implementation - a defect that a round-trip test against ourselves cannot see. AvroSchemaBuilder now generates a self-contained .avsc for the DataSet from the DataSetMetaData, following the value mapping of section 5 and the generation algorithm of section 6.2: fields in declaration order, Avro name conversion, null-first nullable unions, the array and matrix forms, and named types inlined at first occurrence and referenced afterwards. The record shapes follow the normative opcua.builtins.avsc from the specification rather than being invented here. Variant and ExtensionObject use the shared form of section 6.6, which is what the published reference schemas use. AvroSchemaLineage accumulates the observed field types per DataSet so the Variant body union grows append-only, as section 5.8 requires. Building each schema only from the message at hand would reorder or drop branches whenever a Variant field carried a new body type, silently changing the index of every existing branch and making previously written values undecodable. The fingerprint fallback is removed: a document that does not parse as a schema has no defined SchemaId and is now rejected. SchemaCache.Add wraps that failure so the ingest path keeps its existing exception contract for hostile input. Two tests asserted behaviour the specification rules out, both because the old descriptor embedded the ConfigurationVersion in the "schema": - a MinorVersion bump with an unchanged DataSet shape was expected to produce a new SchemaId and re-announce. Section 6.3 states the SchemaId identifies the Parsing Canonical Form and specifically not a PubSub ConfigurationVersion, and section 8.4 has the schema change advance the version, not the reverse. Both tests now grow the DataSet shape to grow the schema, and assert that a version bump alone changes nothing. The schema-cache bounds test relied on the removed raw-bytes fallback to mint distinct ids from integers; it now uses distinct real schemas, which tests the same eviction behaviour without depending on a non-conformant path. Tests 1410/0/1 on net8.0 (baseline 1401 plus 9 new); net48 builds clean.
Section 6.7 of the Avro companion specification allows a DataSetMetaData to be used as an alternative input to schema generation, which is what a translation bridge or an existing JSON verbose publisher needs: both hold the metadata for the writers they forward but usually have no session with the originating server and therefore no AddressSpace to read DataTypeDefinitions from. AvroDataSetSchema.Create generates the schema, and optionally its SchemaId, from a DataSetMetaDataType alone. AvroMetaDataTypeResolver consumes the type descriptions the metadata already carries: StructureDataTypes expand to records with the optional-field wrapper of section 5.6 and the switch/value form of section 5.7, EnumDataTypes map to the numeric int of section 5.3, and SimpleDataTypes map through their base built-in type. Recursive structures reference the enclosing record by name so the document stays finite. The specification requires this path to produce the schema the encode-time path produces, and therefore the same SchemaId, since otherwise a SchemaId would no longer identify one canonical schema. That is enforced structurally rather than by two implementations that are merely intended to agree: the encoder now projects its metadata-driven fields through the same collection routine and the same builder. A test asserts the two documents are byte-identical. Metadata is not always complete, and incompleteness is detected rather than papered over. A field whose DataType is neither a built-in nor declared by the metadata fails generation, because substituting an opaque type would emit a schema that looks correct while silently losing the structure. Fields[].MaxStringLength, DataSetFieldId, Properties and Description are deliberately not read: they constrain or annotate values but must not alter the Parsing Canonical Form. The ConfigurationVersion is likewise not an input. Field framing is not carried by the metadata at all and is applied afterwards from the DataSetFieldContentMask, so the same DataSet yields a different schema for RawData, Variant and DataValue framing. Tests 1422/0/1 on net8.0; net48 builds clean.
Section 9 of the Avro companion specification defines how the Avro message mapping is selected and configured in the Part 14 PubSub configuration model. Nothing of it existed: encoders were chosen only by TransportProfileUri, and the sole consumer of MessageSettings read JsonWriterGroupMessageDataType. Opc.Ua.Avro.NodeSet2.xml adds the namespace http://opcfoundation.org/UA/Avro/ with the two content-mask OptionSets of section 9.2, the three MessageSettings DataTypes of section 9.3 and the three ObjectTypes of section 9.4. Each MessageSettings DataType subtypes its abstract Part 14 base, so a WriterGroup, DataSetWriter or DataSetReader selects the Avro mapping by carrying one of them in MessageSettings, exactly as UADP and JSON are selected today. It ships as an independent Information Model extension, embedded in the assembly and fed to the model source generator, following the Schema Registry NodeSet already in this project. The base NodeIds were read out of Opc.Ua.NodeSet.xml rather than assumed. That mattered: the values suggested by documentation search were wrong for all three MessageSettings bases, and the WriterGroupMessageType ObjectType is i=17998 and not the adjacent id it is easy to reach for. The model deliberately adds no DataTypeEncoding Object and no HasEncoding reference for Default Avro, because section 4.2 states the encoding has no AddressSpace representation and is named only for symmetry with Default Binary, Default XML and Default JSON. A payload is identified by its SchemaId. A test asserts the absence so a later edit cannot quietly reintroduce one. AvroWellKnown carries the provisional NodeIds, matching how the Schema Registry exposes its own; the OPC Foundation assigns the final identifiers. The content masks and MessageSettings are also expressed as C# types, and tests check every mask bit against the NodeSet so the model and the code cannot drift into agreeing on a name while disagreeing on its bit. NodeSet validation reports 0 errors. Tests 1436/0/1 on net8.0; net48 builds clean.
Mirrors the Avro configuration model for the Arrow message mapping (Arrow companion specification section 6.5). Opc.Ua.Arrow.NodeSet2.xml adds the namespace http://opcfoundation.org/UA/Arrow/ with the three supporting enumerations, the three MessageSettings DataTypes and the three ObjectTypes. Each MessageSettings DataType subtypes its abstract Part 14 base, so a WriterGroup selects the Arrow mapping by carrying one in MessageSettings. The section 6.1 mapping parameters that until now existed only as encoder properties - framing, batching target, schema metadata, delta frame mode and compression - are expressed as configuration. Arrow wire output is unchanged; the round-trip tests confirm it. One trap is worth naming, because it is invisible at the type level. The configuration model numbers ArrowIpcFormatEnum as Batch = 0, Stream = 1, File = 2, while the encoder's internal ArrowIpcFraming declares Stream first. Casting between the two would compile, run, and select exactly the opposite framing. The conversion is therefore written out by name, and a test asserts both that the two enumerations really are numbered differently and that the conversion maps each member correctly. The File framing is rejected rather than downgraded to Batch, since silently emitting a payload the configuration did not ask for is worse than failing. The converter is guarded to the frameworks that build the Arrow encoder, because ArrowIpcFraming only exists there. The configuration types themselves are deliberately not guarded: a configuration tool must be able to read and write an Arrow WriterGroup on any target, including one that cannot encode Arrow. Tests 1451/0/1 on net8.0; net48 builds clean.
Enum.Parse<T> is not available on net48, so the content-mask tests used the non-generic overload with a cast. Building only the src project for net48 hid this, because the test project multi-targets too; the net48 test run is what surfaced it. Tests 1451/0/1 on net8.0 and 1424/0/1 on net48 (Arrow is net8.0-only).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds two experimental OPC UA DataEncodings to the stack — Avro and Apache Arrow — covering boththe Part 6 encoder/decoder surface and Part 14 (PubSub) network messages, and removes the previously prototyped Protobuf/gRPC encoding. All new codec surface is annotated with
[Experimental("UA_NETStandard_1")], so consumers must opt in by acknowledging that diagnostic; the wire format and API may change without a major-version bump.What changed
Part 6 encoders (
Stack/Opc.Ua.Types/Encoders)AvroEncoder/AvroDecoder(with a self-containedAvroBinaryReader/AvroBinaryWriter) andArrowEncoder/ArrowDecoder(columnar Apache Arrow), plus theEncodingType.Avro/EncodingType.Arrowenum members and codec self-identification.
they are exercised by the identical built-in /
Variant/ExtensionObject/Enumerationtest cases.net472,net48,netstandard2.0/2.1,net8.0+) via thespan/stream/
Encodingpolyfills inOpc.Ua.Types/Polyfills; thenet8.0+ fast paths are preserved sothere is no net10 performance regression. Arrow targets
net8.0+ only (Apache.Arrow dependency),and its source files self-exclude on the legacy frameworks via
#if NET8_0_OR_GREATERguards.IEncodeable/ExtensionObjectround-trip (bodies are binaryserialized and reconstructed to the concrete
IEncodeablevia the message-contextEncodeableFactory,falling back to the raw binary body for unregistered types) and
EnumerationVariants(scalar/array/matrix), so it runs the shared matrix ungated.
PubSub / Part 14 (
Libraries/Opc.Ua.PubSub)handshake (
SchemaCache, schema announcements/requests,ISchemaResolver). Avro and the schema exchangebuild on every target framework; only the Arrow adapter is
net8.0+.NetworkMessageProfileProjector,TranscodeEncoding) with progressive schema generation and reset (SchemaCache.Reset()).Experimental JSON schema exchange
Extends the experimental schema-exchange handshake (previously Avro + Arrow) to JSON, producing a real
JSON Schema (json-schema.org draft 2020-12) describing the JSON DataSetMessage encoding by reusing the
existing
Opc.Ua.Core.Schemagenerator. The feature is opt-in and OFF by default and strictly additive:the standard
JsonNetworkMessageencode path is unchanged with zero overhead unless enabled.JsonSchemaAnnouncement/JsonSchemaRequestrecords +SchemaId.JsonSchemaId;SchemaCache.JsonFormatand
Add(JsonSchemaAnnouncement);IDataSetJsonSchemaProvider/DataSetJsonSchemaProvider.JsonEncoder/JsonDecoder(EnableSchemaExchange,LastSchemaAnnouncement,Ingest), plus JSON participation in the transcoder's progressive announce/reset.AddJsonSchemaExchange(...)+PubSubApplicationOptions.EnableJsonSchemaExchange; adirect-construct fallback is also available. All new public API is
[Experimental("UA_NETStandard_1")].DataTypeDefinitionpre-registered in the
DataTypeDefinitionRegistry(built-in / scalar / array fields work out of the box).Note: source paths above predate the repository reorganization on
master(Stack/Libraries->src,Tests->tests), which this branch has been merged up to. The coverage-collection fix(
tests/Directory.Build.props+.targets) that restores coverlet instrumentation ofOpc.Ua.Types/Opc.Ua.Coreshipped with #4097 and is already onmaster.Documentation
Docs/PubSub.md(Encodings + Transcoding sections) andStack/Opc.Ua.Types/Encoders/readme.mddocumentthe experimental Avro/Arrow encodings and the Avro transcoder. (The 2.0 migration sub-docs intentionally
omit these additive experimental features.)
Validation
Opc.Ua.Core.Encoders.Tests— net10: 3904 passed / 0 failed (Arrow + Avro run the full shared matrixungated); net48: 3361 passed / 0 failed (Arrow excluded where Apache.Arrow is unavailable).
Opc.Ua.PubSub.Tests— net10: 1347 passed / net48: 1341 passed / 0 failed (Avro + schema exchangerun on every TFM; only Arrow tests are net8.0+).