Support Dynamic Complex Types #2146

MarkusHorstmann · 2023-05-10T22:48:56Z

Proposed changes

Complex Type support assumes that every OPC type is represented by exactly one .Net type, which in turn knows encoding ids, XML names etc.
These changes enable use of a single type that implements encoding/decoding based on OPC type information at runtime.

Related Issues

Fixes JsonDecoder.ReadExtensionObject returns no or partial Json #2145

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply. You can also fill these out after creating the PR.

Bugfix (non-breaking change which fixes an issue)
Enhancement (non-breaking change which adds functionality)
Test enhancement (non-breaking change to increase test coverage)
Breaking change (fix or feature that would cause existing functionality to not work as expected, requires version increase of Nuget packages)
Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have read the CONTRIBUTING doc.
I have signed the CLA.
I ran tests locally with my changes, all passed.
I fixed all failing tests in the CI pipelines.
I fixed all introduced issues with CodeQL and LGTM.
I have added tests that prove my fix is effective or that my feature works and increased code coverage.
I have added necessary documentation (if appropriate).
Any dependent changes have been merged and published in downstream modules.

codecov · 2023-05-10T23:01:02Z

Codecov Report

Merging #2146 (0f1085a) into master (5f54e08) will increase coverage by 0.01%.
The diff coverage is 70.00%.

@@            Coverage Diff             @@
##           master    #2146      +/-   ##
==========================================
+ Coverage   57.90%   57.92%   +0.01%     
==========================================
  Files         324      324              
  Lines       61826    61833       +7     
==========================================
+ Hits        35801    35816      +15     
+ Misses      26025    26017       -8

Impacted Files	Coverage Δ
...ck/Opc.Ua.Core/Types/Encoders/EncodeableFactory.cs	`70.67% <40.00%> (-1.20%)`	⬇️
Stack/Opc.Ua.Core/Types/Encoders/XmlDecoder.cs	`76.65% <100.00%> (+0.08%)`	⬆️
Stack/Opc.Ua.Core/Types/Encoders/XmlEncoder.cs	`76.89% <100.00%> (+0.16%)`	⬆️

... and 7 files with indirect coverage changes

…2155) Cherry pick bugfix from #2146: ReadExtensionObject returns partial JSON for some types. It appears that this is due to the JSON being read before the JsonTextWriter is closed. see also: 2a4ba56 Co-authored-by: Markus Horstmann <markushorstmann@hotmail.com>

mregen · 2023-06-01T15:17:23Z

Hi @MarkusHorstmann , please elaborate how this extension can be used, or even better provide a test case / sample. I already cherry picked your bug fix, thanks!

Stack/Opc.Ua.Core/Types/Encoders/IComplexTypeInstance.cs

MarkusHorstmann · 2023-06-01T23:38:10Z

Hi @MarkusHorstmann , please elaborate how this extension can be used, or even better provide a test case / sample. I already cherry picked your bug fix, thanks!

I have added test cases using a dynamic encodeable with simple type information (just a list of string-valued fields). For a realistic use case feel free to look at the CESMII NodeSet utilities. It is not meant to be a sample, though.

mregen · 2023-06-14T09:50:18Z

Hi @MarkusHorstmann, looks pretty good, any idea why the tests are failing?

mregen · 2023-06-14T09:53:07Z

Check failure on line 1 in Opc.Ua.Core.Tests.Types.Encoders.JsonEncoderTests.Test_ExtensionObjectWithDynamicEncodeable

@azure-pipelinesazure-pipelines
/ OPCFoundation.UA-.NETStandard
Opc.Ua.Core.Tests.Types.Encoders.JsonEncoderTests.Test_ExtensionObjectWithDynamicEncodeable
System.InvalidOperationException : Collection was modified; enumeration operation may not execute.

mregen

please fix the test cases

…plicitly

MarkusHorstmann · 2023-06-14T22:00:58Z

please fix the test cases

The new tests should now run fine. There are two other tests failing (ReadOnDiscoveryChannel and AppVeyor build) that don't seem to be related to my changes, and I've seen them fail in other branches as well. Please let me know if you think otherwise.

MarkusHorstmann · 2023-06-14T22:01:48Z

Oops, thought I was closing the review comment, not the PR. Reopened it.

mregen

Hi @MarkusHorstmann, we discussed the changes today but some questions came up. We need to follow up in a seperate thread.

mregen · 2023-06-15T14:30:07Z

Stack/Opc.Ua.Core/Types/Encoders/IComplexTypeInstance.cs

+        /// </summary>
+        /// <returns>The xml qualified name for the type instance</returns>
+        XmlQualifiedName GetXmlName(IServiceMessageContext context);
+    }


It is unclear why the IServiceMessageContext is needed here

A IComplexTypeInstance implementation needs the Encodable Factory and the Namespace Table from the context. The reason is that the implementation may need to make additional nested types in a structure available via the Factory. Otherwise, all known types would have to be pre-registered, which largely defeats the purpose of a truly dynamic (= meta-data-driven) type implementation.

@mregen Please let me know if you need any additional information or maybe want to discuss on a call?

Hi @MarkusHorstmann , we discussed the PR yesterday in the user group and it is accepted, just taking a final check then its ready to go. Thanks!

Stack/Opc.Ua.Core/Types/Encoders/XmlEncoder.cs

Stack/Opc.Ua.Core/Types/Encoders/XmlDecoder.cs

gabrielgriesser · 2023-06-21T09:52:38Z

Hello, I'm interested about this feature. I have to create Structure Type (so ExtensionObject) XML from my autoDiscovery code.

I am able to auto discover my StructureDefinition :

`
foreach (var result in browseResults)
{
if (result.NodeClass != NodeClass.DataType)
continue;

        var varNodeId = NodeId.Parse(result.NodeId.ToString());

        if (Session.ReadNode(varNodeId) is DataTypeNode dataTypeNode && (dataTypeNode.DataTypeDefinition != null))
            if (result.DisplayName.Text.Contains("ST_"))
            {
                Debug.WriteLine("     DatTypeNode = {0}, NodeClass = {1}, NodeID = {2}, NodeFields = {3}", dataTypeNode.DisplayName.Text, dataTypeNode.NodeClass, dataTypeNode,
                    dataTypeNode.DataTypeDefinition);
                var sd = dataTypeNode.DataTypeDefinition.Body as StructureDefinition;

                // TODO Create my own XML from sd structure
            }


        BrowseRecursive(browser, varNodeId);
    }

`

But now, how can I dynamically create an XML file and how to use it to decode my ExtensionObject when I read my variable using Session.ReadValue() ?

MarkusHorstmann · 2023-06-27T02:52:10Z

Hello, I'm interested about this feature. I have to create Structure Type (so ExtensionObject) XML from my autoDiscovery code.

I am able to auto discover my StructureDefinition :

` foreach (var result in browseResults) { if (result.NodeClass != NodeClass.DataType) continue;
        var varNodeId = NodeId.Parse(result.NodeId.ToString());

        if (Session.ReadNode(varNodeId) is DataTypeNode dataTypeNode && (dataTypeNode.DataTypeDefinition != null))
            if (result.DisplayName.Text.Contains("ST_"))
            {
                Debug.WriteLine("     DatTypeNode = {0}, NodeClass = {1}, NodeID = {2}, NodeFields = {3}", dataTypeNode.DisplayName.Text, dataTypeNode.NodeClass, dataTypeNode,
                    dataTypeNode.DataTypeDefinition);
                var sd = dataTypeNode.DataTypeDefinition.Body as StructureDefinition;

                // TODO Create my own XML from sd structure
            }


        BrowseRecursive(browser, varNodeId);
    }
`

But now, how can I dynamically create an XML file and how to use it to decode my ExtensionObject when I read my variable using Session.ReadValue() ?

Hi, @gabrielgriesser! This Pull Request only provides the necessary hooks in the Core to make it possible implement dynamic types. I have implemented a dynamic type that I need in the context of a NodeSet editing tool, and it uses the editing tool's type system not the OPC SDK's type definition classes. The code is publically available here (https://github.com/cesmii/CESMII-NodeSet-Utilities/blob/main/src/CESMII.OpcUa.NodeSetModel.Factory.Opc/DynamicComplexType.cs). It should be relatively straightforward to change it to read the type information from a DataTypeDefinition instead of the NodeModel's DataTypeModel.

I also implemented a test case that implements a rudimentary dynamic type that handles only string values. This could be another starting point.

mregen · 2023-07-14T05:39:49Z

Stack/Opc.Ua.Core/Types/Encoders/IComplexTypeInstance.cs

+        /// </summary>
+        /// <returns>The xml qualified name for the type instance</returns>
+        XmlQualifiedName GetXmlName(IServiceMessageContext context);
+    }


Hi @MarkusHorstmann , we discussed the PR yesterday in the user group and it is accepted, just taking a final check then its ready to go. Thanks!

mregen · 2023-07-14T06:15:00Z

@MarkusHorstmann please merge master into your PR, the ci VMs had been updated recently so the failing build will not pass otherwise and i can't merge.

Complex Type support assumes that every OPC type is represented by exactly one .Net type, which in turn knows encoding ids, XML names etc. These changes enable use of a single type that implements encoding/decoding based on OPC type information at runtime.

MarkusHorstmann added 2 commits May 10, 2023 15:40

Enable dynamic complex types (type info driven instead of .Net types)

8fee34c

JsonDecoder: actually return json for ExtensionObjects

2a4ba56

XMLDecoder: decode empty arrays

2e9c6ab

mregen mentioned this pull request May 19, 2023

JsonDecoder: ReadExtensionObject returns partial JSON for some types #2155

Merged

13 tasks

mregen added 2 commits May 27, 2023 07:31

Merge branch 'master' into dynamiccomplextypes

9dc5ea3

Update JsonDecoder.cs

0f1085a

mregen reviewed Jun 1, 2023

View reviewed changes

Stack/Opc.Ua.Core/Types/Encoders/IComplexTypeInstance.cs Outdated Show resolved Hide resolved

MarkusHorstmann added 2 commits June 1, 2023 12:29

IDynamicComplexTypeInstance: remove unnecessary EncodingId methods

d399bd8

Add tests for a simple dynamic encodeable

482f1f7

Dynamic test: use better string nodeids

95d2b7c

mregen self-assigned this Jun 3, 2023

mregen added this to the 1.4.372 feature update milestone Jun 3, 2023

mregen added enhancement API or feature enhancement complex Complex Types labels Jun 3, 2023

mregen requested a review from mrsuciu June 14, 2023 09:49

mregen requested changes Jun 14, 2023

View reviewed changes

Test: Prevent collection modified exception, ensure field ordering ex…

ae78c6a

…plicitly

MarkusHorstmann closed this Jun 14, 2023

MarkusHorstmann reopened this Jun 14, 2023

mregen approved these changes Jun 15, 2023

View reviewed changes

mregen requested review from ThomasNehring and salihgoncu June 15, 2023 07:46

mregen requested changes Jun 15, 2023

View reviewed changes

mregen reviewed Jun 15, 2023

View reviewed changes

Stack/Opc.Ua.Core/Types/Encoders/XmlEncoder.cs Show resolved Hide resolved

mregen reviewed Jun 15, 2023

View reviewed changes

Stack/Opc.Ua.Core/Types/Encoders/XmlDecoder.cs Outdated Show resolved Hide resolved

Code review: remove XmlDecoder/Encoder changes, move to encodable.Encode

1ad9829

mregen approved these changes Jul 14, 2023

View reviewed changes

Merge branch 'OPCFoundation:master' into dynamiccomplextypes

a9f20ef

mregen merged commit 2738a29 into OPCFoundation:master Jul 16, 2023
43 of 44 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Dynamic Complex Types #2146

Support Dynamic Complex Types #2146

MarkusHorstmann commented May 10, 2023

codecov bot commented May 10, 2023 •

edited

mregen commented Jun 1, 2023

MarkusHorstmann commented Jun 1, 2023

mregen commented Jun 14, 2023

mregen commented Jun 14, 2023

mregen left a comment

MarkusHorstmann commented Jun 14, 2023

MarkusHorstmann commented Jun 14, 2023

mregen left a comment

mregen Jun 15, 2023

MarkusHorstmann Jun 26, 2023

MarkusHorstmann Jul 5, 2023

mregen Jul 14, 2023

gabrielgriesser commented Jun 21, 2023

MarkusHorstmann commented Jun 27, 2023 •

edited

mregen Jul 14, 2023

mregen commented Jul 14, 2023

Support Dynamic Complex Types #2146

Support Dynamic Complex Types #2146

Conversation

MarkusHorstmann commented May 10, 2023

Proposed changes

Related Issues

Types of changes

Checklist

codecov bot commented May 10, 2023 • edited

Codecov Report

mregen commented Jun 1, 2023

MarkusHorstmann commented Jun 1, 2023

mregen commented Jun 14, 2023

mregen commented Jun 14, 2023

mregen left a comment

Choose a reason for hiding this comment

MarkusHorstmann commented Jun 14, 2023

MarkusHorstmann commented Jun 14, 2023

mregen left a comment

Choose a reason for hiding this comment

mregen Jun 15, 2023

Choose a reason for hiding this comment

MarkusHorstmann Jun 26, 2023

Choose a reason for hiding this comment

MarkusHorstmann Jul 5, 2023

Choose a reason for hiding this comment

mregen Jul 14, 2023

Choose a reason for hiding this comment

gabrielgriesser commented Jun 21, 2023

MarkusHorstmann commented Jun 27, 2023 • edited

mregen Jul 14, 2023

Choose a reason for hiding this comment

mregen commented Jul 14, 2023

codecov bot commented May 10, 2023 •

edited

MarkusHorstmann commented Jun 27, 2023 •

edited