LwM2M interoperability tests [0 - 399]

[SeppoTakalo]

I have implemented LwM2M interoperability tests from the test specification:
OMA Enabler Test Specification (Interoperability) for Lightweight M2M

This PR contains test specified in following areas:

• Bootstrap Interface: [0-99]
• Registration Interface [100-199]
• Device management & Service Enablement Interface [200-299]
• Information Reporting Interface [300-399]

Tests and environment are documented in tests/net/lib/lwm2m/interop/README.md

Current status

Test case	Status	Notes
LightweightM2M-1.1-int-0 - Client Initiated Bootstrap	✅
LightweightM2M-1.1-int-1 - Client Initiated Bootstrap Full (PSK)	✅
LightweightM2M-1.1-int-101 - Initial Registration	✅
LightweightM2M-1.1-int-102 - Registration Update	✅
LightweightM2M-1.1-int-103 - Deregistration	🔶	We don't have "disabled" functionality in server object
LightweightM2M-1.1-int-104 - Registration Update Trigge	✅
LightweightM2M-1.1-int-105 - Discarded Register Update	✅
LightweightM2M-1.1-int-107 - Extending the lifetime of a registration	✅
LightweightM2M-1.1-int-108 - Turn on Queue Mode	✅
LightweightM2M-1.1-int-109 - Behavior in Queue Mode	✅
LightweightM2M-1.1-int-201 - Querying basic information in Plain Text	✅
LightweightM2M-1.1-int-203 - Querying basic information in TLV format	✅
LightweightM2M-1.1-int-204 - Querying basic information in JSON format	✅
LightweightM2M-1.1-int-205 - Setting basic information in Plain Text	✅
LightweightM2M-1.1-int-211 - Querying basic information in CBOR format	✅
LightweightM2M-1.1-int-212 - Setting basic information in CBOR format	✅
LightweightM2M-1.1-int-215 - Setting basic information in TLV format	✅
LightweightM2M-1.1-int-220 - Setting basic information in JSON format	✅
LightweightM2M-1.1-int-221 - Attempt to perform operations on Security	✅
LightweightM2M-1.1-int-222 - Read on Object	✅
LightweightM2M-1.1-int-223 - Read on Object Instance	✅
LightweightM2M-1.1-int-224 - Read on Resource	✅
LightweightM2M-1.1-int-225 - Read on Resource Instance	✅
LightweightM2M-1.1-int-226 - Write (Partial Update) on Object Instance	✅
LightweightM2M-1.1-int-222 - Read on Object	✅
LightweightM2M-1.1-int-223 - Read on Object Instance	✅
LightweightM2M-1.1-int-224 - Read on Resource	✅
LightweightM2M-1.1-int-225 - Read on Resource Instance	✅
LightweightM2M-1.1-int-226 - Write (Partial Update) on Object Instance	✅
LightweightM2M-1.1-int-227 - Write (replace) on Resource	✅
LightweightM2M-1.1-int-228 - Write on Resource Instance	✅	#64011
LightweightM2M-1.1-int-229 - Read-Composite Operation	✅	#64012 #64189
LightweightM2M-1.1-int-230 - Write-Composite Operation	✅
LightweightM2M-1.1-int-231 - Querying basic information in SenML JSON format	✅
LightweightM2M-1.1-int-232 - Querying basic information in SenML CBOR format	✅
LightweightM2M-1.1-int-233 - Setting basic information in SenML CBOR format	✅
LightweightM2M-1.1-int-234 - Setting basic information in SenML JSON format	✅
LightweightM2M-1.1-int-235 - Read-Composite Operation on root path	🔶	Root Path is not yet supported by Leshan.
LightweightM2M-1.1-int-236 - Read-Composite - Partial Presence	✅
LightweightM2M-1.1-int-237 - Read on Object without specifying Content-Type	✅
LightweightM2M-1.1-int-241 - Executable Resource: Rebooting the device	✅
LightweightM2M-1.1-int-256 - Write Operation Failure	✅
LightweightM2M-1.1-int-257 - Write-Composite Operation	✅
LightweightM2M-1.1-int-260 - Discover Command	✅
LightweightM2M-1.1-int-261 - Write-Attribute Operation on a multiple resource	🔶	Leshan don't allow writing attributes to resource instance
LightweightM2M-1.1-int-280 - Successful Read-Composite Operation	✅
LightweightM2M-1.1-int-281 - Partially Successful Read-Composite Operation	✅
LightweightM2M-1.1-int-301 - Observation and Notification of parameter values	✅
LightweightM2M-1.1-int-302 - Cancel Observations using Reset Operation	✅
LightweightM2M-1.1-int-303 - Cancel observations using Observe with Cancel parameter	🔶	Leshan only supports passive cancelling
LightweightM2M-1.1-int-304 - Observe-Composite Operation	✅
LightweightM2M-1.1-int-305 - Cancel Observation-Composite Operation	🔶	Leshan only supports passive cancelling
LightweightM2M-1.1-int-306 – Send Operation	✅	#64290
LightweightM2M-1.1-int-307 – Muting Send	✅
LightweightM2M-1.1-int-308 - Observe-Composite and Creating Object Instance	✅	#64634
LightweightM2M-1.1-int-309 - Observe-Composite and Deleting Object Instance	✅	#64634
LightweightM2M-1.1-int-310 - Observe-Composite and modification of parameter values	✅
LightweightM2M-1.1-int-311 - Send command	✅
LightweightM2M-1.1-int-401 - UDP Channel Security - PSK Mode	✅

• ✅ Working OK.
• 🔶 Feature or operation not implemented.
• 🔴 Broken

Bugfixes

As I'm writing tests, I'll try to write bugfixes as well. Those mentioned bugs are fixed in this branch, I'll submit a fix to main separately.

• net: lwm2m: Fix composite operations #64016
• net: lwm2m: Fix overlapping buffers #64015
• net: lwm2m: Fix overlapping buffers from Portfolio object #64636

[SeppoTakalo]

This PR is now ready to be reviewed & merged. Tests are passing and I have implemented testcases 0 - 399 from the spec, if those can be implemented. Found issues have been fixed, and fixes have been merged.

Work can continue to implement the remaining ones in a separate follow up PR.

[jukkar]

Impressive work 👍
LGTM

[jvermillard]

Be careful with Leshan demo HTTP API; it's not stable and tightly coupled with the demo UI

[sbernard31]

Hi, I'm Leshan project lead.

Glad to see most of the test are ✅.

Regarding the comment concerning Leshan :

🔶 Root Path is not yet supported by Leshan.

We are working on it. I hope this will be available soon.

🔶 Leshan don't allow writing attributes to resource instance

I will try to take time next week to verify that but it should work... 🤔

🔶 Leshan only supports passive cancelling

This is supported by the library but not server-demo.
I plan to add it because it's the only way to cancel observation with coap+tcp

About using leshan-server-demo and leshan-bsserver-demo :

Even if I understand why people reuse it, our demos are not really intended to be reused. Especially in production.
For tests, this is maybe less a problem but you should keep in mind that API could be break at any moment.

See : https://github.com/eclipse-leshan/leshan/wiki/F.A.Q.#could-i-reuse-leshan--demo-

But once we say that a question remains : "what should be the good way to do that ?" 🤔
If you want to discuss about that : eclipse-wakaama/wakaama#617

[SeppoTakalo]

@sbernard31

Hi, I'm Leshan project lead.
Glad to see most of the test are ✅.
Nice to meet you, and thank you for taking interest on this PR.

Myself, I'm not a maintainer here, but I have been accepted as a "collaborator" in Zephyr's LwM2M stack. It means that I'm a contributor whose review also counts as approval. Past two years me and my team have been quite actively developing this stack forward.

Even if I understand why people reuse it, our demos are not really intended to be reused. Especially in production. For tests, this is maybe less a problem but you should keep in mind that API could be break at any moment.

I am aware of this guidance. However, the REST API on it, makes if by far the most easiest interface to attach into when we run tests using Pytest.
Since the code we develop is C and interactive tests are written in Python using this new integration of our test framework and Pytest, I don't want to introduce yet another layer there, for example by introducing another language.

Having no promises on API stability is a fact that we can live with. When tests break, we can roll back to some known version tag and continue using that until we update the tests.

I'll go through the discussion chain that you pointed out. I might leave some comments as well, if there is anything I think is worth mentioning.

[rlubos]

LGTM