Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Test Object (3441) to LWM2M Registry. #639

Merged
merged 12 commits into from
Sep 28, 2021

Conversation

sbernard31
Copy link
Contributor

@sbernard31 sbernard31 commented Sep 10, 2021

Add new Test Object to LWM2M Registry.

The idea is to have an object which could help to do some interoperability test.
(Maybe it could even be used during test fest ?)

  1. There is RW resources for each data type as single resource and multiple resource.
  2. I also added some executable resource which allow to reset/clear/randomize resource value I feel this could help to test observe. Tell me if you think this makes sense or if some/all of those executable resources should be removed ?
  3. For now there is no Unsigned interger resource because this first version target LWM2M 1.0 but I plan to push a version which target LWM2M 1.1. I don't know if this is a good idea or if we should start directly with LWM2M 1.1 ?

Any other idea/resource addition is welcomed too.

This Leshan PR (eclipse-leshan/leshan#1093) allows to test the Test Object.

This PR followed discussion at #623.

(Corresponding issue #636)

- XML schema does not match LWM2MVersion value.
- Description2 is not well placed.
@sbernard31
Copy link
Contributor Author

@bocajim, @mlasch, @qleisan, @tuve, @rettichschnidi could you review this ? 🙏

@tuve
Copy link

tuve commented Sep 10, 2021

I like this but, how long is a string? Pun intended but the question is serious, when randomizing values would also the string length be randomized? Couldn't optional arguments be added to the randomize resource to control the length of the string and the number of multiple instances. This in itself is a valid test case.

@sbernard31
Copy link
Contributor Author

@tuve thx for feedback

About reset/clear/randomize arguments I have the idea that we could add as argument the resource ids which should be reset/cleared/randomized (no args means all resources) but I finally don't add it because I'm not sure this is really useful 🤔

About your concern about string size and multi value size, I'm not sure I get it, could you be more precise ? 🙏
Maybe you're afraid that size was too big ?
For testing could you describe the test you have in mind ?

Currently the random function is more for "visual"/"playing" test, I mean using leshan-server-demo you can observe the object instance than click on randomize and see values changes but I agree this is maybe not so useful too write real/deterministic tests. In this case maybe is better to use Write Request with precise value ?

@tuve
Copy link

tuve commented Sep 10, 2021

Testing the support for arguments, but it may also be useful to test that resources of a certain size are supported. This can of course be achieved deterministic by writing.

@sbernard31
Copy link
Contributor Author

sbernard31 commented Sep 10, 2021

Testing the support for arguments

OK so what do you think about the idea above ?
exec on Clear Values resource with arguments 10, 20 will clear only resource 10 and 20 ?

@tuve
Copy link

tuve commented Sep 10, 2021 via email

Replace the file name from 3441-1.0.xml to 3441-1_0.xml.
@sbernard31
Copy link
Contributor Author

(@jpradocueva thx for fixing file naming 🙏)

@tuve About testing support for arguments of executable resource.
Maybe my previous idea has a drawback.
To test result you need to read the whole object and ensure that only requested values changed.
Using a UI it is OK but maybe programmatically this is a bit painful ?

So another idea is to create 2 other resources :

  • 1 executable resource which accept parameters
  • 1 readable multi-instance String readable resource where each instance is a parameter pass to last exec of the resource above ☝️

WDYT ?

Copy link
Contributor

@akeranen akeranen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggesting small changes but overall LGTM.

3441.xml Outdated
<LWM2M xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://openmobilealliance.org/tech/profiles/LWM2M.xsd">
<Object ObjectType="MODefinition">
<Name>Test Object</Name>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More descriptive name would be good (perhaps something like "LwM2M Specification Test Object"?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the name by LwM2M Specification Test Object (commit 972ab9f)

xsi:noNamespaceSchemaLocation="http://openmobilealliance.org/tech/profiles/LWM2M.xsd">
<Object ObjectType="MODefinition">
<Name>Test Object</Name>
<Description1><![CDATA[This object contains resources for each datatype allowing easy interoperability testing.]]></Description1>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More details or reference to a spec with more details about how this is supposed to be used could be good here too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this could be a next step. Asking the person who did a lot of work already to do more work appears to be unfair to me.

3441.xml Outdated
<RangeEnumeration />
<Units />
<Description>Set random value to all resources. For
multiple value, the number of resource instances is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

meaning of "multiple value" is not clear here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should say "For multi-instance resources, the number of resource instances is also randomized."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it (commit be3c599)

I also add some advice about limiting size of randomized value. (See #639 (comment))

<Type>Opaque</Type>
<RangeEnumeration />
<Units />
<Description>Initial value must be "0x0123456789ABCDEF".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is the value encoded? In binary I suppose (not as ASCII characters)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure to get the point.
This is an Opaque resource, so not limited to ASCII chars.
About encoding it depends of ContentFormat used by the request.

(I feel I missed something 🤔)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant how the value "0x0123456789ABCDEF" is encoded to the resource. I assume you mean this is a sequence of 16 octets that have values (shown in hex) "0x0123456789ABCDEF". Alternative interpretation would be sequence of ASCII characters.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, 0x0123456789ABCDEF is the byte array value in Hexadecimal.
Do you feel the description is not clear enough ?

Should I replace by :

Initial value must be 0123456789ABCDEF (Hexadecimal notation)

OR

Initial value must be 0x0123456789ABCDEF (Hexadecimal notation)

OR

any other better idea ?

Copy link
Contributor

@akeranen akeranen Sep 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both are better than the original but if you want to be even more clear I would suggest something like:

Initial value is sequence of 16 octets with value (shown in hex) "0x0123456789ABCDEF".

I wonder if there is some precedent of such description in another object we could replicate here. But you probably know best what is the most clear description for developers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it (commit daabe13)

3441.xml Outdated
<Type>String</Type>
<RangeEnumeration />
<Units />
<Description>Initial value must be 1 instance with id 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<Description>Initial value must be 1 instance with id 0
<Description>Initial value must be 1 instance with ID 0

(and same with other instances of "id" too)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed by commit 8c7ee24

@tuve
Copy link

tuve commented Sep 21, 2021

(@jpradocueva thx for fixing file naming 🙏)

@tuve About testing support for arguments of executable resource.
Maybe my previous idea has a drawback.
To test result you need to read the whole object and ensure that only requested values changed.
Using a UI it is OK but maybe programmatically this is a bit painful ?over

So another idea is to create 2 other resources :

  • 1 executable resource which accept parameters
  • 1 readable multi-instance String readable resource where each instance is a parameter pass to last exec of the resource above ☝️

WDYT ?

Does it has to be multi instance resource? In a real life use case it kind of doesn't make sense to have an exec just overwrite another resource when you just as easily could do a write, but since this is a test object, it would be easier to use in tests and implement.

One exec resource with argument that writes the argument to a read only resource?

@hannestschofenig
Copy link
Contributor

Thanks for doing this work! We discussed this object at the last conference call and I have solicited reviews from the group. This is certainly a useful object for future interoperability events.

@kFYatek
Copy link
Contributor

kFYatek commented Sep 21, 2021

A couple of notes from me:

This object seems to be somewhat similar in intent to the private /33605 object that we use in Anjay demo application. There is no XML definition for that, but you can take a look at the list of resources in the source code.

I'm not suggesting to use our object instead - our object is rather Anjay-specific (it includes resources that only make sense in the context of Anjay internals), however I believe that some aspects of our version may be worth including in the official object:

  • Including all data types from LwM2M TS 1.1 and 1.2 (such as unsigned integers) might make sense.
  • We have a resource that you can Execute to increment the value of another integer resource (i.e. Execute /33605/1/2 will increment the value of /33605/1/1). This is useful for testing proper handling of packet retransmissions, by checking if a duplicated packet increments the value only once.
  • Keeping with the Execute operation, we also have resources that allow testing the Execute arguments by placing their values in another resource. For example, Execute /33605/1/2, Content: 1='test',7='123' will cause /33605/1/4, which is an array of strings, to be set so that /33605/1/4/1 is test and /33605/1/4/7 is 123. This might be useful for testing the Execute argument string parsing. In our object resource no. 2 performs both this task and the one mentioned above, but those very well might be separate resources.

These are just loose suggestions - you may or may not want to include before publishing this object, or they might be included in a subsequent version of it.

@dnav
Copy link
Member

dnav commented Sep 21, 2021

I just checked the test Object we are using internally. Let's just say that "great minds think alike".

As @kFYatek wrote, adding the 1.1 and 1.2 data types would be nice.

I also suggest having two additional resources: one read-only and one write-only.

Regards,

@sbernard31
Copy link
Contributor Author

sbernard31 commented Sep 23, 2021

About Including all data types from LwM2M TS 1.1 and 1.2

@dnav, @kFYatek, as explained in this PR description :

For now there is no Unsigned interger resource because this first version target LWM2M 1.0 but I plan to push a version which target LWM2M 1.1. I don't know if this is a good idea or if we should start directly with LWM2M 1.1 ?

I plan to put another version of the Object which targets LWM2M 1.1.
I don't know if I should do that in this PR or create a new one once we agree on the first version of this object.

@sbernard31
Copy link
Contributor Author

About testing Execute operation with Arguments

@tuve, @kFYatek, I created 2 resources to test it. (commit 8334d13 )

@sbernard31
Copy link
Contributor Author

sbernard31 commented Sep 23, 2021

About testing duplicate,

We have a resource that you can Execute to increment the value of another integer resource (i.e. Execute /33605/1/2 will increment the value of /33605/1/1). This is useful for testing proper handling of packet retransmissions, by checking if a duplicated packet increments the value only once.

@kFYatek , I'm not so sure for now so I didn't add it. But maybe if more feedback in favor of this, we can add it.

@sbernard31
Copy link
Contributor Author

sbernard31 commented Sep 23, 2021

About adding Read and Write only resource,

I also suggest having two additional resources: one read-only and one write-only.

@dnav, I'm not sure to understand if you want to :

  1. add 2 resources (1 read-only + 1 write-only)
  2. or if you want to add it 1 by type (string/opaque/...) and kind (single/multiple), so 28 new resources

I guess this is rather 1)
With #639 (comment) and commit 8334d13 we have a read-only resource.
For write-only resource, I can add it but I have no idea about its purpose and how we test it.
Or maybe you want to do a kind of RW resource but with 2 resources (1 for read and the other for write) ?

Maybe I totally missed your point, so please do not hesitate to elaborate on this.

@sbernard31
Copy link
Contributor Author

I updated Leshan PR to be compliant with last modification see eclipse-leshan/leshan#1093.

@dnav
Copy link
Member

dnav commented Sep 28, 2021

About adding Read and Write only resource,

I also suggest having two additional resources: one read-only and one write-only.

@dnav, I'm not sure to understand if you want to :

  1. add 2 resources (1 read-only + 1 write-only)
  2. or if you want to add it 1 by type (string/opaque/...) and kind (single/multiple), so 28 new resources

I guess this is rather 1) With #639 (comment) and commit 8334d13 we have a read-only resource. For write-only resource, I can add it but I have no idea about its purpose and how we test it. Or maybe you want to do a kind of RW resource but with 2 resources (1 for read and the other for write) ?

Maybe I totally missed your point, so please do not hesitate to elaborate on this.

My suggestion was item 1. The goal was to test that allowed operations are respected. But this can be done with already existing Objects. So forget about it.

@mkgillmore mkgillmore merged commit 974ae3a into OpenMobileAlliance:test-object Sep 28, 2021
@sbernard31
Copy link
Contributor Author

I don't know if this should have been merged.

At least there are this #639 (comment) which is not addressed.

@mkgillmore
Copy link
Contributor

mkgillmore commented Sep 28, 2021 via email

@sbernard31
Copy link
Contributor Author

In support of your project we wanted to move forward with this submission.

🙏

If you wish to clarify how "0x0123456789ABCDEF" is encoded, we can always revise the object.

Now the object is merged, I'm not sure to understand the process to clarify the "0x0123456789ABCDEF" description ?
Should I create a new PR againts the test-object branch ?

OMA met today and we agreed to support a lwm2m 1.0 object and in the future separate objects for 1.1, 1.2, etc

You mean :

  1. a new Object with a different ID and name ?
  2. or a new version for this Object (so with same ID and name) ?

My idea initially was rather 2) but I vaguely understand that we rather go with 1). Do I get correctly ?

@mkgillmore
Copy link
Contributor

mkgillmore commented Sep 28, 2021 via email

@sbernard31
Copy link
Contributor Author

The group (OMA) agreed to allocate 3441 for lwm2m 1.0 xsd test, 3442 for 1.1 and 3443 for 1.2

Does this help?

I understand now.
I guess there is pros and cons for both solutions.

  1. Same object ID/Name consumes less IDs and seems more consistent. Because this seems really to be different version of the same object and this will be more like any other objects are managed.
  2. Different object ID/Name consumes more IDs, seems less consistent but will make easier to handle changes on this Object.

As 2. was chosen, should we change the name accordingly ? I feel that LwM2M Specification Test Object is no more adapted.
Should it be renamed as LwM2M v1.0.x Specification Test Object or LwM2M v1.0 Specification Test Object

@sbernard31
Copy link
Contributor Author

sbernard31 commented Sep 29, 2021

As 2. was chosen, should we change the name accordingly ? I feel that LwM2M Specification Test Object is no more adapted.
Should it be renamed as LwM2M v1.0.x Specification Test Object or LwM2M v1.0 Specification Test Object

Thinking a bit more about those names seem too long to me so maybe :

LwM2M v1.0 Test Object or LwM2M v1.0 Specification Tester or LwM2M v1.0 Tester

@sbernard31
Copy link
Contributor Author

sbernard31 commented Sep 29, 2021

I created a new PR #642 for final polish of this object for LWM2M v1.0.

I think the last missing point is the name question.
My preference name order (if we keep 1 ID/name by LWM2M version) :

  1. LwM2M v1.0 Tester
  2. LwM2M v1.0 Test Object
  3. LwM2M v1.0 Specification Tester
  4. LwM2M v1.0 Specification Test Object

Once we agree with that I will provide a PR for LWM2M v1.1 Test Object.

Thx all for your valuable feedbacks 🙏 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants