Future: Freezing DSDL signature

[pavel-kirienko]

It is expected that some data types will require modifications affecting their signature but not their binary layout. E.g. a message containing a voidX definition may require an update so that the unused padding bits will be replaced with an actual field. Such a change does not affect binary compatibility, hence it makes sense to retain the DSDL signature unchanged.

For this purpose, the following DSDL directive may be added:

@signature <value>

Example:

@signature 0x12345678

[antoinealb]

I am not sure this is the best approach because it might break the compatibility detection system. I don't have a better approach in mind though.

[pavel-kirienko]

@antoinealb The whole idea is to use this feature only in the case if the compatibility will not be affected, i.e. to prevent false-positive compatibility break detections. There is an example of exactly that case described in the linked issue.

[kjetilkjeka]

Consider the following uavcan message

uint2 status
void2 reserved1
void4 reserved2

and its extension

uint2 status

uint2 STATE_IDLE = 0
uint2 STATE_TRANSMITTING = 1
uint2 STATE_RECEIVING = 2
uint2 STATE_ERROR = 3
uint2 state

void4 reserved2

If the old signature is retained one can never use the state info as it's impossible to know if the node never gets out of idle or it's simply an old message.

Even the use case in #7 would require knowledge that if 0 was reported, it might be because of an old data type.

I believe letting different data types use the same signature is a bad idea. I understand the desire to not break every unit using uavcan on every small update. However, I believe it will be impossible to make this work without making any assumptions that won't be programmatically enforced.

Versioning

A thing I've been thinking a bit about lately is the concept of versioning in dsdl. It could be supported as

@version <MAJOR>.<MINOR>.<PATCH>

Changing the patch version should be things that would not change the signature. Improving comments, adding constants, etc

Changing the minor version should be things that don't break binary compatibility. Adding a field in place of void, improving the name of field, etc

Changing the major version should represent a breaking change.

The new dsdl of the previous example could perhaps look like the following

@version 1.0.0    
uint2 status
void2 reserved1
void4 reserved2

@version 1.1.0
uint2 status

uint2 STATE_IDLE = 0
uint2 STATE_TRANSMITTING = 1
uint2 STATE_RECEIVING = 2
uint2 STATE_ERROR = 3
uint2 state

void4 reserved2

It would be possible to compile in 3 different ways.

• Strict: This would use the latest version and only accept the latest version
• Liberal: This would use the latest version but accept all versions with matching major version
• Compatible: This would use the lowest compatible (high major but low minor/patch), and accept newer versions by masking values.

Implementation

• Accepting several versions of a message could be implemented by running the crc with the few different options and accept it if one of them passes.
• When a new major version of the message type is made, it will no longer be required to list the older versions in the dsdl.

Stabilization

A message type should be considered stable when the major version reaches 1. Before that everything goes, even without storing the old formats. A message type without a version directive should be allowed and equivalent to @version 0.0.0 (regarded as unstable).

Something like this would be useful for stabilization as it makes it possible to stabilize one message type at the time. Practically speaking, the most used message like NodeStatus and LogMessage might be mature while some messages are still in the testing phase. Giving the promise that some central messages would not be touched without allowing some sort of backward compatibility is very useful. At the same time having the possibility to change experimental messages makes it even more useful.

Additions

• In liberal mode, if an old version matches with the signature. It might be desirable to define what the new fields should default to.

Disclaimers

I think the idea might be promising but haven't thought it completely through yet. For instance, it would perhaps be enough to store the signatures of the old messages instead of the whole message?

I also realize this is a big change at a somewhat late stage. I believe it should be possible to do without breaking anything (by accepting no version as version 0.0.0). And changing the dsdl compilers before the dsdl.

[pavel-kirienko]

The overall idea seems very useful indeed. I should give it a careful consideration, which takes me a long time, but sooner or later I should get back to this.

I agree that there should be a way to implement this in a backward-compatible way, not breaking the existing definitions; we won't be able to sacrifice backward compatibility even for such a useful change.

Using several compilation modes is likely not such a great idea as the rest of the proposition. Things should be automatic as much as possible, the protocol should do the right thing by default, otherwise, human errors are bound to be introduced, not to mention the steeper learning curve.

[pavel-kirienko]

@kjetilkjeka do you have any ideas how your proposition could be extended to support single-frame transfers, which have no CRC?

[kjetilkjeka]

The @version will still express compatibility. But just as before the suggestion, it's not enforced dynamically.

We do still get the benefits from a partial stabilization of the dsdl, expressing binary compatibility, and a tool for expressing breakage of data format when it's deemed necessary.

For instance, if we are forced to break a single frame (or any frame) in the future. it's easier to say "support <frame_name> 2.0" insted of "support post breakage frame variant"

I believe that this data type compatibility is better solved statically. Having a version number goes a long way. Specifying an esc firmware is compatible with esc message 1.0 to 1.4 and sends esc status message 1.2 should be the "first line of defense".

Data type signature should be the second line defense (only applicable for multi frame transfers)

A "fix" could be to implement a service that asks for which versions of a message that is supported. This could be provided as part of what I call "application level" functions in uavcan.rs (things such as time server, id server, etc). I'm not convinced this would be useful at all since it would be optional to support.

A very long term alternative might be to include CRC in CAN-FD single frames (for the CAN-FD extension) and wait until CAN-FD is the norm.

tl;dr I think it's very hard to dynamically enforce this in a backward compatible way

[kjetilkjeka]

The overall idea seems very useful indeed. I should give it a careful consideration, which takes me a long time, but sooner or later I should get back to this.

I agree that there should be a way to implement this in a backward-compatible way, not breaking the existing definitions; we won't be able to sacrifice backward compatibility even for such a useful change.

Even if you and I converge to something we think is sane. I believe that for a change of this magnitude, we should first specify every detail and then have a long review period (it's not worth rushing). And yes, if it should be done, it must be done without breaking anything. So take your time and some more.

Using several compilation modes is likely not such a great idea as the rest of the proposition. Things should be automatic as much as possible, the protocol should do the right thing by default, otherwise, human errors are bound to be introduced, not to mention the steeper learning curve.

You're absolutely right, I refer to my disclaimer (knew it was going to be of use).

[kjetilkjeka]

Issue #34 really shows where individual versioning of types would be useful.

• It talks about stabilization and version 1.0. The conflict between stabilizing and testing before release becomes a lot milder when only stabilizing the most critical parts is an option.
• It talks about leaving out definitions in the uavcan.equipment namespace. This would be solved by not upping these types to major version 1
• It contains a list of important things that need to be settled before stabilization. By allowing to stabilize one and one part of the list, usefulness is gained for every item that's checked off instead of when all the items are checked off.
• Perhaps most importantly stabilizing the uavcan.equipment namespace will be easier when using gained knowledge from stabilizing the uavcan.protocol namespace

[kjetilkjeka]

@pavel-kirienko maybe we should bring out the versioning thread as an issue of its own?

I suggest to mark it with [WIP] and have the head post in an RFC kind of style that we may edit as the suggestion improves. When we have something close to a complete solution we remove the [WIP] tag and leave it as something like "RFC: versioning of data types".

If you think this is a good idea, I can do the work one of the upcoming days?

(Having this is an issue on its own is of course not to rush it. I still fully agree that you take your time considering this suggestion. But getting more eyes on it might bring out weaknesses at an earlier stage?)

[pavel-kirienko]

I see the value perfectly and agree that it is quite useful.

maybe we should bring out the versioning thread as an issue of its own?

Sure.