clarification of links and inline content

[tomkralidis]

Should we consider the following

• data SHALL be either encoded in a link object (with rel=canonical) OR inline in a content object via value

Rationale:

• in some instances, the data notification message may actually be larger than the data payload
• in some cases, it may take the broker/cache logic longer to download and put the data to an object in comparison to the actual message dissemination (related to real-time data)

We should also consider which user in a given publish sequence this could apply to (i.e. data provider, global services).

@david-i-berry feel free to update based on our discussion.

cc @golfvert

[golfvert]

I think the link is a MUST in all cases and inline content is a MAY for data below a certain size limit (eg 2kB). For consistency purposes, the link is a must for me. However, it is (probably) a matter of taste… more than anything else. All tastes are equally valid! (Tous les goûts sont dans la nature)

[kaiwirt]

We should not only take the size into account but also the type of the data. Warnings should be inline in my opinion.

[kaiwirt]

• might also be useful for announcements or informational messages like out-of-service announcements

[tomkralidis]

Possible consideration:

• for messages below 2048 bytes:
  • notifications SHALL provide inline (via properties.content)
  • notifications MAY additional provide via link object
• for messages above 2048 bytes:
  • notifications SHALL provide via link object
  • notificaitions SHALL not provide inline

[golfvert]

Just to confirm what was discussed... In fact, it should be:

for messages below 2048 bytes:

- notifications MAY provide inline (via properties.content)

- notifications SHALL additional provide via link object

for messages above 2048 bytes:

- notifications SHALL provide via link object

- notificaitions SHALL not provide inline

And the 2048 bytes limit is to be discussed!

[josusky]

In TT-Protocols we have converged on 4kB limit but the exact value can be re-considered after the pilot phase. Perhaps we could even allow a higher limit for high-priority things that are sent rarely, e.g. warnings. For sure the brokers shall be configured to handle notifications that are by an order of magnitude (or two) bigger - for resilience.
And we shall not forget that we have the possibility to gzip the embedded content. CAP and other XML-based messages are easily packed to 10% of their original size.

[amilan17]

@josusky @tomkralidis is this still an open issue?

[tomkralidis]

We still need clear guidance; updated proposal:

Requirements:

• notifications SHALL always provide a canonical link object (this is already in the specification, so no need to add per se)
• for data whose resulting size (compressed or uncompressed) is greater than 4096 bytes, notifications SHALL NOT provide inline via properties.content.value

Permissions:

• for data whose resulting size (compressed or uncompressed) is less than 4096 bytes, notifications MAY provide inline via properties.content.value

Thoughts @josusky @ktsuboi-jma @davidpodeur @McDonald-Ian @antje-s

[josusky]

In general, I agree.
I suppose that by

(compressed or uncompressed)

you mean that whatever is the intended content encoding, the inlined/embedded data shall not be greater. Perhaps we could clarify it in a separate sentence, like this:
"for data whose resulting size is greater than 4096 bytes, notifications SHALL NOT embed the data via properties.content.value. The limit takes into account the used data encoding, that is, if the data is compressed with gzip (properties.content.encoding has the value gzip), the compressed size matters.

And as stated earlier, we should consider an exception for specific types of data such as alerts/warnings (e.g. CAP).

[tomkralidis]

TT-WISMD 2023-05-22:

• add @josusky comments to normative text/schema
• need to review as part of pilot phase:
  • following pilot phase on 4096 byte
  • perhaps allow greater limits based on data types/frequency

[tomkralidis]

PR in #51

[antje-s]

Close issue? PR #51 is already merged