Uniqueness

[sammysmallman]

Schemas at the moment are inferred to be the same/valid for all devices by the absence of a device_model_id, software_version_id and root boolean value.

If that is the case, these should be made optional in the schema and stated in the standard as to their meaning when absent.

[sammysmallman]

Referencing #12, further thought on the subject of uniqueness has led me down the path that a schema should either be "scoped" as a manufacturer wide parameter (all devices supporting the parameter can use this schema) or specific to a manufacturer device model version (only a specific device model running a certain software version can use this schema).
In my mind there are two main reasons for the need for this "scope":

1. The manufacturer id allows for schemas with the same parameter id to sit in the same namespace as each other. e.g. ETC and HES may have a parameter schema each for a parameter that has the same id.
2. "labels" within the schema are different per device model version. e.g.

Device 1:

  "get_response": [
    {
      "name": "dmx_loss_behavior",
      "type": "uint8",
      "labels": [
        { "name": "dmx_loss_behavior_instant", "value": 0 },
        { "name": "dmx_loss_behavior_wait2min", "value": 1 },
        { "name": "dmx_loss_behavior_hll", "value": 2 }
      ]
    }
  ],

Device 2:

  "get_response": [
    {
      "name": "dmx_loss_behavior",
      "type": "uint8",
      "labels": [
        { "name": "dmx_loss_behavior_hll", "value": 0 }
        { "name": "dmx_loss_behavior_wait5min", "value": 1 },
        { "name": "dmx_loss_behavior_wait10min", "value": 2 },
        { "name": "dmx_loss_behavior_instant", "value": 3 },
      ]
    }
  ],

These "labels" are important as they remove the need to do the typical get request for parameter descriptions. In theory if we have a schema, it contains labels, and we know that the parameter the schema is for is "informedBy" (#14) another parameter, we can omit that informing parameter from the get requests after receiving supported parameters.

The schema used by Device 1 could be used by a large majority of the devices made by a manufacturer as such its scope could be deemed "manufacturer wide" and the schema would look something like this:

{
  "scope": { 
    "esta_manufacturer_id": 12345
  },
  "name": "DMX_LOSS_BEHAVIOR",
  "pid": 33034,
  "version": 1,
  "get_request_subdevice_range": [ "root", "subdevices" ],
  "get_request": [],
  "get_response": [
    {
      "name": "dmx_loss_behavior",
      "type": "uint8",
      "labels": [
        { "name": "dmx_loss_behavior_instant", "value": 0 },
        { "name": "dmx_loss_behavior_wait2min", "value": 1 },
        { "name": "dmx_loss_behavior_hll", "value": 2 }
      ]
    }
  ],
  "set_request_subdevice_range": [ "root", "subdevices", "broadcast" ],
  "set_request": "get_response",
  "set_response": []
}

The schema for Device 2 though is a specific to that particular device and its version and would look something like this:

{
  "scope": { 
    "esta_manufacturer_id": 12345,
    "device_model_id": 1337,
    "software_version_id": 12345678,
    "root": true
  },
  "name": "DMX_LOSS_BEHAVIOR",
  "pid": 33034,
  "version": 1,
  "get_request_subdevice_range": [ "root", "subdevices" ],
  "get_request": [],
  "get_response": [
    {
      "name": "dmx_loss_behavior",
      "type": "uint8",
      "labels": [
        { "name": "dmx_loss_behavior_instant", "value": 0 },
        { "name": "dmx_loss_behavior_wait2min", "value": 1 },
        { "name": "dmx_loss_behavior_hll", "value": 2 }
      ]
    }
  ],
  "set_request_subdevice_range": [ "root", "subdevices", "broadcast" ],
  "set_request": "get_response",
  "set_response": []
}

I wondered about different levels of specificity, for instance omitting the software_version_id would allow the schema to be valid for all device models made by the manufacturer... That might add a bit too much complexity to it though.

[mayanigrosh]

We've discussed this in today's E1.37-5 meeting and agreed that we'll add "device_model_id" and "software_version_id" as keys. The written text of the standard will also provide guidance to the implementor as to the best PIDs to query over RDM in order to get a complete picture to compare these to.

[ssilverman]

@mayanigrosh @sammysmallman See PR #20.

[ssilverman]

I can think of another reason not to include this tuple (manufacturer_id, device_model_id, software_version_id). It has to do with putting too much information inside a message description (i.e. instance of the schema). The information exists externally, and the device that retrieves the information already has access to the tuple data. Why does it also need to be in the message? I mean, if the message is stored somewhere for later retrieval then it will be self-contained if it does contain the tuple, however, the system that stores it could also add this information into its own bucket labeled with that tuple, thus obviating the need for it in each message.

In short: I don't yet necessarily agree with this change.

[ssilverman]

Here's another reason why I don't think this change should go into the schema (adding the 3-item tuple). The schema should be also applicable to the ESTA-defined messages (i.e. each PID) if it's truly general and correct. ESTA doesn't have a Device ID nor a Software Version ID. This means there's another use-case for not having them.

[mayanigrosh]

The Parameter Metadata Language only describes Manufacturer-Specific Parameter Messages. Transmitting ESTA-Defined messages is expressly prohibited. The definition of those messages exists in the standards that describe them and shall not change outside of the ANSI process.
So there is no ESTA use-case for these messages.

[ssilverman]

The schema should be applicable to all messages, ESTA or not. I'm not necessarily taking about transmission. The schema is for definition, not transmission.

[ssilverman]

This is still an outstanding question: Why do the messages need to contain the version tuple as opposed to the system that retrieves the messages storing the messages tagged with the version tuple?

[sammysmallman]

This is still an outstanding question: Why do the messages need to contain the version tuple as opposed to the system that retrieves the messages storing the messages tagged with the version tuple?

One reason could be that parameter definitions may be moved around as files and downloaded from the web. As such it's beneficial that they contain identifiers within themselves.

[sammysmallman]

The schema should be applicable to all messages, ESTA or not. I'm not necessarily taking about transmission. The schema is for definition, not transmission.

I am of the same opinion. I'm not sure I understand why the ESTA PIDs couldn't transmit their definitions but still as definitions I can bundle with the RDM Controller and never transmit they are still very useful.

[ssilverman]

an outstanding question: Why do the messages need to contain the version tuple as opposed to the system that retrieves the messages storing the messages tagged with the version tuple?

One reason could be that parameter definitions may be moved around as files and downloaded from the web. As such it's beneficial that they contain identifiers within themselves.

I don't disagree that it makes things easier. Self-identified data isn't a bad thing. However, I'm of the mind that we shouldn't need to manage how others store the data. It's convenient, yes, but required, no.

[peternewman]

I mean, if the message is stored somewhere for later retrieval then it will be self-contained if it does contain the tuple, however, the system that stores it could also add this information into its own bucket labeled with that tuple, thus obviating the need for it in each message.

This is still an outstanding question: Why do the messages need to contain the version tuple as opposed to the system that retrieves the messages storing the messages tagged with the version tuple?

One reason could be that parameter definitions may be moved around as files and downloaded from the web. As such it's beneficial that they contain identifiers within themselves.

As @sammysmallman said, I'm sure there will be some manufacturers who may want manufacturer specific PIDs and can't afford the storage on their device for some relatively verbose JSON (compared to a binary blob) so could offer their JSON schemas for download from their website. The storage system can only deal with bucket labelling from data sourced this way if that information comes in a standardised way such as in the schema. I agree with others it should be optional parameters though for the ESTA reason.

The Parameter Metadata Language only describes Manufacturer-Specific Parameter Messages. Transmitting ESTA-Defined messages is expressly prohibited. The definition of those messages exists in the standards that describe them and shall not change outside of the ANSI process. So there is no ESTA use-case for these messages.

@mayanigrosh I'd strongly disagree with there being no ESTA use-case, I can think of a few for starters, although I can probably see the argument for not transmitting them from individual fixtures. Or more accurately I can see the logic of banning fixtures from doing it badly or with incorrect/old info.

Firstly there are currently no computer parseable definitions of the ESTA PIDs, people have to manually and laboriously type them in from the standards (or generate header files for enums). Unfortunately that inevitably means typos, mistakes and occasionally misunderstandings in the intention. With no disrespect to Shawn (I'm sure OLA did the same thing over the years but its buried in many other issues), the closed issues for this repo nicely represent that fact:
https://github.com/ssilverman/rdm-schema/issues?q=is%3Aissue+is%3Aclosed

I'm assuming the ANSI process would allow additional JSON files along with the PDFs which could be published officially by ESTA? Or is it only possible to have unofficial attempts like Scott's header file?

@sammysmallman agrees they're useful for the same reason too:

I am of the same opinion. I'm not sure I understand why the ESTA PIDs couldn't transmit their definitions but still as definitions I can bundle with the RDM Controller and never transmit they are still very useful.

Secondly, and I actually think more importantly, as I've said a few times before, if your UI can behave the same/handle them in the same way for standardised ESTA PIDs and manufacturer specific ones, you end up treating manufacturer specific ones the same almost as a side-effect (someone has already designed a schema, why not just use it) and they become first class citizens with the same usability improvements, which can only be a good thing for their uptake and end users.

You can see this quite clearly with OLA, our command line tools primarily use our PID definitions (equivalent to what this schema would provide if ESTA PIDs are kept in scope too), there's a tiny amount of specialised pretty-printing for certain PIDs, but generally it all just works normally, they support all PIDs anyone has written a definition for, things like labels and range validation work the same for a standardised PID as a manufacturer specific one and when a new PID appears (of either type), it generally just needs a definition writing (assuming there aren't new data types. In comparison, our current web UI only supports a small handful of the standardised PIDs and I don't think any manufacturer specific ones because it uses custom template functions, rather than just processing our existing definitions. It's been a plan to move it to a more dynamic UI for a while, but it's waiting on something being standardised in E1.37-5 rather than rolling our own.

If something like #16 was implemented too, then most of the remaining specialisation could be made generic as part of the definitions too, benefiting manufacturer specific ones as well as standardised ones!

[sammysmallman]

if your UI can behave the same/handle them in the same way for standardised ESTA PIDs and manufacturer specific ones, you end up treating manufacturer specific ones the same almost as a side-effect

This is exactly how I have a RDM Controller using the general PID definitions now. It allows your controller to adapt and scale much more easily with a growing list of PIDs.

I'm not necessarily opposed to omitting the manufacturer_id, device_model_id and software_version_id from the schemas. But by doing so and using the response data and cached DEVICE_INFO data (for the model and software id) as @ssilverman is suggesting we are inferring those properties. Thus we are saying a parameter schema is unique to a very specific software version, model and manufacturer which is counter to how many manufacturers design manufacturer specific pids. From my research most manufacturer specific pids are unique only be the manufacturer id and can be used by all manufactured models and software versions.

I am still convinced we need to be including a root boolean value indicating whether the schema applies to the root or its sub devices. Or again if we're omitting it then it is again inferred from the response data.

I think the lack of these values within the schema would reduce the clarity of them. We provide the pid and name for clarity when it is technically unnecessary. Also as mentioned above we're suggesting that they are very specific to model and software so manufacturers are going to have to store these somewhere... it would be easy to accidentally put the json file for one device in the wrong folder and finding that mistake would be cumbersome...

Stating the obvious here but by making the parameter definitions specific to a certain model and software increases RDM traffic... That said by attempting to reduce that by scoping the parameter definitions to be more open and used by all models made by a manufacturer I can't think of a decent mechanism to allow for that specificity for one particular device.

[sammysmallman]

Here's another reason why I don't think this change should go into the schema (adding the 3-item tuple). The schema should be also applicable to the ESTA-defined messages (i.e. each PID) if it's truly general and correct. ESTA doesn't have a Device ID nor a Software Version ID. This means there's another use-case for not having them.

This relates to #17