Add type to model nodes

[frauzufall]

It would be great to specify a type for each model node (outputs for sure, maybe also inputs) - it could be something like image, table, scalar.

[constantinpape]

Would you just add this to give a hint about the type of the tensor, or would you actually use this to check for the correctness of the type, e.g. when building some application and only allowing models that output table?

Also, would you link the dimension of the tensor and the type?
I.e. table must have 2 dimensions (3 including the batch dimension)`, etc.

We could add type as an (optional?) key to the tensor specification.

[esgomezm]

Adding a type would make things much easier!

At the moment, in HPA ShuffleNetV2, the axes in the output are specified as null, which can be used to discriminate between images and scalars/tables/vectors.

What about those networks which have tables and images as outputs, as MaskRCNN?

[frauzufall]

This is still unresolved, right? I agree with @esgomezm, specifying the type would avoid guessing games on the consumer side.

[constantinpape]

I wouldn't mind having type as an optional field in the tensor specification.

[oeway]

I think we should have a type hint key, however, the name type can cause confusion when used together with data_type (float32 etc.), we need to come up with another name. Another way to think about it is to define how the input/output tensor should be rendered in the UI.

For the input tensors UI types:

• image, I think we should perhaps include axes in here because it only make sense for images
• number, will be rendered as a input box, here we can define min and max value
• choice, will be rendered as a dropdown menu with a list of labels, the labels will then be encoded as 1-hot encoding and converted into a numpy array ( see the ANNA-PALM plugin for ImageJ here, we have a mode with microtubule and nuclear pore as options)

For the output tensors UI types:

• image, it should contain optional keys, to hint the LUTtable, the axes will only make sense when the it is an image
• table, similar to axes for images, we should be able to define headers for each column
• class_label, here we should be able to define the class labels

How about name the key as ui, and the value is a dictionary? For example:

  - name: input-1
    data_type: float32
    data_range: [-inf, inf]
    meta:
       type: image
       axes: bcyx
       label: Widefield Image
  - name: input-2
    data_type: float32
    data_range: [-inf, inf]
    meta:
       type: choice
       label: Mode
       options:
           - microtubule
           - nuclear pore
           - action filament
       encoding: one-hot

[constantinpape]

@oeway I agree that type is a bit to ambiguous. Maybe tensor_type.

I am not so sure about the ui field. I think this is the responsibility of the consumer and we shouldn't start to encode this in the yaml, because we will need to enumerate a lot of different options.

[oeway]

I am not so sure about the ui field. I think this is the responsibility of the consumer and we shouldn't start to encode this in the yaml, because we will need to enumerate a lot of different options.

How to render is the responsibility of the consumer, these are just some additional information that are required to render them on UI.

When you work with models that take image as input and output an image, you don't need to care much about the UI, however, for a classification model for example, there is no way to figure it out the class labels from the consumer side without additional hint. The information has come from the yaml.

The ui key might not be the best choice, but, a simple string won't be enough. Also, tensor_type feels very much similar to dtype (example).

We should discuss this tomorrow.