Image panel: support custom message containing nested image

[chenjunnn]

I'd like to publish multiple images in one msg, but foxglove studio can only visualize the standard image msg type.

The custom msg may be like:

sensor_msgs/Image raw1
sensor_msgs/Image raw2
sensor_msgs/CompressedImage compressed1
sensor_msgs/CompressedImage compressed2

[defunctzombie]

@Chenjun00 Is there some limitation you have on-robot that prevents these from being sent as separate topics?

[jtbandes]

Theoretically there is no reason the Image panel shouldn't be able to pull out one field of a message as long as that field has the expected format/datatype.

[chenjunnn]

@Chenjun00 Is there some limitation you have on-robot that prevents these from being sent as separate topics?

I'm writing a node that using OpenCV to process image, so there are many debug images during the processing that I want to publish to visualize. If the images are sent as separate topics, I need to construct lots of publisher which I think is not convenient. But I have figured out a way that using a param to control which image to publish to a single topic. So if you think this feature is unnecessary, please close this issue.

And I'm looking forward to the feature https://github.com/foxglove/studio/issues/1662 to be supported.

[amacneil]

Seems like a reasonable feature to support.

@defunctzombie @jtbandes how difficult would it be to show message fields that match a supported schemaName (in addition to topics that match a support schemaName) in the image selector dropdown?

[defunctzombie]

Somewhat challenging - tho not impossible. We'd need to figure out if we provide the name through our parsed RosDatatypes structure.

I'm just not convinced the UI complexity is worth it tho.

[jtbandes]

What UI complexity? Wouldn't we just replace the dropdown in the Image panel with a MessagePathInput?

[defunctzombie]

@jtbandes Adrian actually proposed keeping a dropdown and in the dropdown showing all the detected renderable topics and fields.

An alternative could be to use MessagePathInput and pre-filter its items to the topics and message paths that are renderable. That could also work well.

[amacneil]

Yeah, I think just predetermine all the message paths, and add them to the existing dropdown. That way for existing users without nested images, there will be no UI change.

[gayar-helm]

This could be a very useful feature. Our use case is that the image is published together with some meta information. e.g:

my_image_msg_type
{
meta_msg msg
sensor_msgs/Image image
}

Would it be possible to specify something like "my_msg.image" at the top of image panel? Just like described here - #203

[defunctzombie]

Would it be possible to specify something like "my_msg.image" at the top of image panel? Just like described here

Yep. Need to think through how we accomplish this in the implementation but I think the UX described above with a dropdown that supports selecting any topic or field that contains a matching datatype seems sensible.