Marker textures

[gbalke]

This is a pre-cursor to a corresponding update to the RViz triangle marker message system. It allows for a texture to be applied to triangles using UV coordinates.

The functionality has been demonstrated here:
https://github.com/gbalke/rviz_shader_tester

[clalancette]

Just so you know, this is quite a large change. In particular, putting this in will basically invalidate all rosbags that were taken with previous ROS distributions, and it ensures that Humble will be wire-incompatible with Galactic (cross-distribution compatibility isn't something we guarantee, but it is nice to keep when we can).

All of that said, we do allow for this kind of breakage in the core if the change has broad enough applicability. Can you describe more about UV coordinates, and why we might want them here?

[gbalke]

Sure thing! UV Coordinates are used to describe the relationship of a point in a geometric feature to a corresponding image in a texture file. I think wikipedia describes this well. As it pertains to robotics, if someone wishes to visualize dynamic geometries with an applied texture, such as a collision mesh with heat map, they can easily do so by specifying these values and having their rendering backend (Rviz -> Ogre) handle applying this texture. UV Coordinates are well established within graphics and this exposes that feature in a direct way to be used with the ros marker message system.

I understand the hope to maintain backwards compatibility. I do know that there are message formats out there, such as protobuf, that have managed to maintain forwards and backwards compatibility and allow for the addition of new features. I am curious if this is something that can be introduced to ros messages so new features can be added without causing damage/major issues.

[tfoote]

Overall I think that this is a good extension of the message. However we need to make sure that the message remains self contained.

I know that often textures are maintained in parallel with meshes and are reused. But keeping things self contained is super valuable in this context. Alternatively we should be setting up a parallel stream possibly. But that's getting complicated again.

[jacobperron]

FWIW, it might be worth taking a look at mesh_tools for inspiration. They appear to have message definitions for textures and UV coordinates.

[gbalke]

FWIW, it might be worth taking a look at mesh_tools for inspiration. They appear to have message definitions for textures and UV coordinates.

I'm happy to go this route if the uri is going to be replaced by an alternate message stream. I think UUID implies a specific implementation which we don't necessarily want. A unique name will work. I think texture_id with a comment describing this should do the trick?

[tfoote]

Great find @jacobperron We should definitely leverage the work that mesh_tools has already demonstrated. As they've shown we can encapsulate the texture relatively easily in a message. Then everything will be self contained and there's no issues with synchronization, access, or id mapping.

[gbalke]

After chatting with @tfoote, we came to the decision to include the texture in the message. This was done to provide an ease-of-access to apply textures. While sub-optimal for certain use-cases, this will be recordable and be quick to bring up. For more optimized solutions, more complicated packages may be used such as mesh_utils.

[wjwwood]

I'm concerned with the approach selected. It deviates from the already existing MESH_RESOURCE marker type and it is not future proof, in the case that you want to use another method of specifying the texture.

I would suggest these fields instead:

# If texture_uri either empty string ("") or something like "texture_embedded://",
# then the texture can be read from texture_image, otherwise it is loaded via
# resource_retriever, just like mesh_resource.
string texture_resource
sensor_msgs/CompressedImage texture_image
UVCoordinate[] uv_coordinates

With code like:

if (
  marker_msg.texture_resource.empty() ||
  marker_msg.texture_resource == "texture_embedded://")
{
  load_texture_from_compressed_image(marker_msg.texture_image);
} else {
  // load_texture_from_resource_retriever(marker_msg.texture_resource);
  throw std::runtime_error("not implemented");  // or some similar error condition
}

That way if/when you want to support loading via package:// or file://, you don't have to change the message definition again.

[gbalke]

The latest changes match @wjwwood's suggestions. I've also added a mesh field at the request of @tfoote, @clalancette, @wjwwood to give meshes the same ability to be recorded in rosbag if one chooses to do so. Implementation will not immediately be available but this is a preemptive change to minimize number of commits modifying this message.

[tfoote]

This is looking good from our discussion. Structurally I think it's basically there. I mostly want to improve the documentation to make sure that we and everyone following are on the same page.

[tfoote]

We should add to the documentation references to resource retriever and use it like we already do for the meshes. (This lets us use package:// syntax too)

https://index.ros.org/p/resource_retriever/

Then for the embedded approach I might suggest embedded://name or attached://name and then we can mirror it and used the same keyword for the mesh.

[wjwwood]

I second @tfoote's comments and also I think generally we could use more detailed docs, because I find myself asking things like:

• what URI are acceptable
• What is the "texture_name" used for, supposed to be?

And generally the Mesh docs could be updated to match.

[wjwwood]

One doc fix up needed, otherwise lgtm

[gbalke]

Complete CI via --packages-above-and-dependencies:

• Linux
• Linux-aarch64
• macOS
• Windows

[gbalke]

CI after merging (IDL changes made last build unstable):

• Linux
• Linux-aarch64
• macOS
• Windows