Tagged components

[emilk]

Consider the Color component as applied to a Mesh3D archetype. Does the Color specify the color of faces, edges, or vertices?

To support this, we could introduced tagged components, allowing each entity to have multiple instances of the same component, if they have different tags.

Each (Tag, ComponentName) tuple would be its own column of data.

Workarounds

For now we can introduce multiple components instead: SolidColor, EdgeColor, VertexColor, etc

References

https://www.flecs.dev/flecs/md_docs_2Relationships.html

[Wumpf]

An instance of tagged component @jleibs and me discussed recently is fill out and wireframe color in a situation where both solid & fill are supported at the same time (another outcome of the discussion was that we don't want to support it right now, more on that later & elsewhere):
Let's say we have a visualizer that supports doing both at the same time:

• if only one of wireframe & fill is enabled: query any color and use that
• if both are enabled: check for specific color tags and act accordingly
  alternatively the tags themselves can also influence the default of what's enabled!

[kpreid]

This seems like a potentially lot of added fundamental complexity in implementation and possibly API. I wonder if there's a way to get the same effect, simpler, in addition to wondering exactly what the semantics of this tag system is going to be.

• What if component types as currently known were removed entirely, and Rerun used only pairs of (Tag, Data), that is, components would be like (SolidColor, Rgba) and (Radius, f32)? That's still a unique static type per component.

• Another, probably more Rust-only frame but this is a brainstorming sort of remark: what if these were just parameterized components? That is, components::Color<Solid> vs components::Color<Edge>. That way, you avoid code duplication, and don't need to extend the existing architecture of component types (but probably need fancier code generation).

[teh-cmc]

We've had a long discussion about leaf (nee "out-of-tree") transforms a couple days back, during which the desire for tagged components came up a lot. Again. We keep running into data modeling deadlocks.

In parallel, @gavrelina has been experimenting with the Dataframe View and has hit more of the same issues.

We keep running into data modeling deadlocks, again and again.

---

My guess is that tagged components will greatly influence the design of our data model and query language, and to an extent even the UX of the viewer itself. As such I think it's really important that we get there sooner rather than later.
It doesn't have to be perfect either: the sooner we get to play with these new tools and refine our data model with them in mind, the sooner we can start stabilizing and ultimately (gasp) standardizing things.

It would also be quite nice to be completely done with major ABI breaking changes ASAP, before we enter the ✨ disk-based era ✨.

Thus, here's a quick proposal to move us towards that goal.

---

Context

Today, the atomic unit of data in Rerun is a Chunk column.

A Chunk column is fully qualified by two things: a Rerun ComponentName and an Arrow Datatype (NOTE: I'm intentionally omitting the 3rd qualifier, the "Rerun datatype name", which is a vague internal concept that is not actually materialized most of the time. It doesn't matter for this discussion.):

• The component name (Position3D, Color, Radius, …) specifies the semantic of the data, "how things behave".
• The datatype ([3]Float32, Uint32, …) specifies the memory layout of the data.

The two are for the most part completely orthogonal to one another.

This information is all stored within the column metadata, and denormalized into the store indices as needed.

At runtime, a Rerun system will look for a piece of data by searching for the semantic its interested in, and then interpreting the returned data based on its datatype:

let data = store.latest_at("my_entity", TimeInt::MAX, "rerun.components.Position3D")?; // untyped (!)
let data = data.try_downcast::<&[[f32; 3]]>()?; // typed

All of this works pretty nicely, except for one major limitation: you cannot re-use the same semantics twice (or more) on a single entity.

Example: imagine an entity "detections" that holds both 3D points and 3D boxes at the same time: they will have to share a batch of "rerun.components.Position3D" (for Points3D::positions and Boxes3D::centers).
Sometimes this is exactly what you want, and things are great. Just enable both the Points3D and Boxes3D visualizers and you're good to go.
Sometimes it is not what you want, and you're doomed.

This problem infects everything and leads to all kinds of nasty data modeling deadlocks all over the place.

Proposal

The core idea of this proposal is trivial: to replace the very limited Rerun ComponentName with a much more flexible ComponentDescriptor.
Of course, this has a ton of ripple effects. I'll try to cover the most important ones, but there's too many to cover them all.

We should be able to get there in small increments that can be merged as they come, with complete feature parity and no visible impact on end-users whatsoever.

Once we're there, we'll be able to start experimenting with all kinds of crazy ideas.

Data model changes

A Chunk column would still be fully qualified by two bits of information: a Rerun ComponentDescriptor (semantics) and an Arrow Datatype (memory layout).
All that metadata is stored in the same place as before (column metadata and/or arrow schema).

A ComponentDescriptor would look like the following:

/// A [`ComponentDescriptor`] fully describes the semantics of a column of data.
pub struct ComponentDescriptor {
    /// Optional name of the `Archetype` associated with this data.
    ///
    /// `None` if the data wasn't logged through an archetype.
    ///
    /// Example: `rerun.archetypes.Points3D`.
    archetype_name: Option<ArchetypeName>,

    /// Semantic name associated with this data.
    ///
    /// Example: `rerun.components.Position3D`.
    component_name: ComponentName,

    /// Optional label to further qualify the data.
    ///
    /// Example: "postions".
    //
    // TODO: Maybe it's a dedicated type or an `InternedString` or w/e, doesn't matter.
    tag: Option<String>,
}

// NOTE: Take a careful look at this implementation, so you know what I mean later in this doc.
//
// Examples:
// * `rerun.archetypes.Points3D::rerun.components.Position3D#positions`
// * `rerun.components.Translation3D#translation`
// * `third_party.barometric_pressure`
impl std::fmt::Display for ComponentDescriptor {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        let ComponentDescriptor {
            archetype_name,
            component_name,
            tag,
        } = self;
        match (archetype_name, component_name, tag) {
            (None, component_name, None) => f.write_str(component_name),
            (Some(archetype_name), component_name, None) => {
                f.write_fmt(format_args!("{archetype_name}::{component_name}"))
            }
            (None, component_name, Some(tag)) => {
                f.write_fmt(format_args!("{component_name}#{tag}"))
            }
            (Some(archetype_name), component_name, Some(tag)) => {
                f.write_fmt(format_args!("{archetype_name}::{component_name}#{tag}"))
            }
        }
    }
}

This information could be trivially code generated already today, and is a strict superset of the status quo.

This would already be enough to get rid of our old indicator components.

IDL changes

I'm going to use Points3D and Mesh3D for demonstration purposes. They showcase our data modeling issues well, always semantically-walking on each others' toes.

The major change at the IDL level is that the entire rerun{.blueprint}.components layer just goes away.
(Well, in reality that layer is very useful to codegen nice user-friendly component-level APIs, so probably not. But you'll get the idea, either way.)

(NOTE: I've omitted the usual attributes in all the IDL samples below. They haven't changed in any way.)

I.e. Points3D turns from this:

table Points3D {
  positions: [rerun.components.Position3D] (/* … */);

  radii: [rerun.components.Radius] (/* … */);
  colors: [rerun.components.Color] (/* … */);

  labels: [rerun.components.Text] (/* … */);
  class_ids: [rerun.components.ClassId] (/* … */);
  keypoint_ids: [rerun.components.KeypointId] (/* … */);
}

into this (the generated ComponentDescriptors are shown as comments):

table Points3D {
  // ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Position3D#positions"
  positions: [rerun.datatypes.Vec3D] ("attr.rerun.component": "rerun.components.Position3D", /* … */);

  // ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Radius#radii"
  radii: [rerun.datatypes.Float32] ("attr.rerun.component": "rerun.components.Radius", /* … */);
  // ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Color#colors"
  colors: [rerun.datatypes.UInt32] ("attr.rerun.component": "rerun.components.Color", /* … */);

  // ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Label#labels"
  labels: [rerun.datatypes.Utf8] ("attr.rerun.component": "rerun.components.Label", /* … */);
  // ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.ClassId#class_ids"
  class_ids: [rerun.datatypes.UInt16] ("attr.rerun.component": "rerun.components.ClassId", /* … */);
  // ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.KeypointId#keypoint_ids"
  keypoint_ids: [rerun.datatypes.UInt16] ("attr.rerun.component": "rerun.components.KeypointId", /* … */);
}

Mesh3D turns from this:

table Mesh3D {
  vertex_positions: [rerun.components.Position3D] (/* … */);

  triangle_indices: [rerun.components.TriangleIndices] (/* … */);
  vertex_normals: [rerun.components.Vector3D] (/* … */);

  vertex_colors: [rerun.components.Color] (/* … */);
  vertex_texcoords: [rerun.components.Texcoord2D] (/* … */);
  albedo_factor: [rerun.components.AlbedoFactor] (/* … */);
  class_ids: [rerun.components.ClassId] (/* … */);
}

into this (the generated ComponentDescriptors are shown as comments):

table Mesh3D {
  // ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Position3D#vertex_positions"
  vertex_positions: [rerun.datatypes.Vec3D] ("attr.rerun.component": "rerun.components.Position3D", /* … */);

  // ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.TriangleIndices#triangle_indices"
  triangle_indices: [rerun.datatypes.UVec3D] ("attr.rerun.component": "rerun.components.TriangleIndices", /* … */);
  // ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Vector3D#vertex_normals"
  vertex_normals: [rerun.datatypes.Vec3D] ("attr.rerun.component": "rerun.components.Vector3D", /* … */);

  // ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Color#vertex_colors"
  vertex_colors: [rerun.datatypes.UInt32] ("attr.rerun.component": "rerun.components.Color", /* … */);
  // ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.TexCoords2D#vertex_texcoords"
  vertex_texcoords: [rerun.datatypes.Vec2D] ("attr.rerun.component": "rerun.components.TexCoords2D", /* … */);
  // ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Color#albedo_factor"
  albedo_factor: [rerun.datatypes.UInt32] ("attr.rerun.component": "rerun.components.AlbedoFactor", /* … */);
  // ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.ClassId#class_ids"
  class_ids: [rerun.datatypes.UInt16] ("attr.rerun.component": "rerun.components.ClassId", /* … */);

}

Logging changes

Logging archetypes will yield fully-specified ComponentDescriptors:

rr.log(
    "points_and_mesh",
    rr.Points3D(
        # ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Position3D#positions"
        [[0, 0, 0], [1, 1, 1]],
        # ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Radius#radii"
        radii=10,
        # ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Color#colors"
        colors=[1, 1, 1],
        # ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.Label#labels"
        labels="some_label",
        # ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.ClassId#class_ids"
        class_ids=42,
        # ComponentDescriptor: "rerun.archetypes.Points3D::rerun.components.KeypointId#keypoint_ids"
        keypoint_ids=666,
    ),
)

rr.log(
    "points_and_mesh",
    rr.Mesh3D(
        # ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Position3D#vertex_positions"
        vertex_positions=[[0.0, 1.0, 0.0], [1.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
        # ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Vector3D#vertex_normals"
        vertex_normals=[0.0, 0.0, 1.0],
        # ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Color#vertex_colors"
        vertex_colors=[[0, 0, 255], [0, 255, 0], [255, 0, 0]],
        # ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.TriangleIndices#triangle_indices"
        triangle_indices=[2, 1, 0],
        # ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.Color#albedo_factor"
        albedo_factor=[32, 32, 32],
        # ComponentDescriptor: "rerun.archetypes.Mesh3D::rerun.components.ClassId#class_ids"
        class_ids=420,
    ),
)

Logging components directly omits the archetype part of the descriptor:

rr.log(
    "points_and_mesh",
    rr.components.Translation3D(
        # ComponentDescriptor: "rerun.components.Translation3D#translation"
        translation=[1, 2, 3],
    ),
)

A third-party ad-hoc component might not even have a tag at all..:

rr.log(
    "points_and_mesh",
    # ComponentDescriptor: "third_party.size"
    rr.AnyValues({"third_party.size": 42}),
)

..although we could expose ways of setting one:

rr.log(
    "points_and_mesh",
    # ComponentDescriptor: "third_party.size#some_tag"
    rr.AnyValues({"third_party.size": 42}, "tag": "some_tag"),
)

Store changes

Columns are now uniquely identified by a (ComponentDescriptor, ArrowDatatype) pair (as opposed to (ComponentName, ArrowDatatype) today).

This means we never overwrite data from an archetype with data from another one. We store everything, we can do whatever we want.

The batcher and other compaction systems will never merge two columns with different descriptors.

Indexing-wise, the store will add an extra layer of indices for tags (ComponentDescriptor::tag).
That is trivial to implement and pretty cheap both compute and space wise.

Query changes

Queries don't look for a ComponentName anymore, they look for fully or partially filled ComponentDescriptors instead.

E.g. to look for all columns with position semantics:

• You used to do this:

  latest_at("my_entity", TimeInt::MAX, "rerun.components.Position3D")

• You would now do this instead:

  // LatestAt(TimeInt::MAX) @ "my_entity" for (*, "rerun.components.Position3D", *)
  latest_at("my_entity", TimeInt::MAX, ComponentDescriptorPattern {
      archetype_name: None, // == any
      component_name: Some("rerun.components.Position3D"),
      tag: None, // == any
  })

Here's a few example queries using the Points3D and Mesh3D data we've logged earlier:

LatestAt(TimeInt::MAX) @ "points_and_mesh" for (*, *, *):
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.Position3D", "positions" }
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.Radius", "radii" }
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.Color", "colors" }
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.Label", "labels" }
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.ClassId", "class_ids" }
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.KeypointId", "keypoint_ids" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Position3D", "vertex_positions" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Vector3D", "vertex_normals" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Color", "vertex_colors" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.TriangleIndices", "triangle_indices" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Color", "albedo_factor" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.ClassId", "class_ids" }


LatestAt(TimeInt::MAX) @ "points_and_mesh" for (*, "rerun.components.Position3D", *):
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.Position3D", "positions" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Position3D", "vertex_positions"  }


LatestAt(TimeInt::MAX) @ "points_and_mesh" for (*, "rerun.components.Color", *):
- ComponentDescriptor { "rerun.archetypes.Points3D", "rerun.components.Color", "colors" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Color", "vertex_colors" }
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Color", "albedo_factor" }


LatestAt(TimeInt::MAX) @ "points_and_mesh" for (*, "rerun.components.Color", "albedo_factor"):
- ComponentDescriptor { "rerun.archetypes.Mesh3D", "rerun.components.Color", "albedo_factor" }

It's basically pattern matching.

This should be fairly trivial to implement on the query side.

Viewer changes

Today, each visualizer indicates the ComponentName it used to fetch a given piece of data:

In that world, each visualizer would not only show the ComponentDescriptor used to source data, but also allow the user to override the descriptors's archetype_name and tag fields. Want to use your vertex_colors as edge_colors? No problem!

Examples

SolidColor, EdgeColor, VertexColor, etc

Just use tags!

table SomeShapes3D {
  // ComponentDescriptor: "rerun.archetypes.SomeShapes3D::rerun.components.Color#solid_colors"
  solid_colors: [rerun.datatypes.Rgba32] ("attr.rerun.component": "rerun.components.Color", /* … */);
  // ComponentDescriptor: "rerun.archetypes.SomeShapes3D::rerun.components.Color#edge_colors"
  edge_colors: [rerun.datatypes.Rgba32] ("attr.rerun.component": "rerun.components.Color", /* … */);
  // ComponentDescriptor: "rerun.archetypes.SomeShapes3D::rerun.components.Color#vertex_colors"
  vertex_colors: [rerun.datatypes.Rgba32] ("attr.rerun.component": "rerun.components.Color", /* … */);
}

Transform3D vs. LeafTransform3D

Share the same exact code between the two, all the way. Just change the archetype_name part of the ComponentDescriptor at log time.

In-tree:

rr.log(
    "moon",
    # ComponentDescriptor: "rerun.archetypes.Transform3D::rerun.components.Rotation#rotation"
    rr.Transform3D(rotation=rr.Quaternion(xyzw=[0, -0.3826834, 0, 0.9238796])),
)

Out-of-tree:

rr.log(
    "moon",
    # `rr.LeafTransform3D` is just a helper that calls `rr.Transform3D` but sets the `archetype_name`
    # to `LeafTransform3D` instead.
    #
    # The `TransformContext` will make use of that information at runtime in order to dispatch things
    # appropriately.
    #
    # ComponentDescriptor: "rerun.archetypes.LeafTransform3D::rerun.components.Rotation#rotation"
    rr.LeafTransform3D(rotation=rr.Quaternion(xyzw=[0, -0.3826834, 0, 0.9238796])),
)

FAQ

What becomes of indicator components?

They're gone; they're just redundant at that point: use the archetype_name field in the ComponentDescriptor instead.

What about datatype conversions?

As far as I can tell, datatype conversions is a completely orthogonal problem.

"Tagged components" is about sharing semantics across columns, "datatype conversions" is about making it easy to change the memory layout of a column.

What about blueprint defaults/overrides?

Mostly nothing changes, except now the blueprint has an opportunity to define a default value for all tags or a specific one, or both.

blueprint.set_default("*::rerun.components.Color#*", Color::Blue);
blueprint.set_default("*::rerun.components.Color#vertex_colors", Color::Green);

What about the DataframeView?

The dataframe view now has all the information it needs to properly distinguish between data with similar semantics.

[jleibs]

As far as I can tell, datatype conversions is a completely orthogonal problem.

I disagree with with this. At most it's partially orthogonal. My big take away from the last year of working with our components and datatypes is that the line between semantics and datatypes is far from clear. I think this gives us an opportunity to improve that dramatically and we might as well plan to take advantage of it.

The previous strict hierarchy of component > datatype forced us into arbitrary distinctions. For example, consider color.

It was fairly clear that Color should be a component. But if color was the component, then the encoding of the color must (by necessity) be a datatype. This lead to datatypes like Rgba32 and the implication that ultimately every color representation should be it's own datatype. But, at the end of the day, rgba32 is still just a sematic tag on top of a datatype primitive such as Vec4u8. Similar story with rotations. Is quaternion its own datatype, or is it a semantic label on top of vec4f?

Tags appears to give us flexible yet structured way to introduce more dimensionality to the way we talk about data. In particular, things like Color-spaces, Units, Encodings, etc. all probably fit much better into semantic tags than they do in the datatype system.

For me this path leads to, for example talking about rotations as:

• tags: (component: rotation, representation: quat_wxyz), datatype: vec4f
• tags: (component: rotation, representation: mat3x3), datatype: mat3x3d

We could theoretically try to have BOTH component-tags and datatype-tags, but I suspect the nitpicking over which kind of tag goes where would be endless while providing little-to-no practical value.

The logical end-point is that datatypes are JUST arrow schema aliases they ONLY talk about the shape of the data (e.g. Vec4f) and never talk about what that data represents (e.g. Quat4f). Everything else becomes a component-descriptor tag.

This generally means that "datatype conversions" are totally generic and follow very standard software equivalents:

• Cast a u8 to an f32.
• Extract the 3rd field of a Vec4x

A big chunk of the complexity previously attributed to datatype conversions now becomes "Component descriptor conversions." Same problem, new name.

[teh-cmc]

Agree 100%

[grtlr]

I've had a discussion with @abey79 today that touched upon this PR. For the graph primitives PR #7500, it would be great if we could log multiple components of the same data type to the same entity, possibly by discerning them with tags so that edges and nodes can live in the same entity while still allowing fields like edge_colors and node_colors.

I was wondering: How will that work with selections? If I understand correctly, we currently refer to "objects" in an entity by their InstancePath. This works because the main (reuqired) component dictates the Instance within a batch. Now if we have an arbitrary amount of components (and therefore multiple batches) per entity wouldn't we need a special way to discern them too?

[abey79]

We don't currently have a solution for this for archetypes which dont have an "instance-to-instance" match semantic. That's the case for the Mesh3D archetype: the instance index of the Position3D and TriangleIndices have no relationship (contrary to e.g. Position3D <-> Color). So if you select an instance path like this, you get unrelated values for TriangleIndices and Position3D:

I don't think we have an established plan for this going forward. We could have an archetype-level field flag (that gets propagated as component tag) to indicate that "this component doesn't have instance-match semantics". It would be applied e.g. to Mesh3D's TriangleIndices or, for the Graph view, the Edge component. In the latter case, we'd want to go even further and tag a subset of components as having a shared instance index semantic (Edge, EdgeColor, EdgeLabel, etc.). Maybe? But that's future truth and will have to start without for now.