RFC-0023: RFC: Updatable Track Events

[github-actions[bot]]

📄 RFC Doc: 0023-updatable-track-event.md

---

RFC: Updatable Track Events

Authors: @etiennep-chromium

Status: Decided

Problem

Currently, the Perfetto Track Event API provides TRACE_EVENT_BEGIN and TRACE_EVENT_END to define slices with duration. However, this model has limitations in a scenario common in Chrome:

Late Tracing of Long-Running Operations (State Machines)

Current Chrome Pattern (_END + _BEGIN):
In performance_manager, a TracedWrapper class observes tracing session starts to ensure properties are traced and emitted.

This is a single example, but this pattern is used by several components (about two dozen) throughout Chrome to emit long-running state.

When a new tracing session starts, TrackEventSessionObserver::OnStart is called, which triggers a re-emission of the state by ending and restarting the slice.

// In properties.h (TracedWrapper)
void OnStart(...) override { Trace(); }

void Trace() {
  if (!TRACE_EVENT_CATEGORY_ENABLED("performance_manager.graph")) {
    return;
  }
  if (slice_is_open_) {
    TRACE_EVENT_END("performance_manager.graph", track_);
    slice_is_open_ = false;
  }
  perfetto::StaticString value_str = converter_func_(value_);
  if (value_str) {
    TRACE_EVENT_BEGIN("performance_manager.graph", value_str, track_);
    slice_is_open_ = true;
  }
}

Drawbacks:

• For an already running tracing session, this unnecessarily breaks a logically single slice into multiple smaller slices every time a new tracing session starts or a value changes. This creates artificial track fragmentation and makes SQL analysis hard or incorrect when trying to overlap or intersect slice intervals.
• The _END and _BEGIN events will sample two separate timestamps, even though there is conceptually only one event.

---

Decision

Pending

Design

PoC: #5597

In practice, these state update don't need to support nested slices, so concerns about nesting are alleviated by explicitly choosing not to support it.

We introduce TRACE_STATE that takes a counter-like track argument, StateTrack, and thus cannot be mixed with TRACE_EVENT_BEGIN/END:

class StateTrack : public Track {
 public:
  StateTrack(StaticName name,
             uint64_t id = 0,
             Track parent = MakeProcessTrack());
  ...
};

TRACE_STATE(cat, <value>, StateTrack(track), ...);

• TRACE_STATE supports all the same arguments (lambda, annotations,
  flows, etc.) that other TRACE_ macros support.
• A special "null" value (e.g. nullptr) needs to be supported to show the empty track.
• TRACE_STATE supports both string and proto enums as values.
• Emitting the same value with different arguments will augment the slice.
• Trace processor emits state "slice" to a separate table (name TBD, e.g. state) that contains values, timestamps and durations.
• Visually state slices are the same as regular slices in perfetto UI.

Pros:

• Separates state machine updates from slice operations.
• Alleviates concerns about slice nesting.

Cons:

• Only solves one use case.

Alternatives considered

Option A: TRACE_EVENT_STEP (Original Proposal)

PoC: #4167

Introduce a new track event type, TYPE_SLICE_STEP, and a corresponding macro, TRACE_EVENT_STEP.

TRACE_EVENT_STEP acts as an "upsert" operation on the track's slice stack:

• If a matching incomplete slice is found on the stack, it updates that slice with any new arguments or flows provided.
• If there is an incomplete slice whose name does not match, that slice is closed and a new slice is opened with the new name.
• If no matching slice is found (e.g., because tracing started late), it starts a new slice, ensuring visibility.

Pros:

• Solves both use cases with a single concept.
• Handles late joining gracefully (becomes BEGIN if no slice found).
• Keeps traces flat and clean without custom RAII helpers.

Cons:

• Nesting can be confusing and will behave in a surprising way. Effectively, TRACE_EVENT_STEP only works for the leaf slice. This means we can't ensure multiple nested slices are present when starting a trace using TRACE_EVENT_STEP.

Open questions

• Lalit voiced concerns that capturing now when emitting events at the start of a tracing session to describe state that started in the past is the wrong timestamp (this happens in both Option A and B, although this is true of the status quo as well).

---

💬 Discussion Guidelines:

• This discussion is automatically synced with the RFC document
• Please provide constructive feedback and suggestions

[primiano]

One question: do you need to solve the two problems at the same time? or they are different instances that show up in different part of the codebase?

My take is that we need independently both option B and C, as they don't sound alternatives to me. they serve different purpose.
The just happen to both need "args update" semantics. But I would not force single solution just based on their need, as the overall semantic they want to express is different.

However, If you tell me you only need them together, then I would do B only for now, as it solves your problem (I think) and keep C to later on, if we ever need args-update for proper nested slices.

Dynamic Argument and Flow Augmentation
There is an alternative to the RAII helper, which would be something along the lines of

TRACE_EVENT_UPDATE_ARGS("navigation,rail", "child_frame_token", child_frame_token);

There is a drawback though: this will cost more bytes in the SMB and in the trace buffer, as it will need to emit a new event with its own timestamp and its own protobuf headers and so on.

If the event you want to agument is frequent, frankly I'd recommend you keep using your RAII helper as it saves trace bandwidth.
If not (sounds like this is one per navigation), we can afford the extra cost.

If we go this way then I'd just ask you to look into the details and formulate a proposal that is sound and composes well with the rest of the APIs.

Option B: Counter-like track with String (State Machines)
I quite like this overall.

We need to come up with a proposal on:

how this would look like API-wise though (I'm for a new TRACE_EVENT_XXX macro for this purpose)
What this will turn it into trace processor
.. and in the UI

[etiennep-chromium]

do you need to solve the two problems at the same time? or they are different instances that show up in different part of the codebase?

They don't need to be solved at once, so I agree with focusing on Use Case 1 here, with Option B (and punt a solution for Use Case 2 for another discussion).

how this would look like API-wise though (I'm for a new TRACE_EVENT_XXX macro for this purpose)

I've made a PoC here: #5597 (comment)
I ended up exposing TRACE_STATE(...) that takes a StateTrack, this works similar to TRACE_COUNTER(), although StateTrack's API is much closer to NamedTrack.

[LalitMaganti]

Lalit voiced concerns that capturing now when emitting events at the start of a tracing session to describe state that started in the past is the wrong timestamp (this happens in both Option A and B, although this is true of the status quo as well).

As long as we emit the timestamp of the start of the trace, there shouldn't be any problems here.

I agree with Primiano that Option B we should solve earlier and we should try and defer C to some point later. I've seen B appear a lot in many different places, I'm much more skeptical on the wide usecase of C.

[LalitMaganti]

Re. timestamp, when running TRACE_STATE() macro, how do we know that we should emit a "start of trace" timestamp rather than "now"; we don't have context on why this event is being emitted (starting a trace or state update).

The point I was making is if we have some hook to "dump on OnStart" in the TrackEvent DataSource, dumping at the "now" timestamp and dump on start is the same thing.

How does this work for TRACE_COUNTER()?

It doesn't :) TRACE_COUNTER has exactly the same problem and is the big reason why we've had push event with SDK + polling via some other pull interface throughout the Perfetto stack (e.g. cpu frequency).

[etiennep-chromium]

So either
1- The user (chrome) registers a TrackEventSessionObserver to watch trace start, and does TRACE_STATE() OnStart. Emitting start of trace timestamp here is tricky, because it would either emit it in all currently running session (which would be wrong for an already running session), or we expect the user to selectively emit only to the new session (that would probably require a separate macro that takes in StartArgs).
2- perfetto keeps track of state updates (pushed by the user) even when no session is running. This seems to fundamentally change how perfetto works though, as it's generally assumed trace macros are no-op when not tracing.

[LalitMaganti]

2- perfetto keeps track of state updates (pushed by the user) even when no session is running. This seems to fundamentally change how perfetto works though, as it's generally assumed trace macros are no-op when not tracing.

I think this is something you have to change if you need this to work properly. There's no other way to get the semantics you want here.

[etiennep-chromium]

There's no other way to get the semantics you want here.

While this is something that we might want to solve, getting the initial timestamp right is not the problem I was trying to solve with this proposal.

Keeping track of state updates in perfetto (at the cost of complexity and overhead) might be necessary to achieve "start of trace" initial timestamp. If we accept the initial timestamp is off (which is already the case with the current pattern of Use Case 1), then this simplifies the proposal (at the cost of more work for the user).

[LalitMaganti]

Sure if you're ok with getting the timestamp much later, then works for me and it's 1:1 with how TRACE_COUNTER works today.

[github-actions[bot]]

📝 RFC Document Updated

View changes: Commit History

[primiano]

@LalitMaganti & @sashwinbalaji : @etiennep-chromium and I found an other interesting problem while discussing #6068

The problem is substantially this, how do we encode the state in the Track event proto?
It mostly boils down to "how will TP figure out how to symbolize enums if they are extensions?"

one option (no brainer) is adding TYPE_STATE=5 (in enum Type) creating a string field. But that should be an escape hatch that should be used as last resort, as strings are wasteful and also create potential PII problems.

Instead we want to use enums as much as possible for the other cases. Now, enums should be extended and the open question is "how should trace processor know which extension are you using?"

So the overall problem is this

message TrackEvent {
...
   enum Type {
     ...
     TYPE_COUNTER = 4;
     TYPE_STATE = 5;   // This part is a no-brainer
}

oneof state_field {
  string state_value_str = ...;   // This part is a no-brainer but also shoudl be a last-resort
  ... This is the part up for discussion
}

Option 1: TraceProcessor guesses

In my_extension.proto

message MyStateExtension {

  enum MyStateEnum {
    MY_STATE_ENUM_UNKNOWN = 0;
    MY_STATE_ENUM_FOO = 1;
    MY_STATE_ENUM_BAR = 2;
  }
  optional Type type = 9;

  extend TrackEvent {
    optional MyStateEnum my_state = 9900;
  }
}

Then in the TrackEvent we would set

[field id 9] type: TYPE_STATE
[field id 9900] my_state = 2;  // _BAR

So far so good, at this level it's all a canonical use of protobuf.

Now think about TP parsing, how does TP know that it should use precisely extension 9900 to set the name of the counter?

The logic could be "Find the first (and likely only) field set in the TrackEvent extension range which is an enum, and use that"

PRO: very pragmatic and proto-conformant, likely JustWorks
CON: if somebody one day introduces another enum extension, which is meant to be added to existing packets (e.g. to achieve some semantic which is orthogonal to the track event itself) it becomes ambiguous for TP to decided which enum should be the one that defines the name.

But then I'd claim that the TrackEvent extensions should be de-facto oneof when you use TYPE_STATE, you are not supposed to set >1 extension value (but who know what the future holds)

Option 2: explicit, proto-conformant, larger and redundant

track_event.proto schema

oneof state_field {
  string state_value_str = 58;   
   int64 state_enum_extension_number = 59;  // Must contain to the extension nmber
}

as above but in the TrackEvent we set:

[field id 9] type: TYPE_STATE
[field id 59] state_enum_ext_number: 9900  // This is part of the schema
[field id 9900] my_state = 2;  // _BAR

Essentially we add a field that says "i'm using extension number 9900"

PRO: very explicit, non ambiguous
CON: redundant, as the number "9900" now shows up twice, one to say "i am using extension 9900" and then to say "the value for field id 9900 is 2 (BAR)"
CON: the programming model is a bit clunky. it could be hidden behind the API impl, for for people writing the protos by hand/tools it's a bit awkward

Option 3: explicit, non-redundant, abuses a bit extensions as a lookup dict

track_event.proto schema

oneof state_field {
  string state_value_str = 58;
  int64 state_value_int =  59; 
}

// Must be set when using state_value_int (if omitted, TP & UI will degenrate into a counter)
int64 state_extension_number = 60;

the TracePacket payload becomes

[field id 9] type: TYPE_STATE
[field id 59] state_value_int = 2;  // (_BAR)
[field id 60] state_ext_number = 9900;

PRO: explicit, non-ambiguous, non-redundant encoding
CON: Effectively we are defining an extension @ 9900 but nobody ever sets field ID 9900. We use it only as a way to "claim an extension number" and later use it to lookup the enum literals in TP.
it's a bit of an abuse of the proto extension mechanism.

My preference

In order:

• Option 3
• Option 2
• Option 1

Not sure if there is something else i'm not considering

[LalitMaganti]

Could state track be templated? That would solve the problem no?

[primiano]

How about Option 4: similar to option 2 but the field id you will use is set in the TrackDescriptor isntead. Wouldn't that solve all the problems?

That makes the protos even harder to write for other people and introduces more coupling with the TrackDescriptor (about interning losses etc)

[primiano]

Could state track be templated? That would solve the problem no?

No please let's not make things even more complicated to solve a rather simple problem (looking up an enum)
how is this supposed to work on the C-sdk if we do that?

[primiano]

TBH the more I think about this the more option 1 is the one that is the cleanest on the writer side.
Maybe we just bite the bullet and if ever introduce the need of filling >1 enum extension we put them in some other extension range (e.g., all the metadata, non-payload, extensions should start from 10k or the like) so it stays non-ambiguous in TP

[LalitMaganti]

There are other problems with option 1) though. People use args as extensions so we need to support that and distinguish them from the state field. Moreover not knowing the id is going to be more expensive to parse as well.