RFC: Event queue buffer for stats/prints

[giactra]

At the moment any simulator event (in its abstract form, to be distinguished from events in the EventQueue) in gem5 is consumed by either a stat update, or by a debug print. The consumption of such events happens locally within the simulator.
For debug prints, it will be directly printed to stdout; for stats, it will increase the embedded gem5 statistic.

DEBUG PRINTS

These can produce lots of information. Unfortunately, harvesting such information usually requires piping to a debug file. Especially for big ROIs, this means producing an unstructured data dump (human readable format) that simply does not scale. Any post processing tool will have to deal not only with its size, but also with parsing a sequence of strings that have no guarantee to be the same between different gem5 releases as users are allowed to modify the text of a dprint.

STATS

Stats are more evolved with respect to DEBUG prints as they already allow to switch the underlying back-end. Users can choose to dump stats to the traditional txt format, to CSV or to Json.
We can dump stats in a structured format but we are still paying the complexity price of having these formats somewhat embedded within the core of gem5.
We also pay the price of duplication: it could very well be that the same event (like a cache miss) should at the same time update a stat and print to a file.

PROPOSAL (RFC)

We should really be defining an abstract unified event system where any simulator events (like a stat update) are inserted into a queue (one per EventQueue?). It is up to a Consumer class to decide what to do with the event, whether to update a stat, print to stdout, serialize to a file, or update a PMU counter.
This is somehow similar to the ProbeListener approach, only with greater flexibility.
Multiple Consumers can be peeking at the same event in the FIFO queue and act on it accordingly. This means that (as an example) someone can write a consumer class that cherry-picks specific data in the event message and serialize it using avro/protobuf or whatever serialization protocol is chosen.

RISKS

This will make internal events somehow part of the API, considering that multiple tools could be relying on some of the specific fields to be present within an event push. This could complicate the gem5 release process and require the gem5 community to commit on maintaining backwards compatibility for some of the Messages

Any comments, ideas?

[powerjg]

Thanks for the RFC proposal! Overall, I like the core direction here of creating a stream/consumer model for statistics and other structured simulation events, but I have a few strong thoughts on the boundary between debugging and tracing:

• Keep Debug Flags Separated for Debugging: Debug flags and DPRINTF statements should remain strictly focused on developer debugging. We shouldn't try to position or use the debugging API as a tracing framework.
• Tracing Belongs in Tracing/Probe Interfaces: Tracing demands structured, performant, and predictable outputs. For structured trace data (like instruction traces or memory accesses), users should rely on probe points, ProbeListeners, and dedicated tracing interfaces rather than parsing debug outputs.
• Unified Event Consumer Model for Stats & Probes: The concept of an abstract consumer interface/queue is very promising if applied to statistics and probe-like events. Expanding this generic streaming idea as a complement or evolution to the Probe/Stats framework (allowing custom consumers to stream or serialize stats via Avro, Protobuf, CSV, etc.) makes a lot of sense, provided debug logging stays out of it.

Honestly, one of the main things here is that we need to write down more clearly how to use the probe interfaces and how to get the information out of them that people want. I agree that right now a lot of people are using debugging APIs to do traces, but I think that's because they don't know how to use the probe interfaces. I think if we document the probe interfaces better or create some agent skills around the probe interfaces, people will start using that, and we won't have people using debug interfaces for traces.