CTF2?

[minad]

@eepp @compudj Is a revision of CTF still planned? There was a proposal a while back. In particular leb128 or other variable length integers and data-stream-id would be useful. Maybe small extensions like this could be done step by step in a 1.9 revision instead of doing the full 2.0 thing with json metadata etc. Let me know if I should rather send an email to some mailing list, but the discussion seems to be spread over multiple lists (babeltrace, eclipse list, ...)

Thank you!

• https://www.eclipse.org/lists/tracecompass-dev/msg00800.html
• https://diamon.org/ctf/files/CTF2-PROP-1.0.html

[eepp]

Yes! It's on the roadmap and will unlock a lot of possibilities.

The plan is to implement both the consuming and producing sides in Babeltrace 2.1 to check if everything makes sense and works as expected. The CTF 2 terminology might change a little to match Babeltrace 2 API's also. Then, once Babeltrace 2.1 is released, we can make the next LTTng release produce CTF 2 traces.

Babeltrace 2.0 is almost ready. I'm currently completing the last ~5% of the C API documentation which is the last stable release blocker. So expect something in January. Then we can start working on 2.1 which includes a CTF 2 input/output implementation.

We don't want to add changes to CTF 1 as its strict TSDL grammar won't allow it.

I'll need to bring a few changes to the CTF 2 specification proposal, things we thought about during those years, namely:

• Remove all the "m-value" stuff and write the document using the JSON terminology to simplify the specification. I don't think we'll ever need something else than JSON so that seems okay, as long as all the objects are duly named.

• Update the terminology to match the Babeltrace 2 one as much as possible.

• Remove the null field type.

• Add the option field type (which is just like a variant field type containing some choice and a null choice).

• Mention that the union field type is semantically similar to the structure field type, that it only exists to optimize data stream size and tracer performance. In other words, a union field can (must?) be exposed as a structure field to the end user by the consuming program.

• Make it possible to embed event record classes into data stream classes, and data stream classes into a trace class, while keeping it possible to append an event class to a data stream class and a data stream class to a trace class.

  This should make the single-file metadata file easier to understand, having a tree structure.

• While a clock class remains an independent fragment, a data stream class fragment must list all the clock classes it intends to use.

• Packet headers and event record headers must only contain specified fields.

By the way, the data stream ID feature exists in CTF 1.8: it's the packet header's stream_instance_id field.

In the end the real list of new features brought by CTF 2 is:

• Terminology update.
• Metadata stream in JSON.
• Custom user attributes for almost all metadata objects.
• Object properties not needed to encode/decode data streams are not part of the specification.
• New field types:
  • Boolean.
  • Option.
  • Text array and sequence.
  • Union field type.
  • Variable-length integer and derivatives.
• Simpler field type aliases.
• Simpler field type resolution.
• Specification fields do not need special names.

[minad]

Good to hear! It seems stream_instance_id is not documented in the 1.8 specification? I meant the feature which allows to have different packets in one "physical stream" and which allows to abstract from the underlying representation of "metadata, stream1, stream2, ..." (https://diamon.org/ctf/files/CTF2-PROP-1.0.html#data-stream-id).

[minad]

Btw, which variable length field types do you consider? Only leb128 or other variants, unicode like prefix encodings?

[eepp]

Good to hear! It seems stream_instance_id is not documented in the 1.8 specification? I meant the feature which allows to have different packets in one "physical stream" and which allows to abstract from the underlying representation of "metadata, stream1, stream2, ..." (https://diamon.org/ctf/files/CTF2-PROP-1.0.html#data-stream-id).

It might not be documented but LTTng uses it and major CTF consumers understand it. Please note that, within a data stream file, all packets must have the same data stream ID. However many data stream files can have packets with the same data stream ID. This is why the feature exists.

So this is valid:

Data stream file A: #1 #1 #1 #1 #1 #1 #1 #1
Data stream file B: #1 #1 #1 #1 #1
Data stream file C: #2 #2 #2 #2 #2 #2 #2

This is not:

Data stream file A: #1 #1 #2 #1 #1 #2 #2 #2
Data stream file B: #2 #2 #1 #2

Btw, which variable length field types do you consider? Only leb128 or other variants, unicode like prefix encodings?

We were pretty set on LEB128 as it has many known users (LLVM, Protobuf, DWARF, WebAssembly, and others).

[minad]

Please note that, within a data stream file, all packets must have the same data stream ID. However many data stream files can have packets with the same data stream ID. This is why the feature exists.

Will this restriction still apply to CTF2? (CTF1.8 is unfortunately not an option for me since I am also using variable width integers leading to much more compact streams). In your proposal you wrote "CTF 2’s specification must not specify how to transport a trace, nor how a trace should be stored."

If yes, would it be possible to relax that restriction? For one, it is no problem to write a multiplexer, which performs the splitting. This has to be done for network streams for example. If you allow multiple streams per file, indexing and binary search is still possible. The producers can then decide themselves if the contention of the single data stream is acceptable.

In my use case I produce event logs in user space from many threads. I am also using LTTng/Systemtap for that purpose, but, if enabled during compilation, there is another event system doing a bit of in-process statistics, which also writes its own log.

Even in that case I do not have big problems using only one data stream file using some kind of ring buffer and writing batches of many events. Unfortunately it is not possible to write everything as one stream, since that constrains the ordering of the incoming events, which would require inefficient synchronization between the threads.

EDIT: Found example 5 in https://diamon.org/ctf/files/CTF2-FS-1.0.html which answers my question, where multiple streams are stored inside one data-stream-file.

[eepp]

If yes, would it be possible to relax that restriction? For one, it is no problem to write a multiplexer, which performs the splitting. This has to be done for network streams for example. If you allow multiple streams per file, indexing and binary search is still possible. The producers can then decide themselves if the contention of the single data stream is acceptable.

We'll think about this, if it's a use case.

The CTF 2 specification itself won't specify how to transport or store a trace, but other (optional) documents can. See, for example, CTF2-FS-1.0. This document indicates how to store a CTF 2 trace on the file system. Then producers (LTTng) can state that they apply CTF2-FS-1.0 and consumers (Babeltrace, Trace Compass) can state that they support CTF2-FS-1.0.

The current situation with CTF 1 is that consumers pretty much follow what LTTng does (or did). We want to formalize as much as we can with CTF 2: the specification itself, but also external documents for optional features.

We could indeed relax what CTF2-FS-1.0 recommends to make it possible to store (in order, per data stream) packets which belong to multiple data streams within the same data stream file. I'll think about this.

In my use case I produce event logs in user space from many threads. I am also using LTTng/Systemtap for that purpose, but, if enabled during compilation, there is another event system doing a bit of in-process statistics, which also writes its own log.

Even in that case I do not have big problems using only one data stream file using some kind of ring buffer and writing batches of many events. Unfortunately it is not possible to write everything as one stream, since that constrains the ordering of the incoming events, which would require inefficient synchronization between the threads.

@compudj knows more about this subject, I'll let him reply.

[minad]

We'll think about this, if it's a use case.

Well it is for me, but I cannot speak for a wider community here. Maybe for kernel logging or other high performance applications it is better to have separate files just to prevent contention as good as possible. But there will probably be contention on the cache/fs layers nonetheless. And maybe it would even be possible to still use a single file and pwrite blockwise to the log, leaving padding. So I am not sure if having multiple files is justified even in the high performance case. But this is speculation - I don't know enough about the low level details here.

It is completely understandable that you mostly follow LTTng. For me CTF almosts fits, except for the multi-file issue and the variable width integer issue. Afaik there are other people adopting CTF, for example the Ocaml people in their compiler/runtime. Right now they are using one stream, but there are also multicore efforts. It would be nice if the format could be adopted for different purposes, in particular since all the analysis infrastructure can be shared (babeltrace, tracecompass etc). I also tried chrome tracing, which is painful (json datastream and viewer in the browser). So I think CTF fills a nice niche here.

We could indeed relax what CTF2-FS-1.0 recommends to make it possible to store (in order, per data stream) packets which belong to multiple data streams within the same data stream file. I'll think about this.

As far as I see it, this is how it is described as of now in CTF2-FS-1.0, see example 5 in https://diamon.org/ctf/files/CTF2-FS-1.0.html

streamW (contains some packets of data stream 0 and of data stream 1)

[eepp]

Well it is for me, but I cannot speak for a wider community here. Maybe for kernel logging or other high performance applications it is better to have separate files just to prevent contention as good as possible. But there will probably be contention on the cache/fs layers nonetheless. And maybe it would even be possible to still use a single file and pwrite blockwise to the log, leaving padding. So I am not sure if having multiple files is justified even in the high performance case. But this is speculation - I don't know enough about the low level details here.

Honestly it might just be simpler to implement with multiple files. Each per-CPU ring buffer gets serialized to its own file. But I don't know the exact rationale on the tracer side.

We could indeed relax what CTF2-FS-1.0 recommends to make it possible to store (in order, per data stream) packets which belong to multiple data streams within the same data stream file. I'll think about this.

As far as I see it, this is how it is described as of now in CTF2-FS-1.0, see example 5 in https://diamon.org/ctf/files/CTF2-FS-1.0.html

streamW (contains some packets of data stream 0 and of data stream 1)

Well... you're right. 😲

This document is so old, I lost track of its content.

What I do know however is that current CTF 1 consumers do not support this.

We just need to think about the impact of supporting such a packet layout. Currently, Babeltrace 2 actively checks that each packet of a given file belongs to the same logical data stream.

[eepp]

Okay so I spoke with @compudj and we are not ruling out the single file with multiple logical data streams idea.

The reasons LTTng does this now is:

• In preparation for an eventual upgrade of the LTTng consumer daemon where multiple threads write to their own files, possibly located on multiple disks.

  We never tested this and the consumer daemon is still single-threaded.

• Index files were introduced after LTTng started producing CTF.

  If all the logical data streams are part of the same file, then you must index all the file's packets before you start to read because you have no way to know how many data streams you have. A packet (of data stream 1) at the end of the file could contain records of events which occured before the events recorded in the file's first packet (of data stream 0), for example.

  For a very large file (think 100+ GB), depending on the packet sizes, the indexing operation can be long. This means reading a packet header and then seeking to the next packet; repeat until the end of file. With a precomputed index (like LTTng writes today), this is not a problem.

[minad]

Okay so I spoke with @compudj and we are not ruling out the single file with multiple logical data streams idea.

Cool!

With a precomputed index (like LTTng writes today), this is not a problem.

Maybe the index format could be specified as addition to CTF2. It shouldn't be a problem for a producer to create the index at runtime, at least in the use cases I have in mind it would be easy to add.

Thank you for your answers and I am looking forward to the the next releases :)

[minad]

@eepp @compudj Any news on ctf2?

[PSRCode]

Hi Minad,

Unfortunately our effort have been focused on other projects. So no there is no news on the CTF2 side. With the release of Babeltrace 2 (22 January 2020), we are one step closer to working on CTF2. CTF2 support is probably the next big milestone for Babelrace 2 and LTTng tracers. The need for CTF2 is resurfacing for some features we want to implement for our tracers. Still, as of today, it is unclear if we will have bandwidth to work on that this year. I expect to have a clearer view of the available bandwidth in the upcoming month. I'll make sure to provide you with some update as soon as I have some information on that end.

Cheers

[minad]

@PSRCode thank you for the update!

[PSRCode]

Hi Minad,

Work on CTF2 is ongoing. Please take a look at the new proposal for CTF2 here:

https://lists.lttng.org/pipermail/lttng-dev/2020-November/029777.html

Cheers

[minad]

@PSRCode Thank you for pinging me! I will take a look. What are the plans regarding implementation (babeltrace/tracecompass)?