core: introduce statically discoverable metadata

[davidbarsky]

Feature Request

Crates

• tracing-core

Motivation

A common complaint with tracing's filtering behavior is that feedback for invalid filters can be silent—there is no diagnostic saying "this target does not exist" because tracing-core cannot tell the difference between a nonexistent callsite and a yet-to-be-discovered callsite, as callsite registration occurs as the instrumented code executes for the first time, not ahead of time.

To address this issue, this issue proposes that all callsites became statically visible and discoverable at program startup. This would allow other, higher-level libraries to find all callsites and implement interesting functionality such as:

• Meaningful errors for filters referencing invalid/nonexistent callsites.
• GUIs and other visualizations of tracing-emitted data that know which queries are valid and invalid.
• Testing utilities that allow users to verify whether their intended instrumentation is actually emitted (other alternatives might be preferable).

Proposal

The implementation details might be related to #860. Here are two options that I know of:

• Linker misuse to place all metadata within a single, large array at compile time. A downside of this approach is that it will not work on WASM and doesn't work on MSVC.
• Rewriting tracing's macros to be procedural macros and making use of recently-stabilized const evaluation features. (This was mentioned to me by @oli-obk mentioned this in passing so it's possible I misunderstood!)

Alternatives

• Don't do this.

[oli-obk]

Rewriting tracing's macros to be procedural macros and making use of recently-stabilized const evaluation features.

There are possible future extensions to const eval that are completely in the hypothetical spectrum right now: rust-lang/const-eval#25 (comment). Some variants of this would really help with the problem we have here. Basically each call site would write to a file using a specific naming pattern and then we could collect all these files in the final crate.

While we can add a proc macro and invoke it from the callsite macro (as that seems to be the entry point invoked for anything that has any kind of filtering) and make that proc macro, when invoked, do nothing but mutate global state in the proc macro that can be used in one single invocation to dump an array of all the call site info, that seems very fragile as the compiler can nuke your proc macro's state at any time and invoke the proc macro in arbitrary orders. This will likely not go well with incremental compilation either.

The only workable solution I can come up with is actually exactly what eddyb suggests in rust-lang/const-eval#25 (comment) but done with proc macro attributes. So we basically have only a single #![gather_callsites] attribute and don't change anything else. This attribute is invoked at the top level of the crate and will do nothing to the crate's AST, but it will parse it for any struct constructors of the MetaData struct or invocations of any of the regular tracing macros. It will use this collected data to create a public const item (TRACING_METADATA?) that contains all the info we need.

The thing I don't really know is how we're going to move this data across crate boundaries in a non-manual way. We can start out by requiring the #![gather_callsites] macro to also get a list of crates it should grab the data from and append (this appending is only possible since we got [T]::len stable as const fn) it to the current crate's list. If any crate in the dependency tree does not use gather_callsites, you won't get any callsites from that dependency or from its dependencies (unless these dependencies also show up in another path of your dep tree).

[hawkw]

My assumption is that, since the required const-eval features that @oli-obk mentions are currently hypothetical, we're not going to be able to implement this for tracing 0.2, and it should be removed from the milestone.

[oli-obk]

yes, this is entirely hypothetical and I don't see any of my suggestions happening soon