subscriber: Per-layer filtering #302
Labels
crate/subscriber
Related to the `tracing-subscriber` crate
kind/feature
New feature or request
kind/rfc
A request for comments to discuss future changes
needs/design
Additional design discussion is required.
Comments
hawkw
added
kind/feature
|
@LucioFranco, @davidbarsky, @jarrodldavis it would be great to get your input on this if you're interested in weighing in. I can flesh out the design questions a bit more but wanted to get the basic ideas written down first. |
|
My main use-case would be separate filtering rules for separate sinks (likely via multiple subscribers à la #135). |
Closed
|
Here are my initial thoughts:
I think that a system like
I think that the entire reason for splitting apart instrumentation data into constructs like metrics and logs is to enable necessary cost optimizations pertaining to ingestion volume and storage. For those use-cases, I think that per-layer filtering/selection is critical.
Depending on what you mean by "optimize", but yes, I think that
Given the tree-like structure that emerges once
I'm not sure this can be avoided in the case of multiple, disjoint
To connect back to the tree model of layers and subscribers, I view callsite |
|
Chiming in here as we've moved all of our logging to tracing and have recently added OpenTelemetry + Honeycomb into the mix. Our main use case is to be able to send We generally don't want the verbosity of Looking at #508, the API seems solid. Attaching the filters directly to the layers seems like the right approach. In this scheme, do global filters take precedence over layer-specific filters? |
|
I'll try answering some of the design questions as they pertain to our use case:
I can't speak for everyone of course but IMO as soon as there's more than one layer the need for filtering on a per-layer basis becomes important. In our case, it's basically impossible for someone to make sense of
From our perspective, we've yet to encounter having to call this method at all, which IMO is a win from the UX side so I don't really have any opinions here. It sounds like this is more relevant for
These two questions are beyond my current understanding of tracing :) |
facebook-github-bot
pushed a commit
to facebook/sapling
that referenced
this issue
Feb 23, 2021
Summary:
Previously, one has to set both EDENSCM_LOG=trace and EDENSCM_TRACE_LEVEL=trace
to enable all logging. That is because the filtering is global - one subscriber
(or layer) filtering out a span or event, then the span or event is gone
forever.
For now, let's just set the TracingCollector Subscriber's filter level
(EDENSCM_TRACE_LEVEL) to TRACE so it solely depends on the EnvFilter
(EDENSCM_LOG) if EDENSCM_LOG is set. That's more friendly.
See `impl<L: Layer<S>, S: Subscriber> Subscriber for Layered<L, S>`:
fn enabled(&self, metadata: &Metadata<'_>) -> bool {
if self.layer.enabled(metadata, self.ctx()) {
// if the outer layer enables the callsite metadata, ask the subscriber.
self.inner.enabled(metadata)
} else {
// otherwise, the callsite is disabled by the layer
false
}
}
See also: tokio-rs/tracing#302
Reviewed By: sfilipco
Differential Revision: D26518017
fbshipit-source-id: 9016b940621fc9a883e6ca3f00eaaeccc051ae74
|
Is there any new news about this issue? |
5 tasks
hawkw
added a commit
that referenced
this issue
Sep 9, 2021
## Motivation Currently, filtering with `Layer`s is always _global_. If a `Layer` performs filtering, it will disable a span or event for _all_ layers that compose the current subscriber. In some cases, however, it is often desirable for individual layers to see different spans or events than the rest of the `Layer` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Layer`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Layer::on_event` and `Layer::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-layer filtering implementation would satisfy the following _desiderata_: * If a per-layer filter disables a span, it shouldn't be present _for the layer that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-layer filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular layers. This means that per-layer filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-layer filters shouldn't interfere with non-filtered `Layer`s. If a subscriber contains layers without any filters, as well as layers with per-layer filters, the non-filtered `Layer`s should behave exactly as they would without any per-layer filters present. * Similarly, per-layer filters shouldn't interfere with global filtering. If a `Layer` in a stack is performing global filtering (e.g. the current filtering behavior), per-layer filters should also be effected by the global filter. * Per-layer filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-layer-filtered layers_. * Per-layer filters should be _tree-shaped_. If a `Subscriber` consists of multiple layers that have been `Layered` together to form new `Layer`s, and some of the `Layered` layers have per-layer filters, those per-layer filters should effect all layers in that subtree. Similarly, if `Layer`s in a per-layer filtered subtree have their _own_ per-layer filters, those layers should be effected by the union of their own filters and any per-layer filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-layer filtering correctly is somewhat more complex than simply skipping events and spans in a `Layer`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-layer filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-layer filtering feature _doesn't_ interfere with the global filtering behavior that layers currently have when not using the new per-layer filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-layer filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Layer`. When `enabled` is called on a `Filtered` layer, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-layer filter disables a span/event, and there are no non-per-layer-filtered layers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` layer's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped layer if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_layer` method was added to the `Layer` trait, which allows running a callback when adding a `Layer` to a `Subscriber`. This allows mutating both the `Layer` and the `Subscriber`, since they are both passed by value when layering a `Layer` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` layer is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_layer` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` layer. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-layer filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` layer in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 layers, let alone 64 separate per-layer filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` layers. When `Filtered` layers are nested using the `Layered`, the `Context` _combines_ the filter ID of the inner and outer layers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `layer` module. Since we were adding more code to the `Layer` module (the `Filter` trait), the `layer.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Layered` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-layer filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-layer filtering, I added a new `Layer` implementation that does the same thing, but implements `Layer` rather than `Subscriber`. This allows testing per-layer filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-layer filters. This is in order to avoid stabilizing a lot of the per-layer filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host layers with per-layer filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-layer filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Layer` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-layer filter evaluations, so that we don't need to always use `enabled` when there are several per-layer filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
|
Implemented in #1523! |
davidbarsky
added a commit
that referenced
this issue
Nov 29, 2021
## Motivation Currently, filtering with `Layer`s is always _global_. If a `Layer` performs filtering, it will disable a span or event for _all_ layers that compose the current subscriber. In some cases, however, it is often desirable for individual layers to see different spans or events than the rest of the `Layer` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Layer`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Layer::on_event` and `Layer::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-layer filtering implementation would satisfy the following _desiderata_: * If a per-layer filter disables a span, it shouldn't be present _for the layer that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-layer filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular layers. This means that per-layer filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-layer filters shouldn't interfere with non-filtered `Layer`s. If a subscriber contains layers without any filters, as well as layers with per-layer filters, the non-filtered `Layer`s should behave exactly as they would without any per-layer filters present. * Similarly, per-layer filters shouldn't interfere with global filtering. If a `Layer` in a stack is performing global filtering (e.g. the current filtering behavior), per-layer filters should also be effected by the global filter. * Per-layer filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-layer-filtered layers_. * Per-layer filters should be _tree-shaped_. If a `Subscriber` consists of multiple layers that have been `Layered` together to form new `Layer`s, and some of the `Layered` layers have per-layer filters, those per-layer filters should effect all layers in that subtree. Similarly, if `Layer`s in a per-layer filtered subtree have their _own_ per-layer filters, those layers should be effected by the union of their own filters and any per-layer filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-layer filtering correctly is somewhat more complex than simply skipping events and spans in a `Layer`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-layer filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-layer filtering feature _doesn't_ interfere with the global filtering behavior that layers currently have when not using the new per-layer filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-layer filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Layer`. When `enabled` is called on a `Filtered` layer, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-layer filter disables a span/event, and there are no non-per-layer-filtered layers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` layer's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped layer if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_layer` method was added to the `Layer` trait, which allows running a callback when adding a `Layer` to a `Subscriber`. This allows mutating both the `Layer` and the `Subscriber`, since they are both passed by value when layering a `Layer` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` layer is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_layer` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` layer. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-layer filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` layer in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 layers, let alone 64 separate per-layer filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` layers. When `Filtered` layers are nested using the `Layered`, the `Context` _combines_ the filter ID of the inner and outer layers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `layer` module. Since we were adding more code to the `Layer` module (the `Filter` trait), the `layer.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Layered` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-layer filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-layer filtering, I added a new `Layer` implementation that does the same thing, but implements `Layer` rather than `Subscriber`. This allows testing per-layer filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-layer filters. This is in order to avoid stabilizing a lot of the per-layer filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host layers with per-layer filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-layer filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Layer` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-layer filter evaluations, so that we don't need to always use `enabled` when there are several per-layer filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 23, 2022
; This is the 1st commit message: subscriber: implement per-subscriber filtering (#1523) ## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io> ; The commit message #2 will be skipped: ; fixup: start making changes for filter ; The commit message #3 will be skipped: ; fixup: move stuff to `subscribe` folder from `layer ; The commit message #4 will be skipped: ; fixup: additional compilation errors
hawkw
added a commit
that referenced
this issue
Mar 23, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 23, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 23, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
hawkw
added a commit
that referenced
this issue
Mar 24, 2022
## Motivation Currently, filtering with `Subscribe`s is always _global_. If a `Subscribe` performs filtering, it will disable a span or event for _all_ subscribers that compose the current subscriber. In some cases, however, it is often desirable for individual subscribers to see different spans or events than the rest of the `Subscribe` stack. Issues in other projects, such as tokio-rs/console#64 and tokio-rs/console#76, linkerd/linkerd2-proxy#601, influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681, jackwaudby/spaghetti#86, etc; as well as `tracing` feature requests like #302, #597, and #1348, all indicate that there is significant demand for the ability to add filters to individual `Subscribe`s. Unfortunately, doing this nicely is somewhat complicated. Although a naive implementation that simply skips events/spans in `Subscribe::on_event` and `Subscribe::new_span` based on some filter is relatively simple, this wouldn't really be an ideal solution, for a number of reasons. A proper per-subscriber filtering implementation would satisfy the following _desiderata_: * If a per-subscriber filter disables a span, it shouldn't be present _for the subscriber that filter is attached to_ when iterating over span contexts (such as `Context::event_scope`, `SpanRef::scope`, etc), or when looking up a span's parents. * When _all_ per-subscriber filters disable a span or event, it should be completely disabled, rather than simply skipped by those particular subscribers. This means that per-subscriber filters should participate in `enabled`, as well as being able to skip spans and events in `new_span` and `on_event`. * Per-subscriber filters shouldn't interfere with non-filtered `Subscribe`s. If a subscriber contains subscribers without any filters, as well as subscribers with per-subscriber filters, the non-filtered `Subscribe`s should behave exactly as they would without any per-subscriber filters present. * Similarly, per-subscriber filters shouldn't interfere with global filtering. If a `Subscribe` in a stack is performing global filtering (e.g. the current filtering behavior), per-subscriber filters should also be effected by the global filter. * Per-subscriber filters _should_ be able to participate in `Interest` caching, _but only when doing so doesn't interfere with non-per-subscriber-filtered subscribers_. * Per-subscriber filters should be _tree-shaped_. If a `Subscriber` consists of multiple subscribers that have been `Subscribeed` together to form new `Subscribe`s, and some of the `Subscribeed` subscribers have per-subscriber filters, those per-subscriber filters should effect all subscribers in that subtree. Similarly, if `Subscribe`s in a per-subscriber filtered subtree have their _own_ per-subscriber filters, those subscribers should be effected by the union of their own filters and any per-subscriber filters that wrap them at higher levels in the tree. Meeting all these requirements means that implementing per-subscriber filtering correctly is somewhat more complex than simply skipping events and spans in a `Subscribe`'s `on_event` and `new_span` callbacks. ## Solution This branch has a basic working implementation of per-subscriber filtering for `tracing-subscriber` v0.2. It should be possible to add this in a point release without a breaking change --- in particular, the new per-subscriber filtering feature _doesn't_ interfere with the global filtering behavior that subscribers currently have when not using the new per-subscriber filtering APIs, so existing configurations should all behave exactly as they do now. ### Implementation The new per-subscriber filtering API consists of a `Filter` trait that defines a filtering strategy and a `Filtered` type that combines a `Filter` with a `Subscribe`. When `enabled` is called on a `Filtered` subscriber, it checks the metadata against the `Filter` and stores whether that filter enabled the span/event in a thread-local cell. If every per-subscriber filter disables a span/event, and there are no non-per-subscriber-filtered subscribers in use, the `Registry`'s `enabled` will return `false`. Otherwise, when the span/event is recorded, each `Filtered` subscriber's `on_event` and `new_span` method checks with the `Registry` to see whether _it_ enabled or disabled that span/event, and skips passing it to the wrapped subscriber if it was disabled. When a span is recorded, the `Registry` which filters disabled it, so that they can skip it when looking up spans in other callbacks. A new `on_subscriber` method was added to the `Subscribe` trait, which allows running a callback when adding a `Subscribe` to a `Subscriber`. This allows mutating both the `Subscribe` and the `Subscriber`, since they are both passed by value when subscribering a `Subscribe` onto a `Subscriber`, and a new `register_filter` method was added to `LookupSpan`, which returns a `FilterId` type. When a `Filtered` subscriber is added to a subscriber that implements `LookupSpan`, it calls `register_filter` in its `on_subscriber` hook to generate a unique ID for its filter. `FilterId` can be used to look up whether a span or event was enabled by the `Filtered` subscriber. Internally, the filter results are stored in a simple bitset to reduce the performance overhead of adding per-subscriber filtering as much as possible. The `FilterId` type is actually a bitmask that's used to set the bit corresponding to a `Filtered` subscriber in that bitmap when it disables a span or event, and check whether the bit is set when querying if a span or event was disabled. Setting a bit in the bitset and testing whether its set is extremely efficient --- it's just a couple of bitwise ops. Additionally, the "filter map" that tracks which filters disabled a span or event is just a single `u64` --- we don't need to allocate a costly `HashSet` or similar every time we want to filter an event. This *does* mean that we are limited to 64 unique filters per `Registry`...however, I would be _very_ surprised if anyone _ever_ builds a subscriber with 64 subscribers, let alone 64 separate per-subscriber filters. Additionally, the `Context` and `SpanRef` types now also potentially carry `FilterId`s, so that they can filter span lookups and span traversals for `Filtered` subscribers. When `Filtered` subscribers are nested using the `Subscribeed`, the `Context` _combines_ the filter ID of the inner and outer subscribers so that spans can be skipped if they were disabled by either. Finally, an implementation of the `Filter` trait was added for `LevelFilter`, and new `FilterFn` and `DynFilterFn` structs were added to produce a `Filter` implementation from a function pointer or closure. ### Notes Some other miscellaneous factors to consider when reviewing this change include: * I did a bit of unrelated refactoring in the `subscriber` module. Since we were adding more code to the `Subscribe` module (the `Filter` trait), the `subscriber.rs` file got significantly longer and was becoming somewhat hard to navigate. Therefore, I split out the `Context` and `Subscribeed` types into their own files. Most of the code in those files is the same as it was before, although some of it was changed in order to support per-subscriber filtering. Apologies in advance to reviewers for this... * In order to test this change, I added some new test-support code in `tracing-subscriber`. The `tracing` test support code provides a `Subscriber` implementation that can be configured to expect a particular set of spans and events, and assert that they occurred as expected. In order to test per-subscriber filtering, I added a new `Subscribe` implementation that does the same thing, but implements `Subscribe` rather than `Subscriber`. This allows testing per-subscriber filter behavior in the same way we test global filter behavior. Reviewers should feel free to just skim the test the test support changes, or skip reviewing them entirely. ## Future Work There is a bunch of additional stuff we can do once this change lands. In order to limit the size of this PR, I didn't make these changes yet, but once this merges, we will want to consider some important follow up changes: * [ ] Currently, _only_ `tracing-subscriber`'s `Registry` is capable of serving as a root subscriber for per-subscriber filters. This is in order to avoid stabilizing a lot of the per-subscriber filtering design as public API without taking the time to ensure there are no serious issues. In the future, we will want to add new APIs to allow users to implement their own root `Subscriber`s that can host subscribers with per-subscriber filtering. * [ ] The `EnvFilter` type doesn't implement the `Filter` trait and thus cannot currently be used as a per-subscriber filter. We should add an implementation of `Filter` for `EnvFilter`. * [ ] The new `FilterFn` and `DynFilterFn` types could conceivably _also_ be used as global filters. We could consider adding `Subscribe` implementations to allow them to be used for global filtering. * [ ] We could consider adding new `Filter` implementations for performing common filtering predicates. For example, "only enable spans/events with a particular target or a set of targets", "only enable spans", "only enable events", etc. Then, we could add a combinator API to combine these predicates to build new filters. * [ ] It would be nice if the `Interest` system could be used to cache the result of multiple individual per-subscriber filter evaluations, so that we don't need to always use `enabled` when there are several per-subscriber filters that disagree on whether or not they want a given span/event. There are a lot of unused bits in `Interest` that could potentially be used for this purpose. However, this would require new API changes in `tracing-core`, and is therefore out of scope for this PR. Closes #302 Closes #597 Closes #1348 Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature Request
Crates
tracing-subscriberMotivation
A primary use-case for
tracing-subscriber'sLayertrait is to support teeing trace events between multiple consumers that implement different behaviors. WhenLayeris used in this way, it should be possible to add filtering toLayers to allow the individualLayerto decide what spans and events it is interested in recording, without disabling those spans/events for the entire stack.Currently, however, the
register_callsiteandenabledmethods onLayerare designed to support a use-case where a layer does deny or accept events for the whole stack. This is intended to allow aLayerto globally disable spans and events to support a use-case where a single filter is used for the whole stack, or use-cases such as rate-limiting etc.This should probably be revisited. Ideally, it would be possible to support both use-cases without adding too much more additional complexity to the interface.
Design Questions
There are some design questions that should inform a solution to this problem.
tracing-subscriberwant to be able to add a single filter/sampler layer to a stack that controls the behaviour of the entire composed subscriber. Is this actually the case? In a scenario where multiple layers that implement different ways of recording trace events (e.g. logging, metrics, etc) or output to different sinks (e.g., logging to a file, stderr, and stdout), will users want to have a global filter configuration that applies to all of these? And if so, is this common enough that we should optimize for it?Layer::enabledbe used for per-layer filtering? Or, should a new trait method be added? Having two similar but different methods sounds potentially confusing.Layer::enabledbeing called multiple times for the same event?Interestcaching?The text was updated successfully, but these errors were encountered: