Allow multiple concurrent trace exporter tasks#191
Merged
Conversation
All tasks share a channel (gRPC)/socket (UDP) for output, but can process spans and send batches in parallel. This requires us to spawn the exporter futures rather than add them to TelemetryDriver's `tele_futures`, where they would be polled sequentially. While this changes the behavior of foundations' tracing output, we consider this a bugfix and thus appropriate for a minor release. Previously, trace output stopped as soon as the TelemetryDriver was dropped, but the process-wide tracer was still active.
TheJokr
force-pushed
the
lblocher/trace-num-workers
branch
from
44d2d10 to
2efd790
Compare
fisherdarling
approved these changes
Apr 9, 2026
TheJokr
added a commit
that referenced
this pull request
Apr 9, 2026
Added: - The `ratelimit!` utility macro simplifies the setup required for rate-limiting a code block into a single macro expression. There is also a special `ratelimit=` prefix syntax for log statements specifically. (#182) - The sentry metrics hook added in v5.5 now also supports rate-limiting for sentry events. To make use of this feature, call the new `foundations::sentry::install_hook_with_settings` setup function. (#183) - The telemetry server implements a `/pprof/symbol` endpoint now, which can be used for remote symbolization with pprof-compatible tools. (#186) - `foundations::telemetry::tracing::span_is_sampled()` provides a cheap way to check whether the current trace has been sampled. This allows skipping expensive tag/log formatting code if the values would be discarded anyway. (#187) - `Secret` (string) and `RawSecret` (bytes) wrappers have been added to aid with confidential values in config files. Both types hide their contents from Debug/Display calls and require an explicit accessors to retrieve the secret. Additionally, they zero their memory when dropped. (#188) - `MaybeExternal` is a new settings type that can load plain data (strings, bytes, and secrets) from either inline config or external sources (environment variables or file system). (#188) Improved: - `serde_yaml` was replaced by the new `serde-saphyr` YAML implementation. `serde_yaml` has been unmaintained since 2024. (#181) - Loggers can now be frozen, meaning any further mutation (such as `add_fields!`) will lead to an error. This is useful to catch bugs where mutations are applied to the wrong logger instance. (#189) - The maximum queue size for trace span output can now be limited via telemetry settings. The default has been set at 1 million spans. Additionally, there are new metrics to observe the queue size, total number of spans exported, and how many spans have been dropped. (#190) - Tracing can now be configured with multiple concurrent output tasks to boost span throughput. The tasks now run independently of the TelemetryDriver to ensure spans are output throughout the lifetime of the process. (#191) Fixed: - Log rate limiting now correctly applies across `set_verbosity` calls. (#180) Deprecated: - `foundations::sentry::install_hook` is deprecated in favor of `foundations::sentry::install_hook_with_settings`.
Merged
TheJokr
added a commit
that referenced
this pull request
Apr 9, 2026
Added: - The `ratelimit!` utility macro simplifies the setup required for rate-limiting a code block into a single macro expression. There is also a special `ratelimit=` prefix syntax for log statements specifically. (#182) - The sentry metrics hook added in v5.5 now also supports rate-limiting for sentry events. To make use of this feature, call the new `foundations::sentry::install_hook_with_settings` setup function. (#183) - The telemetry server implements a `/pprof/symbol` endpoint now, which can be used for remote symbolization with pprof-compatible tools. (#186) - `foundations::telemetry::tracing::span_is_sampled()` provides a cheap way to check whether the current trace has been sampled. This allows skipping expensive tag/log formatting code if the values would be discarded anyway. (#187) - `Secret` (string) and `RawSecret` (bytes) wrappers have been added to aid with confidential values in config files. Both types hide their contents from Debug/Display calls and require an explicit accessors to retrieve the secret. Additionally, they zero their memory when dropped. (#188) - `MaybeExternal` is a new settings type that can load plain data (strings, bytes, and secrets) from either inline config or external sources (environment variables or file system). (#188) Improved: - `serde_yaml` was replaced by the new `serde-saphyr` YAML implementation. `serde_yaml` has been unmaintained since 2024. (#181) - Loggers can now be frozen, meaning any further mutation (such as `add_fields!`) will lead to an error. This is useful to catch bugs where mutations are applied to the wrong logger instance. (#189) - The maximum queue size for trace span output can now be limited via telemetry settings. The default has been set at 1 million spans. Additionally, there are new metrics to observe the queue size, total number of spans exported, and how many spans have been dropped. (#190) - Tracing can now be configured with multiple concurrent output tasks to boost span throughput. The tasks now run independently of the TelemetryDriver to ensure spans are output throughout the lifetime of the process. (#191) Fixed: - Log rate limiting now correctly applies across `set_verbosity` calls. (#180) Deprecated: - `foundations::sentry::install_hook` is deprecated in favor of `foundations::sentry::install_hook_with_settings`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All tasks share a channel (gRPC)/socket (UDP) for output, but can process spans and send batches in parallel. This requires us to spawn the exporter futures rather than add them to TelemetryDriver's
tele_futures, where they would be polled sequentially.While this changes the behavior of foundations' tracing output, we consider this a bugfix and thus appropriate for a minor release. Previously, trace output stopped as soon as the TelemetryDriver was dropped, but the process-wide tracer was still active.