[3008.x] Defer OpenTelemetry imports in salt.utils.tracing / salt.utils.metrics (#69855)#69856
Merged
Merged
Conversation
Both modules unconditionally imported the OTel SDK at module load, even though ``tracing.enabled`` and ``metrics.enabled`` default to false. Every salt daemon entry point transitively imports both modules (via salt.master, salt.minion, salt.channel.*, salt.utils.event, salt.netapi.rest_cherrypy.app), so a ~15-process salt-master container was paying ~15 MB per subsystem per process -- ~450 MB total -- for functionality nobody was using. This showed up as a +300 MB baseline / +430 MB peak container RSS shift versus 3006.x on the nightly stress rig, and each ~300 MB salt engine child inherited the same overhead. Move the OTel imports into ``_load_otel()`` helpers invoked only after ``is_enabled()`` returns True, gated on ``_cached_opts`` (which is populated by ``configure()`` but requires the "enabled" key to be true for probes to fire). Public API is preserved; ``SpanKind`` remains a stub always and is translated to the real enum inside ``start_span()``.
twangboy
approved these changes
Jul 23, 2026
dwoz
added a commit
that referenced
this pull request
Jul 23, 2026
Both modules unconditionally imported the OTel SDK at module load, even though tracing.enabled and metrics.enabled default to false. Every salt daemon entry point transitively imports both modules, so a ~15-process salt-master container was paying ~15 MB per subsystem per process (~450 MB total) for functionality nobody was using. Move the OTel imports into _load_otel() helpers invoked only after is_enabled() returns True. Public API is preserved. Backport of 73bca44 from PR #69856.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #69855.
salt.utils.tracingandsalt.utils.metricsunconditionally imported the entire OpenTelemetry SDK at module load, even thoughtracing.enabledandmetrics.enabledboth default to false. Because both modules are imported transitively by every daemon entry point (salt.master,salt.minion,salt.channel.client,salt.channel.server,salt.utils.event,salt.netapi.rest_cherrypy.app,salt.utils.reactor), a ~15-process salt-master container was paying ~15 MB per subsystem per process — roughly 450 MB total — for functionality nobody was using.This landed on 3008.x and shows up on the nightly stress rig as a +300 MB baseline / +430 MB peak container RSS shift versus 3006.x under otherwise-identical workload:
(3008.x nightly: https://github.com/saltstack/salt/actions/runs/29970269739. 3006.x nightly: https://github.com/saltstack/salt/actions/runs/28280683908.)
Each ~300 MB salt engine child inherits the same overhead (engines run as separate Python processes that re-import
salt.master/salt.minion).Change
Move the OTel imports into
_load_otel()helpers invoked only afteris_enabled()returns True.is_enabled()short-circuits on_cached_optsbeing None orenabledbeing false, so the probe (and therefore the OTel import) doesn't fire on the default disabled path. Public API is preserved;SpanKindremains a stub always and is translated to the realtrace.SpanKindenum insidestart_span()when tracing is on.Per-process measurement of the imports removed from module top (Python 3.10 slim):
So each salt process trims by ~15 MB (tracing) + ~15 MB (metrics) = ~30 MB. Across the master container's ~15 Python processes that's ~450 MB reclaimed on a stack that has tracing/metrics disabled.
What issues does this PR fix or reference?
Fixes: #69855
Merge requirements satisfied?
changelog/69855.fixed.mdtests/pytests/unit/utils/test_tracing.py— new subprocess tests:test_import_does_not_load_opentelemetry— importingsalt.utils.tracing(and calling the disabled-path public API) must not put anyopentelemetry.*entries insys.modules.test_enabling_tracing_loads_opentelemetry_lazily— the mirror:configure({..., enabled: True})does trigger the import and a real span comes back.test_master_and_minion_imports_do_not_load_opentelemetry— end-to-end guard: importing the whole daemon import chain (salt.master,salt.minion,salt.channel.*,salt.utils.event,salt.utils.tracing) leavesopentelemetryunimported. Catches any future eager top-level import elsewhere in the chain.tests/pytests/unit/utils/test_metrics.py— matching pair:test_import_does_not_load_opentelemetryandtest_enabling_metrics_loads_opentelemetry_lazily.test_module_works_when_opentelemetry_missingtests updated to reflect the new tri-state_OTEL_AVAILABLE(Noneuntil first probe, thenTrue/False).pre-commitclean on all changed files.