fix(obs): disable profiling export by default and fix Helm env name#2719
fix(obs): disable profiling export by default and fix Helm env name#2719
Conversation
|
CLA requirements are satisfied for this pull request. |
There was a problem hiding this comment.
Pull request overview
Disables Pyroscope profiling export by default to prevent unbounded profile buffering/allocations when the collector endpoint is unreachable, and aligns Helm/config/docs with the corrected env-var behavior.
Changes:
- Flip
DEFAULT_OBS_PROFILING_EXPORT_ENABLEDfromtruetofalse. - Add backward-compatible env handling for the legacy
RUSTFS_OBS_PROFILING_ENABLEDtoggle (with tests) and update docs. - Fix Helm chart to emit the correct env var name for profiling export enablement.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
crates/config/src/constants/app.rs |
Changes the default profiling export toggle to disabled (false). |
crates/obs/src/config.rs |
Adds legacy env alias support + tests; wires new helper into config extraction. |
helm/rustfs/templates/configmap.yaml |
Renames Helm-emitted env var to RUSTFS_OBS_PROFILING_EXPORT_ENABLED. |
crates/obs/README.md |
Updates documented default for profiling export to false. |
| | `RUSTFS_OBS_TRACES_EXPORT_ENABLED` | `true` | Toggle trace export | | ||
| | `RUSTFS_OBS_METRICS_EXPORT_ENABLED` | `true` | Toggle metrics export | | ||
| | `RUSTFS_OBS_LOGS_EXPORT_ENABLED` | `true` | Toggle OTLP log export | | ||
| | `RUSTFS_OBS_PROFILING_EXPORT_ENABLED` | `true` | Toggle profiling export | | ||
| | `RUSTFS_OBS_PROFILING_EXPORT_ENABLED` | `false` | Toggle profiling export | |
There was a problem hiding this comment.
The README’s “Routing Logic” section earlier states that any OTLP endpoint implies a “Full OTLP/HTTP pipeline (… + profiling)”. With RUSTFS_OBS_PROFILING_EXPORT_ENABLED now defaulting to false, that statement is no longer accurate—profiling export requires an explicit opt-in. Please update the routing logic text to reflect the new default/opt-in behavior to avoid operator confusion.
| logs_export_enabled: Some(get_env_bool(ENV_OBS_LOGS_EXPORT_ENABLED, DEFAULT_OBS_LOGS_EXPORT_ENABLED)), | ||
| profiling_export_enabled: Some(get_env_bool(ENV_OBS_PROFILING_EXPORT_ENABLED, DEFAULT_OBS_PROFILING_EXPORT_ENABLED)), | ||
| profiling_export_enabled: Some(get_env_bool_with_aliases( | ||
| ENV_OBS_PROFILING_EXPORT_ENABLED, | ||
| &[LEGACY_ENV_OBS_PROFILING_ENABLED], | ||
| DEFAULT_OBS_PROFILING_EXPORT_ENABLED, | ||
| )), |
There was a problem hiding this comment.
The PR description mentions adding a read_profiling_export_enabled() helper, but the implementation here directly calls get_env_bool_with_aliases(...) inline. Either update the PR description to match the actual implementation or add the helper as described (if that was the intent) to keep the change log accurate.
Type of Change
Related Issues
#2715
Summary of Changes
Problem
When
RUSTFS_OBS_ENDPOINTis set but the Pyroscope/OTel collector endpoint isunreachable, the profiling export background task accumulates data indefinitely.
After 2-3 days of idle (no bucket traffic) this eventually triggers an extremely
large allocation request (
memory allocation of 2267059503403440 bytes failed)and causes the process to abort.
Root cause
DEFAULT_OBS_PROFILING_EXPORT_ENABLEDwastrue, meaning any deployment thatset an OTLP root endpoint automatically activated the Pyroscope profiling export
path on Linux/macOS — even when no Pyroscope endpoint was reachable or intended.
Additionally, the Helm chart emitted
RUSTFS_OBS_PROFILING_ENABLED(wrong name)instead of
RUSTFS_OBS_PROFILING_EXPORT_ENABLED, so the intendedfalsevaluein the chart was silently ignored and the runtime default (
true) took effect.Changes
crates/config/src/constants/app.rsDEFAULT_OBS_PROFILING_EXPORT_ENABLED:true→false.crates/obs/src/config.rsread_profiling_export_enabled()that recognisesthe legacy alias
RUSTFS_OBS_PROFILING_ENABLED(as emitted by older Helmcharts) so existing deployments with
RUSTFS_OBS_PROFILING_ENABLED=truecontinue to work without a config change.
RUSTFS_OBS_PROFILING_EXPORT_ENABLED>RUSTFS_OBS_PROFILING_ENABLEDhelm/rustfs/templates/configmap.yamlRUSTFS_OBS_PROFILING_ENABLED→RUSTFS_OBS_PROFILING_EXPORT_ENABLED.crates/obs/README.mdRUSTFS_OBS_PROFILING_EXPORT_ENABLEDdefault column fromtrue→false.Migration notes
explicitly set
RUSTFS_OBS_PROFILING_EXPORT_ENABLED=true(or the legacyalias) and ensure a reachable
RUSTFS_OBS_PROFILING_ENDPOINT.profiling.enabled: falsedefault actually take effect (previously it wassilently ignored due to the wrong env var name).
Checklist
make pre-commitImpact
RUSTFS_OBS_PROFILING_ENABLEDis stillaccepted. Only the default changes (off instead of on).
Additional Notes
Verification steps used during development:
All 3 tests pass.