Skip to content

perf: coordinate per-BMC collectors to avoid overlapping Redfish request bursts #4751

Description

@jayzhudev

Is this a new feature, an enhancement, or a change to existing functionality?

Enhancement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

health starts several independent collectors for each switch BMC. Their schedules could overlap, producing repeated and synchronized requests against resource-constrained BMC HTTP services.

Observed behavior

All configured switch BMC endpoints enable:

[collectors.sensors]
enabled = true
sensor_fetch_interval = "1m"
sensor_fetch_concurrency = 1

[collectors.firmware]
enabled = true
firmware_refresh_interval = "5m"

[collectors.leak_detector]
enabled = true
poll_interval = "1m"

[collectors.logs]
enabled = true
mode = "sse"

Sensors also enable entity discovery, which defaults to a 5-minute interval.

For switch BMC endpoints, health overrides the configured SSE log mode and uses periodic collection with a five-minute default interval. As a result:

  • Sensors and leak detection run every minute.
  • Discovery, firmware, and log collection run every 5 minutes.
  • The 5-minute collectors can occasionally align with the 1-minute collectors and create request bursts.
  • The per-endpoint request lock serializes requests but does not coalesce or stagger them.

Expected behavior

health should coordinate collectors sharing a BMC so required data is collected without synchronized request bursts or repeated reads of the same Redfish resources.

Feature Description

  • Measure Redfish requests by endpoint, collector, and resource path.
  • Identify requests that can share discovery or cached results.
  • Stagger or add deterministic jitter to collector schedules per endpoint.
  • Preserve per-endpoint request serialization.

Describe your ideal solution

No response

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

  • I agree to follow NVIDIA Infra Controller's Code of Conduct
  • I have searched the open feature requests and have found no duplicates for this feature request

Metadata

Metadata

Assignees

Labels

featureFeature (deprecated - use issue type, but it's needed for reporting now)rack health

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions