Describe the bug
The current OpenCHAMI cloud-init server (v1.4.5) periodically refreshes its in-memory SMD cache. During this refresh, PopulateNodes() acquires the global nodesMutex write lock before making any SMD requests and holds that lock for the duration of the refresh.
While the lock is held, the refresh performs:
- A request to retrieve all Ethernet interfaces from SMD.
- A separate membership lookup for every cached node.
- Rebuilding of the reverse lookup indexes.
Because the lock is held across all network I/O, any request that needs to read or modify the cache must wait for the entire refresh to complete.
The /wg-init endpoint is directly affected because it must:
- Resolve the requesting node from its IP address (IDfromIP()), which requires a read lock.
- Store the allocated WireGuard IP (AddWGIP()), which requires the write lock.
As a result, if a /wg-init request arrives while a cache refresh is in progress, it blocks until the refresh completes. Under normal conditions this simply increases request latency. However, if the refresh is slow (for example due to SMD latency or large cluster size), the request can exceed the reverse proxy timeout and fail with an HTTP 504 Gateway Timeout.
Steps to reproduce
populate SMD with lot's of components, cache refresh takes a long time
Relevant log output
Code of Conduct
Describe the bug
The current OpenCHAMI cloud-init server (v1.4.5) periodically refreshes its in-memory SMD cache. During this refresh, PopulateNodes() acquires the global nodesMutex write lock before making any SMD requests and holds that lock for the duration of the refresh.
While the lock is held, the refresh performs:
Because the lock is held across all network I/O, any request that needs to read or modify the cache must wait for the entire refresh to complete.
The /wg-init endpoint is directly affected because it must:
As a result, if a /wg-init request arrives while a cache refresh is in progress, it blocks until the refresh completes. Under normal conditions this simply increases request latency. However, if the refresh is slow (for example due to SMD latency or large cluster size), the request can exceed the reverse proxy timeout and fail with an HTTP 504 Gateway Timeout.
Steps to reproduce
populate SMD with lot's of components, cache refresh takes a long time
Relevant log output
Code of Conduct