fix: bound the eventlet Glance API worker count by default#725
Merged
Conversation
Below release 2026.1 Glance launches the eventlet glance-api server, whose worker count comes from [DEFAULT] workers in the config. The operator rendered that key only when spec.apiServer.workers was set explicitly; when it was nil the eventlet server fell back to its own default of one worker per host CPU. That fallback ignores the pod's CPU limit, so on a multi-core node Glance forked one worker per core into a container capped at 512Mi and was OOMKilled under concurrent image uploads, crash-looping the API. The uWSGI launch mode (2026.1+) and keystone already pin a bounded process count (DefaultUWSGIProcesses = 2); only the eventlet path lacked the equivalent, even though the spec.apiServer.workers godoc already promised the operator applies a hardcoded default when nil. Add DefaultEventletWorkers = 2 next to DefaultUWSGIProcesses as the single source of truth, and render [DEFAULT] workers in eventlet mode whenever the field is unset. An explicit value still wins, and the uWSGI mode is unchanged: the key stays inert there and is emitted only when set. The worker count -- and thus the memory footprint -- is now deterministic regardless of node size. Regenerate the CRD description for spec.apiServer.workers and sync the Helm chart copy. Cover the launch-mode-conditional rendering with a table test. On-behalf-of: @SAP Assisted-by: Claude:claude-opus-4-8[1m] Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running a sustained image-churn load against a ControlPlane-managed
Glance (release 2025.2) crash-loops the API pod. The container is
OOMKilled(exit 137) within seconds of taking load and never becomesready, so clients see connection resets and 5xx while it flaps.
Root cause
Below release 2026.1 Glance launches the eventlet
glance-apiserver, whose worker count comes from
[DEFAULT] workersin theconfig. The operator rendered that key only when
spec.apiServer.workerswas set explicitly; when it was nil theeventlet server fell back to its own default of one worker per host
CPU. That fallback ignores the pod's CPU limit, so on a multi-core
node Glance forks one worker per core (e.g. 6 workers + master) into a
container capped at 512Mi and is OOMKilled under concurrent uploads.
This is a parity gap: the uWSGI launch mode (2026.1+) and keystone
already pin a bounded process count (
DefaultUWSGIProcesses = 2); onlythe eventlet path lacked the equivalent — even though the
spec.apiServer.workersgodoc already promised the operator applies ahardcoded default when the field is nil.
Fix
DefaultEventletWorkers = 2next toDefaultUWSGIProcessesasthe single source of truth.
[DEFAULT] workersin eventlet mode whenever the field isunset. An explicit value still wins, and the uWSGI mode is unchanged
(the key stays inert there and is emitted only when set).
The worker count — and thus the memory footprint — is now
deterministic regardless of node size.
Testing
TestOperatorDefaults_EventletWorkerscovers thelaunch-mode-conditional rendering (eventlet unset → default,
eventlet explicit wins, uWSGI unset → absent, uWSGI explicit → inert
render).
operators/glancemodule tests,go vet, andgofmtpass.spec.apiServer.workersregenerated and the Helmchart copy synced (
make verify-crd-syncpasses).Verified end-to-end on a local kind ControlPlane: pinning the worker
count drops the pod from 7 processes to 3 and steady-state memory from
the 512Mi ceiling to ~130Mi, ending the OOM crash-loop.
🤖 Generated with Claude Code