Skip to content

fix: bound the eventlet Glance API worker count by default#725

Merged
berendt merged 1 commit into
mainfrom
fix/glance-eventlet-workers-default
Jul 23, 2026
Merged

fix: bound the eventlet Glance API worker count by default#725
berendt merged 1 commit into
mainfrom
fix/glance-eventlet-workers-default

Conversation

@berendt

@berendt berendt commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Problem

Running a sustained image-churn load against a ControlPlane-managed
Glance (release 2025.2) crash-loops the API pod. The container is
OOMKilled (exit 137) within seconds of taking load and never becomes
ready, so clients see connection resets and 5xx while it flaps.

Root cause

Below release 2026.1 Glance launches the eventlet glance-api
server, whose worker count comes from [DEFAULT] workers in the
config. The operator rendered that key only when
spec.apiServer.workers was set explicitly
; when it was nil the
eventlet server fell back to its own default of one worker per host
CPU
. That fallback ignores the pod's CPU limit, so on a multi-core
node Glance forks one worker per core (e.g. 6 workers + master) into a
container capped at 512Mi and is OOMKilled under concurrent uploads.

This is a parity gap: the uWSGI launch mode (2026.1+) and keystone
already pin a bounded process count (DefaultUWSGIProcesses = 2); only
the eventlet path lacked the equivalent — even though the
spec.apiServer.workers godoc already promised the operator applies a
hardcoded default when the field is nil.

Fix

  • Add DefaultEventletWorkers = 2 next to DefaultUWSGIProcesses as
    the single source of truth.
  • Render [DEFAULT] workers in eventlet mode whenever the field is
    unset. An explicit value still wins, and the uWSGI mode is unchanged
    (the key stays inert there and is emitted only when set).

The worker count — and thus the memory footprint — is now
deterministic regardless of node size.

Testing

  • New table test TestOperatorDefaults_EventletWorkers covers the
    launch-mode-conditional rendering (eventlet unset → default,
    eventlet explicit wins, uWSGI unset → absent, uWSGI explicit → inert
    render).
  • Full operators/glance module tests, go vet, and gofmt pass.
  • CRD description for spec.apiServer.workers regenerated and the Helm
    chart copy synced (make verify-crd-sync passes).

Verified end-to-end on a local kind ControlPlane: pinning the worker
count drops the pod from 7 processes to 3 and steady-state memory from
the 512Mi ceiling to ~130Mi, ending the OOM crash-loop.

🤖 Generated with Claude Code

Below release 2026.1 Glance launches the eventlet glance-api server,
whose worker count comes from [DEFAULT] workers in the config. The
operator rendered that key only when spec.apiServer.workers was set
explicitly; when it was nil the eventlet server fell back to its own
default of one worker per host CPU. That fallback ignores the pod's
CPU limit, so on a multi-core node Glance forked one worker per core
into a container capped at 512Mi and was OOMKilled under concurrent
image uploads, crash-looping the API.

The uWSGI launch mode (2026.1+) and keystone already pin a bounded
process count (DefaultUWSGIProcesses = 2); only the eventlet path
lacked the equivalent, even though the spec.apiServer.workers godoc
already promised the operator applies a hardcoded default when nil.

Add DefaultEventletWorkers = 2 next to DefaultUWSGIProcesses as the
single source of truth, and render [DEFAULT] workers in eventlet mode
whenever the field is unset. An explicit value still wins, and the
uWSGI mode is unchanged: the key stays inert there and is emitted only
when set. The worker count -- and thus the memory footprint -- is now
deterministic regardless of node size.

Regenerate the CRD description for spec.apiServer.workers and sync the
Helm chart copy. Cover the launch-mode-conditional rendering with a
table test.

On-behalf-of: @SAP
Assisted-by: Claude:claude-opus-4-8[1m]
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
@berendt
berendt merged commit 2159720 into main Jul 23, 2026
50 checks passed
@berendt
berendt deleted the fix/glance-eventlet-workers-default branch July 23, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant