Control server error: [Errno 13] Permission denied 0 PermissionError(13, 'Permission denied')

[dkliban]

Version
"core": "3.106.0",

When pulpcore-content redeploys we get the following error in our logs:

Control server error: [Errno 13] Permission denied 0 PermissionError(13, 'Permission denied')

[pedro-psb]

This doesn't look directly related to any pulp changes, but rather to this new gunicorn feature. Basically it looks like it crates a file, and if some volume is persisted on worker/pod restart, the new process won't have the rights to access it (see claude's words on it below). If that's correct, this isn't a bug in Pulp, but we could pick an ephemeral path for this unix-socket file by default (e.g, /tmp, as the doc's itself put as an example of custom path).

Claude's words after some back and forth

The "Control server error: [Errno 13] Permission denied" is caused by gunicorn's control socket feature (introduced in gunicorn 23.x). By default, gunicorn creates gunicorn.ctl relative to its working directory. Since pulpcore-content sets its working
directory to WORKING_DIRECTORY (/var/lib/pulp/tmp by default), which is on a shared PVC in k8s setups, the socket lands at /var/lib/pulp/tmp/gunicorn.ctl.

During a rolling update:

1. Old pod is running, gunicorn.ctl exists on the PVC owned by that pod's UID
2. New pod starts, gunicorn tries to create gunicorn.ctl at the same path
3. It can't unlink or overwrite the existing file → Permission denied

This appeared on upgrade because gunicorn 23.x was pulled in — the constraint >=22.0,<25.2.0 allows it, and the 3.105 → 3.106 deployment likely resolved to a newer gunicorn that introduced this feature.

Fix: Set an explicit absolute path in /tmp in PulpcoreContentApplication.load_app_specific_config():

self.set_option("control_socket", "/tmp/pulpcore-content.ctl")

gunicorn.ctl is a Unix socket — an IPC mechanism, not persistent data. It has no meaning outside of a running gunicorn process and is recreated on every startup. /tmp is the conventional location for Unix sockets on Linux (PostgreSQL does the same with
/tmp/.s.PGSQL.5432). The fact that /tmp is ephemeral is a feature here, not a limitation: the root cause of this bug is precisely that /var/lib/pulp/tmp is persistent, causing stale socket files to outlive the pods that created them.

We deliberately avoid enforced=True so that users on other deployment models (bare metal, quadlet) can override the path in their gunicorn.conf.py if needed.