Skip to content

Fix gunicorn "Control server error" on kubernetes #7591

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pedro-psb wants to merge 1 commit into
base: main
Choose a base branch
from
+5 −0
Open

Fix gunicorn "Control server error" on kubernetes #7591

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES/7574.bugfix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Fixed gunicorn control socket being placed on persistent location.
This could cause `Control server error: [Errno 13]` errors on kubernetes environements on pod restarts.
3 changes: 3 additions & 0 deletions pulpcore/content/entrypoint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ def load_app_specific_config(self):
)
self.set_option("default_proc_name", "pulpcore-content", enforced=True)
self.set_option("worker_class", worker_class, enforced=True)
# On k8s, the default location may persist across restarts and cause permission errors
# See: <https://github.com/pulp/pulpcore/issues/7574>
Comment on lines +18 to +19
Copy link
Copy Markdown
Member

@mdellweg mdellweg Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the default location?
This file belongs in /run/ somewhere.

This directory contains system information data describing the system since it was booted. Files under this directory must be cleared (removed or truncated as appropriate) at the beginning of the boot process.
[...]
System programs that maintain transient UNIX-domain sockets must place them in this [/run] directory or an appropriate subdirectory as outlined above.

https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s15.html

Copy link
Copy Markdown
Member Author

@pedro-psb pedro-psb Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It defaults to the current directory: https://gunicorn.org/guides/gunicornc/#start-gunicorn-with-control-socket
I wonder why.

Copy link
Copy Markdown
Member Author

@pedro-psb pedro-psb Apr 15, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, the code apparently says something else:
https://github.com/benoitc/gunicorn/blob/9aa54703f4950818aed538dbee9578e868375cc9/gunicorn/config.py#L3138-L3146

Copy link
Copy Markdown
Member Author

@pedro-psb pedro-psb Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, it was changed in 25.2: benoitc/gunicorn@0ad47db

I guess we should not do anything, then.
This change was to improve on gunicorn's default (of version 25.1), but they have improved by themselves.

Copy link
Copy Markdown
Member Author

@pedro-psb pedro-psb Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or do you think it's still worth it, to account for the case 25.1 is installed?

Copy link
Copy Markdown
Member

@mdellweg mdellweg Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what I understand is that gunicorn tries XDG_RUNTIME_DIR first and falls back to HOME.
I would claim that the variable XDG_RUNTIME_DIR should have been set. Not sure if the os in the container or the container runtime is to blame, but the default gunicorn behaviour seems sound to me and your change makes that unnecessarily rigid.
We should probably propagate the option instead so it stays possible to overwrite it.

Copy link
Copy Markdown
Member Author

@pedro-psb pedro-psb Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably propagate the option instead so it stays possible to overwrite it.

That sounds good.

Copy link
Copy Markdown
Member Author

@pedro-psb pedro-psb Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what I understand is that gunicorn tries XDG_RUNTIME_DIR first and falls back to HOME.

The first release of the feature the default was the current directory (whathever that was...). In the following Y they've changed to this, which I agree is sane.

self.set_option("control_socket", "/tmp/pulpcore-content.ctl")

def load(self):
import pulpcore.content
Expand Down
Loading