fix: disable gunicorn worker recycling for pulp-api, raise for pulp-content

[dkliban]

Problem

Gunicorn --max-requests=20 causes workers to recycle after every 20 requests. With only 1 worker per pod, this creates frequent counter resets in Prometheus histogram metrics. When the post-reset counter value coincidentally equals the pre-reset value (common with small counters in the 0–20 range), Prometheus cannot detect the reset — producing SLI ratios > 1 for the APIWriteRequestLatency SLO.

This is the root cause behind the histogram invariant violations. The OTel pipeline fix in #1127 (removing groupbyattrs worker aggregation) reduced the problem but couldn't eliminate it because the counter values are too small for reliable reset detection.

Fix

• pulp-api: Remove --max-requests and --max-requests-jitter entirely. Memory analysis over 24h shows no leak — usage oscillates 825–1004 MiB against a 5120 MiB limit (4x headroom) with no upward drift.
• pulp-content: Raise --max-requests from 20 to 10000 (jitter 500) as a memory safety net. Counter values will reach ~10000 before reset, making coincidental value matches statistically negligible.

Test plan

• Deploy to ephemeral — confirm pulp-api and pulp-content start correctly
• Deploy to stage — monitor memory usage over 24h, confirm no OOM
• Deploy to production — confirm APIWriteRequestLatency SLI stays ≤ 1

🤖 Generated with Claude Code

Summary by Sourcery

Adjust Gunicorn worker recycling settings for pulp-api and pulp-content to improve metric stability and operational behavior.

Build:

• Remove Gunicorn max-requests and jitter configuration from the pulp-api deployment command.
• Increase Gunicorn max-requests and jitter defaults for pulp-content to allow many more requests before workers are recycled.

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Adjusts Gunicorn worker recycling behavior to eliminate frequent worker restarts in pulp-api and significantly reduce them in pulp-content, improving Prometheus histogram metric stability and SLI correctness.

Sequence diagram for API write request and stable histogram metrics

sequenceDiagram
  actor User
  participant Ingress
  participant PulpAPI_Pod
  participant GunicornWorker
  participant OTel_Collector
  participant Prometheus

  User->>Ingress: HTTP API write request
  Ingress->>PulpAPI_Pod: Forward request
  PulpAPI_Pod->>GunicornWorker: Dispatch via pulpcore-api
  GunicornWorker-->>User: Response

  GunicornWorker->>OTel_Collector: Emit APIWriteRequestLatency metrics
  OTel_Collector->>Prometheus: Export histogram time series

  loop Many_requests_without_worker_recycle
    User->>Ingress: Subsequent write request
    Ingress->>PulpAPI_Pod: Forward request
    PulpAPI_Pod->>GunicornWorker: Reuse same long-lived worker
    GunicornWorker-->>User: Response
    GunicornWorker->>OTel_Collector: Increment histogram counters
    OTel_Collector->>Prometheus: Send monotonically increasing counters
  end

Loading

File-Level Changes

Change	Details	Files
Disable Gunicorn max-requests-based worker recycling for pulp-api to prevent frequent worker restarts and metric counter resets.	Update pulp-api container command to remove use of max-requests and max-requests-jitter Gunicorn options, keeping timeout and workers configuration intact. Rely on Kubernetes pod lifecycle and existing timeout settings for pulp-api worker recycling instead of Gunicorn request-count-based restarts.	deploy/clowdapp.yaml
Relax Gunicorn recycling thresholds for pulp-content to keep worker recycling as a memory safety net while avoiding low-count metric reset issues.	Increase PULP_CONTENT_GUNICORN_MAX_REQUESTS default from 20 to 10000 so workers handle far more requests before recycling. Increase PULP_CONTENT_GUNICORN_MAX_REQUESTS_JITTER default from 5 to 500 to maintain staggered recycling across workers.	deploy/clowdapp.yaml
Remove unused pulp-api Gunicorn max-requests parameters from the ClowdApp parameter set to reflect the new configuration model.	Delete PULP_API_GUNICORN_MAX_REQUESTS parameter definition and default value. Delete PULP_API_GUNICORN_MAX_REQUESTS_JITTER parameter definition and default value.	deploy/clowdapp.yaml

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}