RPM repository modify/ endpoint blocks synchronously for ~110 seconds with large payloads (~43K packages)

[dkliban]

Summary

The modify/ endpoint for RPM repositories blocks synchronously for ~108–113 seconds when processing large payloads (~43K packages). This causes upstream proxies with shorter timeouts (e.g. Akamai's 30s origin timeout) to return 504 Gateway Timeout errors to clients, even though pulp-api eventually completes the request successfully.

Expected Behavior

The modify/ endpoint should dispatch work to a background task and return 202 Accepted in milliseconds, regardless of payload size.

Actual Behavior

With a ~108–113 KB request body containing ~43K packages, the endpoint takes ~110 seconds to respond. The response is eventually 202, but any proxy or client with a timeout shorter than ~110 seconds will give up before receiving it.

Evidence

From pulp-api access logs (%(M)s = request duration in milliseconds):

POST /api/pulp/public-copr/api/v3/repositories/rpm/rpm/019cfcd6-0252-79c5-ab12-87c22cde04b4/modify/ 202 - "crc-pulp-client" 113374
POST /api/pulp/public-copr/api/v3/repositories/rpm/rpm/019cfcd6-0252-79c5-ab12-87c22cde04b4/modify/ 202 - "crc-pulp-client" 108566
POST /api/pulp/public-copr/api/v3/repositories/rpm/rpm/019cfcd6-0252-79c5-ab12-87c22cde04b4/modify/ 202 - "crc-pulp-client" 108235

Akamai CDN logs show repeated 504s on the same endpoint with turnAroundTimeMSec of ~30 seconds (Akamai's hard origin timeout):

{"reqMethod":"POST","reqPath":"api/pulp/public-copr/api/v3/repositories/rpm/rpm/019cfcd6-0252-79c5-ab12-87c22cde04b4/modify/","statusCode":"504","turnAroundTimeMSec":"30411"}
{"reqMethod":"POST","reqPath":"api/pulp/public-copr/api/v3/repositories/rpm/rpm/019cfcd6-0252-79c5-ab12-87c22cde04b4/modify/","statusCode":"504","turnAroundTimeMSec":"30188"}
{"reqMethod":"POST","reqPath":"api/pulp/public-copr/api/v3/repositories/rpm/rpm/019cfcd6-0252-79c5-ab12-87c22cde04b4/modify/","statusCode":"504","turnAroundTimeMSec":"30092"}

Timeline (2026-03-18)

The pattern repeats consistently — every scheduled attempt times out at Akamai, and a retry ~1m48s later succeeds (because pulp-api has finished by then):

Time (UTC)	Status	Notes
12:23:07	504	First observed timeout
13:46:09	504	Akamai times out at 30s
13:48:03	202	Retry succeeds, pulp-api took ~113s
13:56:55	504	Akamai times out at 30s
13:58:43	202	Retry succeeds, pulp-api took ~108s
14:07:45	504	Akamai times out at 30s
14:09:33	202	Retry succeeds, pulp-api took ~108s

Suspected Root Cause

The modify/ endpoint is likely doing expensive work synchronously (e.g. validating or resolving ~43K package references against the database) before dispatching the background task. This work should either be moved into the task itself or optimized to avoid O(n) database lookups on large package lists.

Environment

• Endpoint: RPM repository modify/
• Payload: ~108–113 KB, ~43K packages
• pulp-api workers: 5
• Gunicorn timeout: 1800s
• Client: crc-pulp-client

[dralley]

Use the pyinstrument diagnostic and provide the report please

[dkliban]

The user is submitting a list of pulp_href strings. The issue does not occur when a list of prn strings is submitted.