Configure automatic gunicorn worker restart in the pulpcore-api service configuration

[dralley]

This will help to address https://bugzilla.redhat.com/show_bug.cgi?id=2122872

Gunicorn provides --max-requests and --max-requests-jitter settings which will restart the web worker process after handling some number of requests. This will help to eliminate any memory leaks which may be present.

I advise to set

--max-requests 50 \
--max-requests-jitter 30 \

, which will restart workers after handling some random number of requests between 50 and 80.

https://github.com/theforeman/puppet-pulpcore/blob/master/templates/pulpcore-api.service.erb

[evgeni]

this would apply to API only, right? the content app, while also being gunicorn, shouldn't leak?

[dralley]

Yes, that's correct.

I'm testing --max-requests-jitter currently, and actually it does not appear to necessarily be working? --max-requests 80 visibly does work well, but I cannot tell that --max-requests-jitter is doing anything.

[evgeni]

do you call it with --max-requests-jitter J --max-request X or just --max-requests-jitter J?

https://docs.gunicorn.org/en/latest/settings.html#max-requests-jitter is not exactly clear in the wording, but I think that the restarts happen after X+rand(0, J) requests if you provide both, and if you don't provide --max-requests the jitter is just ignored (vs executed at rand(0,J))

[dralley]

Ah, you are correct, you have to provide both.

[evgeni]

How did you get those numbers? They feel very low, but I can't really tell why.

[ekohl]

They do feel very low, because if you serve just 10 requests/second you'll have a restart every 5 seconds. In really doubt you'll have memory leaks in such a short amount of time. You may be trading in better memory consumption for higher CPU usage. I'm not sure which one I'd rather have.

When you have these settings I often see values of 1000 or 10000 requests to start with and then tune them further.

[dralley]

They are pretty low, but unfortunately at the moment it seems we're leaking memory quite quickly during repository index operations.

The API server is easily the least sensitive to this sort of thing, so I think the low limits are OK, though of course we do want to be able to raise them later.

If you look at the BZ it's pretty clear we should prefer the lower memory usage at the moment.

[evgeni]

moving to https://projects.theforeman.org/issues/36438