Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes #36438 - configure API request limit to avoid memory leaks #289

Merged
merged 1 commit into from May 25, 2023
Merged

Fixes #36438 - configure API request limit to avoid memory leaks #289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 8 additions & 0 deletions manifests/init.pp
View file
Open in desktop
Expand Up @@ -162,6 +162,12 @@
# @param api_service_worker_timeout
# Timeout in seconds of the pulpcore-api gunicorn workers.
#
# @param api_service_worker_max_requests
# Number of requests a gunicorn worker will process before restarting.
#
# @param api_service_worker_max_requests_jitter
# Jitter to add to the max_requests setting of the gunicorn worker.
#
# @param api_client_auth_cn_map
# Mapping of certificate common name and Pulp user to authenticate to Pulp API.
#
Expand Down Expand Up @@ -233,6 +239,8 @@
Integer[0] $api_service_worker_count = min(4, $facts['processors']['count']) + 1,
Integer[0] $content_service_worker_timeout = 90,
Integer[0] $api_service_worker_timeout = 90,
Integer[0] $api_service_worker_max_requests = 50,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@evgeni We discovered the "real" root cause now, so we can relax this number significantly. I would back it off to a much higher value, like say 1000 for the limit and 200 for the jitter.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a link to info about the real root cause?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the discussion near the end on the BZ https://bugzilla.redhat.com/show_bug.cgi?id=2122872

Integer[0] $api_service_worker_max_requests_jitter = 30,
Hash[String[1], String[1]] $api_client_auth_cn_map = {},
Boolean $cache_enabled = false,
Optional[Variant[Integer[1], Enum['None']]] $cache_expires_ttl = undef,
Expand Down
18 changes: 18 additions & 0 deletions spec/classes/pulpcore_spec.rb
View file
Open in desktop
Expand Up @@ -150,6 +150,8 @@
is_expected.to contain_pulpcore__socket_service('pulpcore-api')
is_expected.to contain_systemd__unit_file('pulpcore-api.socket')
is_expected.to contain_systemd__unit_file('pulpcore-api.service').with_content(%r{--workers 2})
.with_content(%r{--max-requests 50})
.with_content(%r{--max-requests-jitter 30})
is_expected.to contain_file('/etc/systemd/system/pulpcore-api.socket').that_comes_before('Service[pulpcore-api.service]')
is_expected.to contain_pulpcore__socket_service('pulpcore-content')
is_expected.to contain_systemd__unit_file('pulpcore-content.socket')
Expand All @@ -162,6 +164,22 @@
end
end

context 'with changed max-requests' do
let :params do
{
api_service_worker_max_requests: 100,
api_service_worker_max_requests_jitter: 10
}
end

it do
is_expected.to compile.with_all_deps
is_expected.to contain_systemd__unit_file('pulpcore-api.service')
.with_content(%r{--max-requests 100})
.with_content(%r{--max-requests-jitter 10})
end
end

context 'with allowed import paths' do
let :params do
{
Expand Down
2 changes: 2 additions & 0 deletions templates/pulpcore-api.service.erb
View file
Open in desktop
Expand Up @@ -15,6 +15,8 @@ ExecStart=/usr/libexec/pulpcore/gunicorn pulpcore.app.wsgi:application \
--preload \
--timeout <%= scope['pulpcore::api_service_worker_timeout'] %> \
--workers <%= scope['pulpcore::api_service_worker_count'] %> \
--max-requests <%= scope['pulpcore::api_service_worker_max_requests'] %> \
--max-requests-jitter <%= scope['pulpcore::api_service_worker_max_requests_jitter'] %> \
--access-logfile - \
--access-logformat 'pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'
ExecReload=/bin/kill -s HUP $MAINPID
Expand Down