Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(docker): Make Gunicorn Keepalive Adjustable #20348

Merged
merged 1 commit into from
Jun 10, 2022
Merged

fix(docker): Make Gunicorn Keepalive Adjustable #20348

merged 1 commit into from
Jun 10, 2022

Conversation

mdeshmu
Copy link
Contributor

@mdeshmu mdeshmu commented Jun 10, 2022

SUMMARY

This PR helps to solve 16742

We were getting frequent 502 errors during loading of charts in dashboard forcing us to refresh the page.

This is how traffic flows in our Superset setup: AWS ALB --> Gunicorn --> Superset App
We are using official Superset docker image.

I modified timeout settings i.e. GUNICORN_TIMEOUT and SUPERSET_WEBSERVER_TIMEOUT but it didn't resolve the problem.
I have already increased SERVER_WORKER_AMOUNT to 8 and using default thread value of 20 but it didn't resolve the issue.
My ECS task cpu/memory are underutilized so scaling is not a cause of the problem.

Finally, I saw this blog which says solution is to keep Gunicorn --keep-alive more than alb idle timeout: https://www.tessian.com/blog/how-to-fix-http-502-errors/

Default value for --keep-alive is 2, please see https://docs.gunicorn.org/en/stable/settings.html#keepalive

Even Gunicorn's official documentation here https://docs.gunicorn.org/en/stable/settings.html#keepalive says:
"Generally set in the 1-5 seconds range for servers with direct connection to the client (e.g. when you don’t have separate load balancer). When gunicorn is deployed behind a load balancer, it often makes sense to set this to a higher value."

But run-server.sh in official docker image doesn't has an option for setting Gunicorn's --keep-alive to a custom value.
So I have done improvements in run-server.sh to allow this customization keeping the default value as 2 taken from Gunicorn's default.

I hope this helps people with similar setup as mine.

Note: This change is non-disruptive, no end user will be impacted unless they turn on settings.
You can also refer https://apache-superset.slack.com/archives/C014LS99C1K/p1653678959786229

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

133964881-01a3075a-1bf4-4079-ac99-85df6670c4b3
133964682-578da6b7-859b-4d61-93fc-eff21d6a0fd2

TESTING INSTRUCTIONS

Set GUNICORN_KEEPALIVE to more than alb timeout in your docker containers environment and see issue disappear.

image

image

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@nytai nytai merged commit 86368dd into apache:master Jun 10, 2022
@mdeshmu mdeshmu deleted the fix-make-gunicorn-keepalive-adjustable branch June 11, 2022 05:17
@mdeshmu
Copy link
Contributor Author

mdeshmu commented Jun 11, 2022

Thanks @nytai

michael-s-molina pushed a commit that referenced this pull request Aug 30, 2022
Co-authored-by: Multazim Deshmukh <multazim.deshmukh@morningstar.com>
(cherry picked from commit 86368dd)
@mistercrunch mistercrunch added 🍒 1.5.2 🍒 1.5.3 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.0.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/XS 🍒 1.5.2 🍒 1.5.3 🚢 2.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants