fix(docker): Make Gunicorn max_requests and max_requests_jitter adjustable

[mdeshmu]

SUMMARY

This PR allows to adjust Gunicorn max_requests and max_requests_jitter which allows to restart the Gunicorn worker process periodically.

This is a simple method to help limit the damage of memory leaks like one reported in 15132.

For more details, Please refer to: https://docs.gunicorn.org/en/stable/settings.html#max-requests

Default values are set to zero to match gunicorn's default i.e. automatic worker restarts are disabled by default.

Note: This change is non-disruptive, no end user will be impacted unless they turn on settings.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

Set the values like:
gunicorn --max-requests 1024 --max-requests-jitter 100 ....

ADDITIONAL INFORMATION

• Has associated issue: 15132
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[mdeshmu]

@zhaoyongjie please review and approve.

[zhaoyongjie]

lgtm, it's a useful workaround for memory leaks.

[zac-roberts]

Seeing this in logs during docker compose (posting in case related):

superset_app | gunicorn: error: argument --max-requests: expected one argument

[zhaoyongjie]

Seeing this in logs during docker compose (posting in case related):

superset_app | gunicorn: error: argument --max-requests: expected one argument

getting default value should be ${parameter:-word} instead of ${parameter:word}. I will push the fix.

[mdeshmu]

sorry. my bad, i missed the hyphen. Thanks for reporting @z6cr0b3rts and thanks for the quick fix @zhaoyongjie.

[evkuzin]

What's the timeline on the docker hub image? How long would it take to release a new image? The one published is broken now. @zhaoyongjie

[mdeshmu]

@evkuzin updated build is available now.

[low-on-mana]

We have tried this solution sometime back with custom docker image, but faced lot of intermittent http504s.
Our setup to serve requests is aws alb -> superset pods in k8.
Please share your findings as well @mdeshmu @zhaoyongjie @evkuzin

[mdeshmu]

@low-on-mana We have it enabled for a while now but I don’t see any 504's.

For you, Is it 502 bad gateway or 504 gateway time out?

[low-on-mana]

504 gateway timeout @mdeshmu .

[mdeshmu]

@low-on-mana Please make sure you have appropriate jitter parameter value as it is needed to avoid simultaneous restarts of workers.

Another possibility, Maybe your queries are occasionally taking longer time. Please make sure you are having large enough timeout values at all places.

This is what i have,

ALB idle timeout (120s) >
GUNICORN_TIMEOUT (115s) =
SUPERSET_WEBSERVER_TIMEOUT

I also have ecs task (gunicorn + superset) directly under alb and was intermittently getting 502 bad gateway because of low keep alive of gunicorn. Then i set it below and all errors were gone.

GUNICORN_KEEPALIVE (125s) >
ALB idle timeout (120s)

[low-on-mana]

@mdeshmu Thanks for sharing this, I will check these configuration out.