feat(helm): Added env variable SERVER_WORKER_AMOUNT

[rathberm]

SUMMARY

I added the environment variable SERVER_WORKER_AMOUNT to the helm values file. It is commented out so it won't affect any running instances. We had severe performance problems caused by superset workers timing out. This was solved and the general Superset performance was improved by increasing the Server Worker Amount. Although this variable can have a huge impact on performance it is never mentioned, neither in the docs nor in the helm chart.
The standard value of 1 is set in run-server.sh.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

Change the variable, at the start of the containers you can see the workers starting up in the logs.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[danielhass]

From my point of view I would also appreciate the consolidation of gunicorn related environment variables into a common prefix as some of them already start with GUNICORN_*. This makes the configuration interface more consistent and easier to read.

However this is a more breaking change as a switch from SERVER_WORKER_AMOUNT to e.g. GUNICORN_WORKER_AMOUNT might break existing settings and deployments.

[dpgaspar]

would be great if you could add the rest of the variables: SERVER_THREADS_AMOUNT, GUNICORN_KEEPALIVE, WORKER_MAX_REQUESTS , WORKER_MAX_REQUESTS_JITTER, SERVER_LIMIT_REQUEST_LINE, SERVER_LIMIT_REQUEST_FIELD_SIZE

modify the link to just: https://docs.gunicorn.org/en/stable/design.html#how-many-workers

[craig-rueda]

I can't find SERVER_WORKER_AMOUNT anywhere in the Gunicorn docs

[rathberm]

@craig-rueda as you can see in run-server.sh the variable SERVER_WORKER_AMOUNT is introduced by Superset und used to set the flag --workers when starting gunicorn. It's not a gunicorn variable.

[rathberm]

@dpgaspar I added the variables and a link to the general documentation where users can find the settings.

[dpgaspar]

From my point of view I would also appreciate the consolidation of gunicorn related environment variables into a common prefix as some of them already start with GUNICORN_*. This makes the configuration interface more consistent and easier to read.

However this is a more breaking change as a switch from SERVER_WORKER_AMOUNT to e.g. GUNICORN_WORKER_AMOUNT might break existing settings and deployments.

Totally agree, this can probably be done without breaking anything

[dpgaspar]

LGTM, would be great also if you could add a link to where these are used run-server.sh since their names are a bit confusing (optional)

[danielhass]

Totally agree, this can probably be done without breaking anything

@dpgaspar what would be your idea for implementing this without breaking the existing configuration interface? A change of all variables to a common GUNICORN_* prefix might break configurations that users currently applied via the old environment variables.

Offering backward compatibility for these options from my point of view would require a lot of logic checking for the different variants and defining a set of precedence rules for the different options.

What was your idea?

[dpgaspar]

Totally agree, this can probably be done without breaking anything

@dpgaspar what would be your idea for implementing this without breaking the existing configuration interface? A change of all variables to a common GUNICORN_* prefix might break configurations that users currently applied via the old environment variables.

Offering backward compatibility for these options from my point of view would require a lot of logic checking for the different variants and defining a set of precedence rules for the different options.

What was your idea?

Could be DRYer but something similar to:

if [[ -z "${SERVER_WORKER_AMOUNT}" ]]; then
  GUNICORN_WORKERS=${GUNICORN_WORKERS:-1}
elif [[ -z "${GUNICORN_WORKERS}" ]]; then
  GUNICORN_WORKERS=${SERVER_WORKER_AMOUNT}
fi

echo ${GUNICORN_WORKERS}

?