Error on webserver: worker was sent SIGKILL

[AlexandreGCastor]

Hello,

On airflow 2.6.2 deployed via helm chart 1.10.0, I have the following error on webserver log: [2023-08-02 07:45:10 +0000] [13] [ERROR] Worker (pid:23) was sent SIGKILL! Perhaps out of memory?

I have remote logging on gcs and for live logging the webserver request the airflow worker. Sometimes logs are slow to open, may timeout or 404 on worker (and bucket).

Airflow webserver config:
expose_config: "True"
worker_refresh_interval: 600
reload_on_plugin_change: "True"
expose_stacktrace: "True"
log_fetch_delay_sec: 5
log_fetch_timeout_sec: 20
web_server_worker_timeout: 300
worker_refresh_batch_size: 0
workers: 3

resources:
requests:
cpu: '1'
limits:
memory: 8Gi
ephemeral-storage: 512Mi

Airflow webserver memory consumption is below 25%

I would like to know how to fix this error.

Thanks in advance

Alexandre Gué,
Lead SRE, Castordoc

[potiuk]

I think you should look at the logs of your K8S. For some reason it's killing webserver. And you will not find it in the logs of airlfow because if teh deployment is killing it with SIGKILL, then the webserver has no chance to react.

You need to (as with any other k8s app) look in your k8s logs for reason why you get SIGKILL. It could be memory, other resources, maybe the liveness probe is doing it (then you need to check liveness probe logs). Generally this is usually indicatin of some resource problem. It's part of the deployment management to make sure the resources are fine-tuned for the particular deployement.

[AlexandreGCastor]

Thanks, my webserver is not killed, only a worker. I see no restart and pod still logs long time after.

But checking in previous logs, I find that:
[2023-08-02 09:23:23 +0000] [14] [CRITICAL] WORKER TIMEOUT (pid:23)

So the sigKill is a consequence of the timeout which is 5min !

[duongphannamhung]

Hi @AlexandreGCastor , have you found the way to fix it, in my case the worker continuously killed. My webserver still works with low memory but it's not work properly: slow, missing dags, ...

[PuneeshMotwani007]

I have the same Issue. @AlexandreGCastor @duongphannamhung - Did you find some resolution?

[duongphannamhung]

Down and reup make it works, it may because of logs are missing after you delete manually dag

[PuneeshMotwani007]

I now recycle the gunicorn workers. This is what I added in the webserver container. This is how you can configure gunicorn settings.
https://docs.gunicorn.org/en/stable/settings.html

env:
            - name: GUNICORN_CMD_ARGS
              value: >-
                --max-requests 20 --max-requests-jitter 3 --worker-tmp-dir
                /dev/shm
            - name: AIRFLOW__WEBSERVER__WORKERS
              value: '5'

Offcorse, These values, max_request, jitters, worker number, depends on your setup. But it looks promising for me. We haven't seen any crashes since yesterday.