Spawning multiple workers and [ERROR] Worker (pid:699) was sent SIGILL!

[ab6995]

#pip list
Package Version

---

absl-py 2.1.0
accelerate 0.22.0
aniso8601 9.0.1
annotated-types 0.6.0
ansi2html 1.9.1
apex 0.1
archspec 0.2.2
arrow 1.3.0
asttokens 2.4.1
attrdict 2.0.1
attrs 23.2.0
awscli 1.32.48
bcrypt 4.1.2
blinker 1.8.2
blis 0.7.10
bokeh 3.3.4
boltons 23.1.1
boto3 1.34.48
botocore 1.34.48
Brotli 1.1.0
cached-property 1.5.2
catalogue 2.0.10
certifi 2024.2.2
cffi 1.16.0
charset-normalizer 3.3.2
click 8.1.7
cloudpathlib 0.16.0
cloudpickle 2.2.1
colorama 0.4.4
comm 0.2.1
conda 23.11.0
conda-libmamba-solver 23.12.0
conda-package-handling 2.2.0
conda_package_streaming 0.9.0
confection 0.1.4
contextlib2 21.6.0
contourpy 1.2.0
cryptography 42.0.4
cycler 0.12.1
cymem 2.0.8
Cython 3.0.8
debugpy 1.8.1
decorator 5.1.1
dill 0.3.8
diskcache 5.6.3
distro 1.8.0
docker 7.0.0
docutils 0.16
einops 0.7.0
exceptiongroup 1.2.0
executing 2.0.1
fastai 2.7.14
fastcore 1.5.29
fastdownload 0.0.7
fastprogress 1.0.3
filelock 3.13.1
flash-attn 2.0.4
Flask 3.0.3
Flask-RESTful 0.3.10
fonttools 4.49.0
fsspec 2024.2.0
gevent 24.2.1
gmpy2 2.1.2
google-pasta 0.2.0
greenlet 3.0.3
grpcio 1.62.0
gunicorn 22.0.0
h5py 3.10.0
idna 3.6
imageio 2.34.0
importlib-metadata 6.11.0
inotify-simple 1.2.1
ipykernel 6.29.2
ipython 8.22.1
itsdangerous 2.2.0
jedi 0.19.1
Jinja2 3.1.3
jmespath 1.0.1
joblib 1.3.2
jsonpatch 1.33
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter_client 8.6.0
jupyter_core 5.7.1
kiwisolver 1.4.5
langcodes 3.3.0
libmambapy 1.5.5
llama_cpp_python 0.2.75
llvmlite 0.42.0
mamba 1.5.5
Markdown 3.5.2
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.8.3
matplotlib-inline 0.1.6
mdurl 0.1.2
menuinst 2.0.1
mpi4py 3.1.5
mpmath 1.3.0
multiprocess 0.70.16
munkres 1.1.4
murmurhash 1.0.10
mypy-extensions 1.0.0
nest_asyncio 1.6.0
networkx 3.2.1
ninja 1.11.1.1
nltk 3.8.1
numba 0.59.0
numpy 1.26.4
nvgpu 0.10.0
opencv-python 4.9.0.80
packaging 23.2
pandas 2.2.1
paramiko 3.4.0
parso 0.8.3
pathos 0.3.2
pathy 0.10.2
patsy 0.5.6
pexpect 4.9.0
pickleshare 0.7.5
pillow 10.2.0
pip 24.0
platformdirs 4.1.0
plotly 5.19.0
pluggy 1.3.0
ply 3.11
pox 0.3.4
ppft 1.7.6.8
preshed 3.0.9
prompt-toolkit 3.0.42
protobuf 3.20.3
psutil 5.9.8
ptyprocess 0.7.0
pure-eval 0.2.2
pyarrow 15.0.0
pyasn1 0.5.1
pybind11 2.11.1
pybind11-global 2.11.1
pycosat 0.6.6
pycparser 2.21
pydantic 2.6.1
pydantic_core 2.16.2
pyfunctional 1.4.3
Pygments 2.17.2
PyNaCl 1.5.0
pynvml 11.5.0
pyOpenSSL 24.0.0
pyparsing 3.1.1
PyQt5 5.15.9
PyQt5-sip 12.12.2
pyre-extensions 0.0.30
PySocks 1.7.1
python-dateutil 2.8.2
pytz 2024.1
PyYAML 6.0.1
pyzmq 25.1.2
referencing 0.33.0
requests 2.31.0
retrying 1.3.4
rich 13.7.0
rpds-py 0.18.0
rsa 4.7.2
ruamel.yaml 0.18.5
ruamel.yaml.clib 0.2.7
s3fs 0.4.2
s3transfer 0.10.0
sagemaker 2.208.0
sagemaker-experiments 0.1.45
sagemaker-pytorch-training 2.8.1
sagemaker-training 4.7.4
schema 0.7.5
scikit-learn 1.4.1.post1
scipy 1.12.0
seaborn 0.13.2
sentence-transformers 2.2.2
setuptools 68.2.2
shap 0.44.0
shellingham 1.5.4
sip 6.7.12
six 1.16.0
slicer 0.0.7
smart-open 5.2.1
smclarify 0.5
smdebug-rulesconfig 1.0.1
smprof 0.3.334
spacy 3.7.2
spacy-legacy 3.0.12
spacy-loggers 1.0.5
srsly 2.4.8
stack-data 0.6.2
statsmodels 0.14.1
sympy 1.12
tabulate 0.9.0
tblib 2.0.0
tenacity 8.2.3
tensorboard 2.16.2
tensorboard-data-server 0.7.2
termcolor 2.4.0
thinc 8.2.2
threadpoolctl 3.3.0
toml 0.10.2
tomli 2.0.1
torch 2.2.0
torchaudio 2.2.0
torchdata 0.7.1
torchtext 0.17.0
torchtnt 0.2.2
torchvision 0.17.0
tornado 6.4
tqdm 4.66.1
traitlets 5.14.1
transformer-engine 0.12.0+170797
triton 2.1.0
truststore 0.8.0
typer 0.9.0
types-python-dateutil 2.9.0.20240316
typing_extensions 4.9.0
typing-inspect 0.9.0
tzdata 2024.1
unicodedata2 15.1.0
urllib3 1.26.18
wasabi 1.1.2
wcwidth 0.2.13
weasel 0.3.4
Werkzeug 3.0.1
wheel 0.42.0
xyzservices 2023.10.1
zipp 3.17.0
zope.event 5.0
zope.interface 6.2
zstandard 0.22.0

Server is spawning workers and receiving [ERROR] Worker (pid:699) was sent SIGILL! Below is the gunicorn config:

[2024-05-21 18:21:57 +0000] [11] [DEBUG] Current configuration:
config: ./gunicorn.conf.py
wsgi_app: None
bind: ['unix:/tmp/gunicorn.sock']
backlog: 2048
workers: 2
worker_class: gevent
threads: 1
worker_connections: 1000
max_requests: 0
max_requests_jitter: 0
timeout: 1800
graceful_timeout: 30
keepalive: 2
limit_request_line: 4094
limit_request_fields: 100
limit_request_field_size: 8190
reload: False
reload_engine: auto
reload_extra_files: []
spew: False
check_config: False
print_config: False
preload_app: False
sendfile: None
reuse_port: False
chdir: /opt/program
daemon: False
raw_env: []
pidfile: None
worker_tmp_dir: None
user: 0
group: 0
umask: 0
initgroups: False
tmp_upload_dir: None
secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
accesslog: None
disable_redirect_access_to_syslog: False
access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
errorlog: -
loglevel: debug
capture_output: False
logger_class: gunicorn.glogging.Logger
logconfig: None
logconfig_dict:
{}

logconfig_json: None
syslog_addr: udp://localhost:514
syslog: False
syslog_prefix: None
syslog_facility: user
enable_stdio_inheritance: False
statsd_host: None
dogstatsd_tags:
statsd_prefix:
proc_name: None
default_proc_name: wsgi:app
pythonpath: None
paste: None
on_starting: <function OnStarting.on_starting at 0x7f11c9b78c10>
on_reload: <function OnReload.on_reload at 0x7f11c9b78d30>
when_ready: <function WhenReady.when_ready at 0x7f11c9b78e50>
pre_fork: <function Prefork.pre_fork at 0x7f11c9b78f70>
post_fork: <function Postfork.post_fork at 0x7f11c9b79090>
post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f11c9b791b0>
worker_int: <function WorkerInt.worker_int at 0x7f11c9b792d0>
worker_abort: <function WorkerAbort.worker_abort at 0x7f11c9b793f0>
pre_exec: <function PreExec.pre_exec at 0x7f11c9b79510>
pre_request: <function PreRequest.pre_request at 0x7f11c9b79630>
post_request: <function PostRequest.post_request at 0x7f11c9b796c0>
child_exit: <function ChildExit.child_exit at 0x7f11c9b797e0>
worker_exit: <function WorkerExit.worker_exit at 0x7f11c9b79900>
nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f11c9b79a20>
on_exit: <function OnExit.on_exit at 0x7f11c9b79b40>
ssl_context: <function NewSSLContext.ssl_context at 0x7f11c9b79c60>
proxy_protocol: False
proxy_allow_ips: ['127.0.0.1']
keyfile: None
certfile: None
ssl_version: 2
cert_reqs: 0
ca_certs: None
suppress_ragged_eofs: True
do_handshake_on_connect: False
ciphers: None
raw_paste_global_conf: []
strip_header_spaces: False
permit_unconventional_http_method: False
permit_unconventional_http_version: False
casefold_http_method: False
header_map: drop
tolerate_dangerous_framing: False

Any idea why i am seeing this gunicorn behaviour?

[pajod]

I see nothing in your report pointing to the trigger. Bisect or attach a debugger to find where this is from. Try a worker class other than gevent first, as that is a likely candidate for unexpected interactions.

If one of your imported python (extension) libraries does funny stuff and gets killed by the system, there is little gunicorn can do about it.

[benoitc]

no activity since awhile. closing feel free to create a new ticket if needed.