clusterfuzz service crashing on fuzzbots

[nguyendon]

I'm running into an issue on our deploy of v2.6.0 where the fuzz bots and the clusterfuzz service are created but immediately crash.

Apr 29 09:12:16 clusterfuzz-linux-21mc bash[1463]: + sudo -E -H -u clusterfuzz bash -c 'PATH='\'':/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-15-openjdk-amd64/bin'\'' PYTHONPATH='\''/mnt/scratch0/clusterfuzz/src'\'' LD_L
IBRARY_PATH='\'''\'' python3.7 /mnt/scratch0/clusterfuzz/src/python/bot/startup/run.py'
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: Traceback (most recent call last):
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/python/bot/startup/run.py", line 29, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     from clusterfuzz._internal.base.untrusted import untrusted_noop
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/base/untrusted.py", line 18, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     from clusterfuzz._internal.base import tasks
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/base/tasks.py", line 22, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     from clusterfuzz._internal.base import external_tasks
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/base/external_tasks.py", line 16, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     from clusterfuzz._internal.datastore import data_handler
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/datastore/data_handler.py", line 30, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     from google.cloud import ndb
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/__init__.py", line 28, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     from google.cloud.ndb.client import Client
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 18, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     import grpc
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:   File "/mnt/scratch0/clusterfuzz/src/third_party/grpc/__init__.py", line 23, in <module>
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]:     from grpc._cython import cygrpc as _cygrpc
Apr 29 09:12:17 clusterfuzz-linux-21mc bash[1463]: ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/mnt/scratch0/clusterfuzz/src/third_party/grpc/_cython/__init__.py)
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Main process exited, code=exited, status=1/FAILURE
Apr 29 09:12:18 clusterfuzz-linux-21mc docker[1908]: Error response from daemon: No such container: clusterfuzz
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Control process exited, code=exited, status=1/FAILURE
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Failed with result 'exit-code'.
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: clusterfuzz.service: Scheduled restart job, restart counter is at 2.
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: Stopped clusterfuzz.service.
Apr 29 09:12:18 clusterfuzz-linux-21mc systemd[1]: Starting clusterfuzz.service...


ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/mnt/scratch0/clusterfuzz/src/third_party/grpc/_cython/__init__.py)
root@clusterfuzz-linux-21mc:/mnt/scratch0# python3.7 -m pip list
Package                 Version
----------------------- ---------
absl-py                 1.0.0
astunparse              1.6.3
cachetools              5.0.0
certifi                 2021.10.8
charset-normalizer      2.0.11
crcmod                  1.7
distlib                 0.3.4
filelock                3.4.2
future                  0.17.1
gast                    0.3.3
google-auth             2.6.0
google-auth-oauthlib    0.4.6
google-pasta            0.2.0
grpcio                  1.43.0
h5py                    2.10.0
idna                    3.3
importlib-metadata      4.10.1
Keras-Preprocessing     1.1.2
Markdown                3.3.6
numpy                   1.18.5
oauthlib                3.2.0
opt-einsum              3.3.0
pip                     19.2.3
pipenv                  2022.1.8
platformdirs            2.4.1
protobuf                3.15.3
psutil                  5.7.0
pyasn1                  0.4.8
pyasn1-modules          0.2.8
requests                2.27.1
requests-oauthlib       1.3.1
rsa                     4.8
scipy                   1.4.1
setuptools              41.2.0
six                     1.12.0
tensorboard             2.8.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.1
tensorflow              2.3.0
tensorflow-estimator    2.3.0
termcolor               1.1.0
typing-extensions       4.0.1
urllib3                 1.26.8
virtualenv              20.13.1
virtualenv-clone        0.5.7
Werkzeug                2.0.2
wheel                   0.37.1
wrapt                   1.13.3
zipp                    3.7.0

The fuzz bot images seem to be pinned to gcr.io/clusterfuzz-images/base:a2f4dd6-202202070654. Has anyone else seen this issue?

[nguyendon]

Actually I think I know what my issue is. I patched v2.6.0 to use python38 since python37 wasnt working. The deploy bundle is packaged with python38 so that is probably why its failing on the fuzzbot image.

[nguyendon]

Yup that was the issue.