[WIP] Bumping deps to an intermediary step compatible with Python3.11

[vitorguidi]

Motivation

Some of our current dependencies are incompatible with Python 3.11. This PR attempts to bump the least possible amount of dependencies (more specifically, grpcio, grpcio-tools, psutil and markupsafe), so we can isolate deployment issues between dependency bumps and the python upgrade itself.

[jonathanmetzman]

This doesn't seem right that we need to add so many platform_requirements.

[vitorguidi]

There are transitive dependencies which are C extensions, and were previously not declared.

[jonathanmetzman]

Ah.
So I don't always love the idea of pinning dependencies because it can confuse what we are actually interested in and makes it harder to upgrade the main package. Can we just leave these out?

[vitorguidi]

I would rather declare every single C dependency. Otherwise pipenv lock will choose those for us, and I have no assurance that whatever it chooses will be compatible with python 3.11.

[jonathanmetzman]

most of these are defined in the pipfile and you can just define them there.

[oliverchang]

+1. Everything in the root Pipfile (https://github.com/google/clusterfuzz/blob/master/Pipfile) is expected to be installed on the system. This will certainly require C-extensions (and possibly transitively).

src/Pipfile (which is separate from the root Pipfile) contains the packages that we download and re-package into zips for deployment. In theory, only grpcio and protobuf should require C extensions there, because we've never had any issue on Mac/Windows with just patching in the platform-specific versions of these two (which is what platform_requirements.txt does).

Eventually, we do want to move everything from the root Pipfile into platform_requirements.txt, but perhaps we can do this more iteratively. e.g. just move a minimal seet that we have to absolutely upgrade for Python 3.11 support for now (e.g. psutil?)

[oliverchang]

and perhaps with PYTHONPATH/sys.path ordering, we don't even have to uninstall the existing version of psutil on the system if the newer one we provide in the source zip takes precedence over the system one.

see

clusterfuzz/src/clusterfuzz/_internal/base/modules.py

Line 47 in 5d4bd65

def fix_module_search_paths():

for how we mess with python module search paths on startup.

[vitorguidi]

Hmm there is a >0 chance that @jonathanmetzman #3663 was actually correct, and Windows only broke because it was not running psutil >= 5.9.4. The largest version I saw in our windows init scripts was 5.7.0, which is not compatible with py 3.11.

I might be wrong, but this seems very plausible

[jonathanmetzman]

I'm confused, i thought this was your theory. I thought i succeeded in upgrading the version on Windows.

[vitorguidi]

I was working under the assumption that the update succeeded on linux but not on windows . Is that the case?