chore: bump internal_deps

[aignas]

Before this PR the dependencies were out of date and the pyproject_hooks was
missing allowing pip-compile to use pyproject.toml instead of
requirements.in. With the update we are more likely to support the new
features but they are not tested in this PR as the main goal is to update all of
the files that need regenerating due to upgraded pip-tools.

Here we:

• Upgrade all of the deps to the latest versions.
• Update all of the requirements and gazelle manifest files.
• Add a quirk to ensure fix pip_compile on Windows.

Summary of version changes:

• build: 0.9.0 -> 0.10.0
• click: 8.0.1 -> 8.1.6
• importlib_metadata: 1.4.0 -> 6.8.0
• more_itertools: 8.13.0 -> 10.1.0
• packaging: 22.0 -> 23.1
• pip: 22.3.1 -> 23.2.1
• pip_tools: 6.12.1 -> 7.2.0
• pyproject_hooks: 1.0.0
• setuptools: 60.10.0 -> 68.0.0
• wheel: 0.38.4 -> 0.41.0
• zipp: 1.0.0 -> 3.16.2

Fixes #1351

[aignas]

This is still failing on Windows, but I am not entirely sure why, maybe someone with Windows could look deeper in to this.

[rickeylev]

The windows error is:

      File "C:\temp\Bazel.runfiles_nst1tke9\runfiles\python39_x86_64-pc-windows-msvc\lib\pathlib.py", line 276, in gethomedir
        raise RuntimeError("Can't determine home directory")
    RuntimeError: Can't determine home directory

It's originating via pip -> setuptools -> pathlib. As part of, what appears to be, looking for configuration files. Perhaps there's an issue filed upstream? Is this setuptools version compatible with Python 3.9?

Looking at the Python 3.9 pathlib code, it checks for USERPROFILE in the environment, which is set. Maybe that isn't filtering down to the the action? Adding --test_env=USERPROFILE in the CI config might fix that.

[chrislovecnm]

It's originating via pip -> setuptools -> pathlib. As part of, what appears to be, looking for configuration files. Perhaps there's an issue filed upstream? Is this setuptools version compatible with Python 3.9?

I'm not finding an issue upstream in setuptools. And I cannot find a compatibility matrix.

[chrislovecnm]

So adding the parameter --no-user-cfg will bypass the code causing this problem. See the parameter here:

https://github.com/pypa/setuptools/blob/1d9420746741c6ad8bc0757b5dcf6cadf3c1444d/setuptools/_distutils/dist.py#L279

Set the parameter and setuptools does not hit the pathlib call here:

https://github.com/pypa/setuptools/blob/1d9420746741c6ad8bc0757b5dcf6cadf3c1444d/setuptools/_distutils/dist.py#L354C60-L354C60

I don't think we want to pick up user configuration anyways.

There are long discussions around '~,' and Windows python/cpython#80445 is one of them.

setuptools moved from using os.path.expanduser('~') a while ago, and there are inconsistencies between the two implementations in Python that I have read about. See the setuptools commit here:

pypa/setuptools@c7d65dd#diff-1da6455033333a4ccf962e1f9f770e138d83c5775c11f72e16ebef1688026284L353-L355

So I am guessing that the older os.path implementation worked while the never implementation of using pathlib does not.

As I mentioned, if we set the --no-user-cfg parameter for setup tools, the setuptools code will not use the pathlib call, and setup tools won't try to find user configuration outside the sandbox.

[aignas]

I spent some time debugging this and trying to understand the root cause of what is going on and I am not sure that the previous analysis in the comment above is entirely correct.

Looking at this log, I think what is happening is the following:

1. The pip-tools is trying to lock the requirements.
2. Then it is calling pip internal to fully resolve everything.
3. Then pip needs to do some processing of the package and it is calling the vendored pyproject-hooks which exists within the pip/_vendor folder.
4. ...
5. It calls a subprocess.run which due to some reason has it's sys.path include Python files from outside the runfiles directory:

   C:\temp\Bazel.runfiles_n6t3rqoq\runfiles\pypi__pip\pip\_vendor\pyproject_hooks\_in_process
   C:\temp\pip-build-env-gejvde8f\site
   C:\temp\Bazel.runfiles_n6t3rqoq\runfiles\python39_x86_64-pc-windows-msvc\python39.zip
   C:\temp\Bazel.runfiles_n6t3rqoq\runfiles\python39_x86_64-pc-windows-msvc\DLLs
   C:\temp\Bazel.runfiles_n6t3rqoq\runfiles\python39_x86_64-pc-windows-msvc\lib
   C:\temp\pip-build-env-gejvde8f\overlay\Lib\site-packages
   C:\temp\pip-build-env-gejvde8f\normal\Lib\site-packages
   

6. We fail whilst we are trying to build a wheel? This is where I get confused,
   as locking down the requirements should not need to build them, but given
   Python packaging complexity and non-hermeticity I would not be surprised if
   this is the actual behaviour that we have here.

I still think it would be good to get to the root cause of this and have the
full picture why multi_versions, pip_install and pip_parse integration
tests. multi_versions example is also only failing with Python 3.11,
whereas other examples are using 3.9. What is more, multi_versions
requirements contain only websockets and I am curious why it would work with
one Python version but not others. The same goes for why it would work on Linux
but not on Windows.

[groodt]

6. We fail whilst we are trying to build a wheel? This is where I get confused,
as locking down the requirements should not need to build them, but given
Python packaging complexity and non-hermeticity I would not be surprised if
this is the actual behaviour that we have here.

Oh, yes, this is precisely what happens when locking dependencies. This happens automatically when an sdist is encountered, because there is no static metadata for sdists. So a wheel is built from the sdist. If I had my way, the rules would only support wheels (--only-binary=:all:), but I think that ship sailed long ago.

The test is failing on yamllint which is only provided as an sdist (https://pypi.org/project/yamllint/#files). Why does the test require yamllint? Can't we do something else instead?

I personally think this might be too aggressive an upgrade? What's the motivation to upgrade everything at once? If we're upgrading the tooling dependencies, such as pip-tools and build, I can get behind that.

In general, upgrading setuptools needs to be done carefully, because we probably want the oldest version of setuptools thats compatible with pip and the rest of our tools? If we upgrade setuptools too aggressively, we risk breaking the ability to build older packages? From my quick skim of the upgrade, I don't see anything obvious, but setuptools is a load-bearing, so unless there's a strong reason to upgrade it, maybe upgrade it independently?

[groodt]

I think the failure on Windows now, is due to the newer setuptools vendored distutils that is attempting to expand a home directory when building an sdist for websocket on py311 for the old version of websocket.

I can bump the pinned version of websocket so that an sdist will not be but I think this will likely break rules_python for windows users who have any sdist that make use of distutils?