env: add support for uv #751

layday · 2024-03-05T21:41:30Z

No description provided.

pyproject.toml

layday · 2024-03-06T15:47:24Z

It looks like uv venv doesn't work with PyPy on Windows.

gaborbernat · 2024-03-06T15:49:44Z

Yeah astral-sh/uv#2096, there's no official support for PyPy today, and what works, it works by chance.

webknjaz · 2024-03-06T16:34:25Z

@layday have you considered exposing an API for pluggable build env provisioners?

layday · 2024-03-06T17:35:08Z

Can you go into more detail?

webknjaz · 2024-03-06T21:35:43Z

Can you go into more detail?

I mean, it'd be nice if third parties could stick their bits of build env provisioning logic through some standardized interface. This could be useful for various downstreams and probably other tools that act as frontends.

19.0 incorrectly interprets Python version constraints resulting in venv being used in pref to virtualenv when an outer pip is too old.

layday · 2024-03-08T13:27:02Z

So, this adds an --env-impl flag to the CLI, with the only accepted value being venv+uv. Old semantics apply if omitted.

I've modified the venv selection order slightly so that venv is preferred whenever a valid outer pip is present for a minor speed boost. virtualenv continues to be preferred with an outdated pip.

The DefaultIsolatedEnv is now tested with all venv and installer combos (virtualenv used to be mocked). This revealed the venv module was preferred with packaging 19.0 even when a valid virtualenv was present. The culprit was a marker evaluation bug in packaging which was fixed in 19.1.

Finally, I've added a setuptools existence check before attempting to uninstall the ensurepip copy of setuptools - it was dropped in Python 3.12. Again, this should make using venv without an outer pip slightly faster.

layday · 2024-03-08T16:04:54Z

~~Currently, we're only looking for a uv executable on $PATH (not attempting an import), but see the discussion happening at wntrblm/nox#795 (comment).~~ Still not importing uv but also looking under sys.prefix for unactivated venvs.

gaborbernat · 2024-03-09T18:49:14Z

src/build/env.py

+        # find it under ``sys.prefix``, essentially potentially rearranging
+        # the user's $PATH.  We'll only look for uv under the prefix of
+        # the running interpreter for unactivated venvs then defer to $PATH.
+        uv_bin = shutil.which('uv', path=sysconfig.get_path('scripts'))


uv comes with a find_uv_bin method we should use that.

The comment explains why it's not being used.

Guess I agree here with uv's choice, I find it more expected, and disagree with the comment. And even if should uv's strategy be an actual problem, I'd like them to fix it, and not us work around it.

How is it expected that a uv you've installed in a venv might load a uv from an outer env?

$ python -m venv --without-pip foo $ VIRTUAL_ENV=foo uv pip install uv Resolved 1 package in 3ms Installed 1 package in 5ms + uv==0.1.16 $ foo/bin/python -c 'import uv; print(uv.find_uv_bin())' /Users/dimitris/code/python-packaging/build/foo/bin/uv $ rm foo/bin/uv $ foo/bin/python -c 'import uv; print(uv.find_uv_bin())' /Users/dimitris/.local/bin/uv

If uv should function as a Python package, it has no business looking outside its prefix.

I'd personally prefer to take a standards-based approach than rely on uv's custom script finder. Is this not the correct way to locate a script under the running interpreter's prefix?

Open an issue in UV and we should fix it there then.

Changed to only use uv.find_uv_bin.

gaborbernat · 2024-03-09T19:57:19Z

src/build/env.py

+        uv_bin = shutil.which('uv', path=sysconfig.get_path('scripts'))
+
+        if not uv_bin:
+            uv_bin = shutil.which('uv')


I strongly disagree with such fallback. Either pick uv from sys.prefix or fail. Using a stray executable we pick up from PATH could use surprises for users...

I've generally been respecting the design of uv, which is to not install itself in environments, but to be used globally (via https://astral.sh/uv/install.sh, cargo install, homebrew, pipx, pacman, etc) and only target environments. If a user does install uv in build's environment, that should take preference, but I think we should try to use an outer/system uv otherwise. I don't think someone passing --installer=uv would be unhappy if a uv executable on their path is used. (same with cmake, gcc, ninja, patchelf, etc., lots of executables get used from the path)

But willing to go with this if you really prefer.

Then remove the UV extra because user should install it separately? Having an extra but then taking a global is unexpected to me.

A user can install it, say if the system uv is older than they like, or if they just use pipx run build[uv] regardless of if the system has uv or not. But, if they are on a system that does not support binary wheels, like BSDs, they might prefer or even need to install it globally and not use the extra.

(This is very common for binary deps with optional PyPI packages like CMake and ninja)

We are not in the business of installing cmake, why is UV any different (other than it happens to be available on pypi, is the same a random binary).

I guess I can accept falling back to the operating system version, but if we do that in my opinion we must print out of warning onto the standard output or error to notify the user that this is happening.

Something like this would be perfect, I think:

* Creating venv isolated environment... * Using uv 0.1.16 from /usr/local/bin/uv to install packages... * Installing packages in isolated environment... (hatch-fancy-pypi-readme, hatch-vcs, hatchling)

(Maybe it could include "found on PATH" or something too?) Whereas if it was from the local environment, it could be simplified:

* Creating venv isolated environment... * Using uv 0.1.16 to install packages... * Installing packages in isolated environment... (hatch-fancy-pypi-readme, hatch-vcs, hatchling)

Would that work?

Reminder that we have a verbosity flag now that will print out the command invoked including the full uv executable path. Printing out the version for a uv executable from $PATH would require an additional subprocess call and parsing the output of -V which is a flimsy affair. This same consideration also applies to an outer pip, although the surprise factor is slightly diminished since we’re only looking in the running interpreter’s Python path.

How's this?

layday · 2024-03-09T22:30:22Z

src/build/env.py

@@ -21,6 +20,11 @@
 from ._util import check_dependency


+Installer = typing.Literal['pip', 'uv']


Do we wanna call this uv-pip instead of simply uv?

I think the --installer implies uv pip. And if they added a “uv install”, we could transparently start using that instead.

layday · 2024-03-11T06:23:11Z

I would’ve preferred to squash both this and the other PR you merged personally. There’s a fair few fix commits and reverts and counter-reverts with attribution ultimately being lost in git blame. I’ll open another PR to add a change log entry.

gaborbernat reviewed Mar 5, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

layday added 4 commits March 6, 2024 00:00

Add uv extra

2f50d8d

Use argparse shortcut for dist args

24d9efe

Rename --config-setting dest to be in the plural

a57609f

Add provision for env param to custom subp runner

db6069c

layday force-pushed the feat-uv-env branch from f185d23 to 6349f84 Compare March 5, 2024 22:00

layday added 2 commits March 6, 2024 17:10

Refactor env impl, adding support for uv

dd1c3f6

Add --env-impl flag to CLI

a66b179

layday force-pushed the feat-uv-env branch 2 times, most recently from 779e7bc to 77bf09d Compare March 6, 2024 15:21

layday marked this pull request as ready for review March 6, 2024 15:30

layday requested a review from FFY00 as a code owner March 6, 2024 15:30

layday changed the title ~~Add support for uv~~ env: add support for uv Mar 6, 2024

layday mentioned this pull request Mar 6, 2024

feat: add a uv backend wntrblm/nox#762

Merged

henryiii mentioned this pull request Mar 6, 2024

perf: avoid compiling unused bytecode #752

Merged

layday added 7 commits March 6, 2024 21:15

Use os.name as is done by the stdlib sysconfig and venv mods

7084cf4

Update tests

fd3563e

Require uv and virtualenv for testing

0c6c261

Add more env tests

5a3609f

Back out of reading pyvenv.cfg

641170c

Create uv envs with venv

c6d86d2

Do not expose pip implementations

2e65e58

layday force-pushed the feat-uv-env branch from b45d04e to e01ad52 Compare March 6, 2024 19:40

Update tests

959f116

layday force-pushed the feat-uv-env branch from e01ad52 to 959f116 Compare March 6, 2024 19:45

Fix coverage

619126b

layday force-pushed the feat-uv-env branch from 9fa1cb5 to 619126b Compare March 8, 2024 12:09

layday added 2 commits March 8, 2024 14:26

Test with original has_virtualenv

430ad00

Bump minimum version of packaging

0680ae7

19.0 incorrectly interprets Python version constraints resulting in venv being used in pref to virtualenv when an outer pip is too old.

layday added 2 commits March 8, 2024 18:29

Test pip and uv verbosity flags

6844fb7

Simplify test_pip_needs_upgrade_mac_os_11

7d3ecf6

layday closed this Mar 8, 2024

layday reopened this Mar 8, 2024

Restore the uv extra, look for uv under sys.prefix

e04d030

layday force-pushed the feat-uv-env branch from 75ea285 to e04d030 Compare March 8, 2024 23:05

layday added 3 commits March 9, 2024 10:43

Fix spelling of 'PyPy'

49d1379

Rm PyPy support warning

bf9681f

Fix xfailing test, only xfail on win

468350e

layday requested review from gaborbernat and henryiii March 9, 2024 18:43

gaborbernat requested changes Mar 9, 2024

View reviewed changes

gaborbernat reviewed Mar 9, 2024

View reviewed changes

layday added 3 commits March 9, 2024 23:37

Rename env_impl param to installer

deabd67

Defer to uv.find_uv_bin

3766079

Fix test param

1408af4

layday commented Mar 9, 2024

View reviewed changes

layday added 2 commits March 10, 2024 11:07

Restore uv $PATH lookup with logging message

c5040f4

Help mypy narrow

a27bb85

gaborbernat approved these changes Mar 10, 2024

View reviewed changes

gaborbernat merged commit 327f280 into pypa:main Mar 10, 2024
57 checks passed

agriyakhetarpal mentioned this pull request Mar 23, 2024

Feature request: provide an option to build just the source distribution, allow skipping wheel builds hynek/build-and-inspect-python-package#83

Closed

astrojuanlu mentioned this pull request Mar 24, 2024

Use uv backend astrojuanlu/pygetimportables#2

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

env: add support for uv #751

env: add support for uv #751

layday commented Mar 5, 2024

layday commented Mar 6, 2024

gaborbernat commented Mar 6, 2024

webknjaz commented Mar 6, 2024

layday commented Mar 6, 2024

webknjaz commented Mar 6, 2024

layday commented Mar 8, 2024 •

edited

layday commented Mar 8, 2024 •

edited

gaborbernat Mar 9, 2024

layday Mar 9, 2024

gaborbernat Mar 9, 2024 •

edited

layday Mar 9, 2024

gaborbernat Mar 9, 2024

layday Mar 9, 2024 •

edited

gaborbernat Mar 9, 2024 •

edited

henryiii Mar 9, 2024 •

edited

gaborbernat Mar 9, 2024

henryiii Mar 9, 2024

gaborbernat Mar 9, 2024

gaborbernat Mar 10, 2024

henryiii Mar 10, 2024 •

edited

gaborbernat Mar 10, 2024

layday Mar 10, 2024 •

edited

layday Mar 10, 2024

layday Mar 9, 2024

henryiii Mar 10, 2024

layday commented Mar 11, 2024 •

edited

		@@ -21,6 +20,11 @@
		from ._util import check_dependency


		Installer = typing.Literal['pip', 'uv']

env: add support for uv #751

env: add support for uv #751

Conversation

layday commented Mar 5, 2024

layday commented Mar 6, 2024

gaborbernat commented Mar 6, 2024

webknjaz commented Mar 6, 2024

layday commented Mar 6, 2024

webknjaz commented Mar 6, 2024

layday commented Mar 8, 2024 • edited

layday commented Mar 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaborbernat Mar 9, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

layday Mar 9, 2024 • edited

Choose a reason for hiding this comment

gaborbernat Mar 9, 2024 • edited

Choose a reason for hiding this comment

henryiii Mar 9, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

henryiii Mar 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

layday Mar 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

layday commented Mar 11, 2024 • edited

layday commented Mar 8, 2024 •

edited

layday commented Mar 8, 2024 •

edited

gaborbernat Mar 9, 2024 •

edited

layday Mar 9, 2024 •

edited

gaborbernat Mar 9, 2024 •

edited

henryiii Mar 9, 2024 •

edited

henryiii Mar 10, 2024 •

edited

layday Mar 10, 2024 •

edited

layday commented Mar 11, 2024 •

edited