On install, add a pep517 backend agnostic and documented option to set a build directory

[manuco]

Description

In the past, there was an option to set the build directory to any directory. This option allowed us to avoid a "no space left on device" error when the /tmp directory was too small for large builds.

Now that this option has disappeared with the PEP517 build backends, there seems to be no consistent way to set this option.

Since the goal of PEP517 was to abstract the build backend, when we install a new package, we don't know which build backend that package will use. So setting an equivalent option for every build backend we could find is not a scalable answer.

How can I set this option for each backend once and for all?

Expected behavior

A documented option that tells pip where any build operation can take place, no matter what the build backend is. An implementation of this option where the provided place is used for builds operations.

pip version

23.3.2

Python version

3.8

OS

Linux

How to Reproduce

1. pip install --build-directory /tmp requests

Output

$ pip install --build-directory /tmp requests

Usage:
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

no such option: --build-directory

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[webknjaz]

This might not be the right place for the issue since pip vendors pyproject_hooks which is where the tmp dir is made. It's created through stdlib tempfile.TemporaryDirectory() and I think it probably supports the standard env vars TMPDIR, TEMP or TMP.
Could you try setting them to see if that solves your issue?
If this doesn't work, please file an issue in the pyproject_hooks repository because it doesn't expose any APIs for changing the temporary directory.

[pfmoore]

To be clear, pip expects to write temporary files into the TMPDIR/TEMP/TMP directory, and we deliberately don't provide any other mechanisms for putting specific files in different places.

In spite of appearances, there has never been an option in pip to set the build directory in a build backend independent way. Instead, prior to pyproject.toml support, pip only supported setuptools and exposed some setuptools-specific options in the pip UI. But there was never any expectation that those options would work the same across all build backends.

If you are looking for a backend-agnostic way of doing this, you'll need to get agreement that all build backends will support a common option. This would probably be a specific setting in the config_options dictionary, but the details don't really matter at this point. What matters is that such an agreement would be a new interoperability standard, and would need to be established as such via the PEP process, as documented here.

So to summarise:

1. There's very little chance of pip implementing anything specific for this.
2. Using the TEMP (etc.) variables is how people typically address this issue. It seems likely this will work for you, from the description you gave of your problem.
3. If you really want a backend-agnostic solution, you should be prepared to propose a new standard for it.

Hope this helps, and sorry if it's not as simple as you were hoping for.

[manuco]

Thanks @webknjaz and @pfmoore for your detailed answer. The TMPDIR environment variable seems to work for us (we don't have "no space left on device" errors anymore).

I'll have a look at the PEP process, but I'm not quite sure I've the ressources (time and motivation) to lead it.

In the meantime, would it be possible to add a section in the pip documentation about this caveat and the associated workaround ?

[pfmoore]

I'll have a look at the PEP process, but I'm not quite sure I've the ressources (time and motivation) to lead it.

Honestly, if setting TMPDIR is enough, a PEP for this would be overkill, so I wouldn't worry about it.

In the meantime, would it be possible to add a section in the pip documentation about this caveat and the associated workaround?

It's mentioned in the changelog entry deprecating --build-dir. IMO that's sufficient for covering the question of "what do we do now that --build-dir is no longer available?" However, I accept that it wasn't discoverable enough in your case - if you have any ideas on how we could improve this, that would be great. But if you feel there's a more general statement that covers more than just "replacing --build-dir", feel free to submit a PR for the docs showing what you'd like to see.

[manuco]

Hi @pfmoore , I'll think about what could be written in the documentation to improve this point.

[webknjaz]

Related discussion: pypa/build#655.

[noctuid]

To be clear, pip expects to write temporary files into the TMPDIR/TEMP/TMP directory

Pip does not seem respect TMPDIR in all cases, which currently makes it impossible for our project to use pip install in our github actions. Our company's github runner mounts /tmp with noexec, so we need pip to use a different directory or pip install fails (relevant stack overflow question). However, setting TMPDIR (or TEMP or TMP) does not change the build-env directory. It's still something like /tmp/pip-build-env...

It's not immediately clear to my why this is. Pip appears to be using tempfile.mkdtemp, which respects TMPDIR, but I don't seem to be the only one having this issue. I found this stackoverflow question where someone needs to change the build directory for a different reason, but setting TMPDIR did not work.

EDIT: Now it's obvious why it didn't work. It falls back to /tmp if the specified directory does not exist instead of erroring or creating it. It looks like everything works fine if the directory exists.

[webknjaz]

Should this issue be turned into "warn if the tmp dir does not exist and a fallback is used"?

[noctuid]

A warning and/or documentation about this would be helpful