pip install from stdin

[kenahoo]

I see that an existing ticket has been rejected: #3880 . I'm requesting that the team reconsider.

The main use case I have for this feature is to build an environment using a set of requirements specified in a URL or coming from a specific git commit:

% curl https://foo.com/requirements.txt | pip install -r -
% git show deadbeef:requirements.txt | pip install -r -

There are workarounds, e.g. to leverage some shell feature for running a subprocess (e.g. pip install -r <(curl https://foo.com/requirements.txt) or similar) or download the file in advance to some temporary location, but these are more esoteric, shell-dependent, or require extra cleanup steps. I believe the cleanest solution for the user is simply to behave like a standard command-line tool and use stdin/stdout in the normal ways.

In the spirit of conforming even better to standard POSIX style, I'd further recommend that the - be unnecessary, e.g. if no file argument is given for -r that input be taken from stdin, just like other input-consuming tools like cat, cut, less, python etc. do:

% curl https://foo.com/requirements.txt | pip install -r
% git show deadbeef:requirements.txt | pip install -r

Thanks for considering.

[pfmoore]

Strong -1 on the -r without arguments form, that seems likely to be very confusing.

I can see the value of -r -, although I doubt I'd use it much myself. There are also likely to be design questions that need to be thrashed out, too. But the biggest blocker is probably that someone needs to actually do the work of writing a PR for this (which includes tests and documentation).

@pypa/pip-committers Is anyone actively against this new feature? I don't want to suggest that someone should go to the effort of writing a PR if it's just going to get rejected.

I've marked the request as "deferred till PR" - I doubt there's much more to discuss until there's actual code to base a discussion on.

[xavfernandez]

Also -1 on the -r without argument.

Concerning -r - I'd be more amenable, but then we'd also have to accept -c - and of course define what should happen for pip install -c - -r -.

[uranusjr]

From personal experience working with other software, a common practice is to accept stdin separated by EOF (^D in readline) by argument order. I have zero idea how this should be implemented though, since optparse does not keep argument ordering AFAIK.

[wchargin]

Also −1 on -r without argument, but definitely +1 on -r -. I just
assumed that this would work and was quite surprised when it didn’t. (My
use case: grep flake8 requirements.txt | pip install -r - in a lint
step.) The existing workarounds are not good; /dev/stdin does not work
on Windows, and <(...) does not work in a POSIX shell.

[duckinator]

I think, if this is added, -r - is the way to go. -r may be more "POSIX style", but it's also an aspect of "POSIX style" programs that I have repeatedly been bamboozled by. 😅

@xavfernandez thank you for pointing out that potential complication. I didn't even know constraints files existed until reading your comment. 🙂

[kenahoo]

I think, if this is added, -r - is the way to go. -r may be more "POSIX style", but it's also an aspect of "POSIX style" programs that I have repeatedly been bamboozled by. 😅

Just to be a POSIX defender: I don't think we'd enjoy a world quite as much where we routinely had to write grep foo file.txt | cut -f2-5 - | sort -r - | less - instead of simply grep foo file.txt | cut -f2-5 | sort -r | less. The cognitive load of | less - instead of | less alone seems like a pretty good validation of the philosophy that "everything works on pipes".

That said, I don't need to debate the philosophy, it would just be nice if we had something in pip that worked easily for this use case.

[pradyunsg]

Added more labels, to clarify state.

@pypa/pip-committers Is anyone actively against this new feature?

Not me.

As far as I can tell:

• we're OK to use - as an indicator of "use stdin", for -r and -c
  • need to define what should happen for pip install -c - -r -
• we DO NOT want to "fall back to stdin" if no arguments are given to -r

[pfmoore]

need to define what should happen for pip install -c - -r -

Personally, I'd just say it's not allowed. It's extremely non-obvious what the best approach is, and it's a pretty obscure corner case. I'd actually be perfectly happy to just call YAGNI on -c - and stick with -r - alone, but there's bound to be someone who wants it "for consistency"...

[sbidoul]

An alternative that works today (example with bash):

pip install $(curl https://foo.com/requirements.txt)

[kenahoo]

@sbidoul I think your workaround is probably less desirable than the pip install -r <(curl https://foo.com/requirements.txt) example I gave in my original message. Yours puts the entire contents of the file into a long shell command, whereas mine at least keeps it in the stdout/stdin of the processes.

The mental exercise of knowing the different consequences of those two variants, for which versions of which shells, is why I think being able to read from stdin natively is so important.

(details:)

If you're using a reasonably modern version of bash, I think your version is at least protected from a shell injection attack where requirements.txt contains something like pandas;\nrm -rf /, but that's up to the details of how subprocess interpolation is implemented. You could still be hijacked if it contained something like -i https://bad.site.com/ pandas, because it gets evaluated to the shell command pip install -i https://bad.site.com/ pandas. You also have to think carefully about what happens if there's an entry like pandas>=0.25 in the file - does it work as desired, or become equivalent to pip install pandas > "=0.25"? Usually it'll be fine, but again it's not obvious.

Although the above talks about security concerns, shell interpolation can also cause insidious bugs in your workflow if you're not really careful, so I'd just recommend avoiding it when it's not needed.

[sbidoul]

@kenahoo sure, your arguments about the risk of shell interpolation are valid. I was not meaning to dismiss the feature request :)

You could still be hijacked if it contained something like -i https://bad.site.com/ pandas

Regarding this specific point, note pip does recognize a number of options inside requirements files, so this risk would still exist when reading requirements from stdin.

[kenahoo]

Regarding this specific point, note pip does recognize a number of options inside requirements files, so this risk would still exist when reading requirements from stdin.

Oh, I didn't realize that! Thanks for pointing it out, will probably be useful someday.

[mitar]

Yes, I realized this would be handy if you want to use --hash from the command line (#3257). Currently --hash only works inside requirements file. But you cannot simply pass it through stdin because -r - does not work.

My current workaround is:

echo 'package_name --hash=sha256:0bf006c3aa74b59bcab0732eeb62e1b43fbbe580237b254957d9f38a31dffa8c' | pip3 install -r /dev/stdin

[andreashe]

Have another solution using xargs (filters all packages with letter "a"):

cat requirements.txt | grep 'a' | xargs pip install

[Matze1224]

Have another solution using xargs (filters all packages with letter "a"):

cat requirements.txt | grep 'a' | xargs pip install

I've using something like this for installing the netbox requirements.txt but it recently start to fail cause of the newly added comment char isn't recognized as as a pip package (but included as argument). So it would be saver to rely on pip's requirements.txt parsing instead appending to the arguments.

[Hubro]

My use case for this is trying to install my requirements inside a remote Docker container:

cat requirements.txt | docker exec -i my-container pip install -r -

But alas 🙁 Seems I'll have to copy them to a temporary file in the container.

[bashlakov]

I think I found elegant solution, which works in Dockerfile RUN statement and correctly handling error in pipe (because pip install -r <(some_command) did not fail if some_command exited unsuccessfully, at least in Dockerfile RUN). Solution is:
some_command | pip3 install -r /dev/stdin

[kenahoo]

@bashlakov that's the same solution @mitar pointed out as a workaround. It has the same problem as several other solutions, which is that it depends on a specific OS (e.g. /dev/stdin won't work at a Windows shell) and/or shell (some shells "emulate" it somehow). On Linux it's pretty much always available, on older Mac systems it's generally not, apparently not either on AIX (if people still use that, I'm not sure). See https://unix.stackexchange.com/questions/123602/portability-of-file-descriptor-links .

It's a good workaround if you don't need it to work on other systems besides the one you wrote it for, though.