PIP update command - No response from CLI on Windows

[seanvree]

• Pip version: pip 9.0.1
• Python version: 2.7 & 3.4
• Operating system: Windows 10E X64

Description:

I have python 2.7 and python 3.6 installed With all the python tools. I can run python scripts all day.
So, I created a bat file that will do this for me once a day, code is the following:
python2.7 -m pip freeze --local | grep -v '-e' | cut -d = -f 1 | xargs -n1 pip install -U
So, when I put this in a windows CLI with admin priv, it just hangs after the command is passed. The CMD window is NOT froze, it just sits there and does nothing.
I can try this with the 3.6 python as well, and it does the same thing:
call python3.6 -m pip freeze --local | grep -v '-e' | cut -d = -f 1 | xargs -n1 pip install -U
I've tried powershell as well and it does the same thing.
Now, I CAN update the packages one by one using PIP, but that's too much.
What am I doing wrong?

What I've run:

c:\Python27>python2.7 -m pip install -r requirements.txt --upgrade
Invalid requirement: 'Terminate batch job (Y/N)?'
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\pip\req\req_install.py", line 82, in __init__
    req = Requirement(req)
  File "C:\Python27\lib\site-packages\pip\_vendor\packaging\requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "u'batch jo'"

[pfmoore]

This seems to be a Windows batch programming issue, not a pip problem.

Your requirements.txt file seems to include "Terminate batch job (Y/N)?" which is output you get if your batch file has an error. I suspect you aren't handling errors properly in the process that generates your requirements.txt file. I'd suggest reading requirements.txt to see if it contains error output from your batch script, and if so, fix those errors.

[seanvree]

@pfmoore

I'm not running a batch script. I'm simply opening an Admin CMD prompt and typing in the following
python2.7 -m pip freeze > requirements.txt
python2.7 -m pip install -r requirements.txt --upgrade

But you are right when I run the first command It adds that line "Terminate batch job (Y/N)?" to the end? Why is that?

I've also tried running the following:

C:\WINDOWS\system32>cd c:\python27

c:\Python27>for /F "delims===" %i in ('pip freeze -l') do pip install -U %i

But the CMD prompt doesn't do anything, it's not froze, it just sits there.

I CAN update all the packages manually by running:

pip install --upgrade

[pfmoore]

It adds that line "Terminate batch job (Y/N)?" to the end? Why is that?

Honestly, I don't know. This isn't really the right place to get help with windows batch file programming. You said your batch file contained:

python2.7 -m pip freeze --local | grep -v '-e' | cut -d = -f 1 | xargs -n1 pip install -U

That's odd, as grep, cut and xargs aren't builtin Windows commands, so presumably you're getting Unix ports from somewhere. You should check they work as expected on Windows. And you use grep -v '-e', but single quotes are a Unix feature, Windows uses double quotes. Did you write this pipeline yourself or copy it from somewhere? And if the latter, did you inadvertently copy something intended for Unix?

You should probably ask for further help on a Windows batch programming forum. I'm going to close this as it doesn't appear to be a pip issue.

[seanvree]

@pfmoore - I got you. I expected that answer. This started happening when I recently updated a few packages, so I know it's a python/pip issue.

I understand they aren't builtin commands, but I'm running python and PIP from the python emulator and it's still doing it.

Again, it's not a batch command, I'm simply running it from a CMD prompt.

I copied the command line from here:

http://stackoverflow.com/questions/2720014/upgrading-all-packages-with-pip/3452888#3452888

I guess I'll look in a windows-based source.

THX

[pfmoore]

Yep, that's a Unix command line. Not suitable for you at all.

If you want a powershell command (I'm not sure how to do it in CMD, as the CMD syntax is horrid...) to get the list of packages you have installed,

py -m pip freeze | % { ($_ -split '==')[0] }

should do the job. Your problem text was in a requirements.txt file, and I don't see where that was coming from - py -m pip freeze >requirements.txt should give a working reqirements file if you want to fix the versions, or py -m pip freeze | % { ($_ -split '==')[0] } >requirements.txt if you want to allow newer versions.

Hope some of that helps.

[seanvree]

@pfmoore - when I run python -m pip freeze | % { ($_ -split '==')[0] } in PS, nothing happens. It returns to the line below and does absolutely nothing.

With all do respect. this is not a CLI issue. There's something that is hanging up the command from running, or the output from being displayed.

[pfmoore]

OK. In which case I'm baffled. Does python -m pip freeze produce output?

[seanvree]

@pfmoore Yes, it does, see below. I'm baffled too. I may be a NOOB, but I know the basics.

Also, if I do python2.7 -m pip freeze > requirements.txt The CMD prompt will hang , but it WILL produce a requirments.txt file.

python -m pip freeze:

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>cd c:\python27

c:\Python27>python -m pip freeze
app==0.0.1
appdirs==1.4.0
autohook==1.1.0
babelfish==0.5.5
backports-abc==0.5
backports.shutil-get-terminal-size==1.0.0
backports.ssl-match-hostname==3.5.0.1
batinfo==0.4.2
beautifulsoup4==4.5.3
bencode==1.0
bernhard==0.2.4
bottle==0.12.13
cassandra-driver==3.7.1
certifi==2017.1.23
cffi==1.9.1
chardet==2.3.0
click==6.7
CodernityDB==0.5.0
colorama==0.3.7
constantly==15.1.0
convertdate==2.1.0
CouchDB==1.1
cryptography==1.7.2
cssselect==1.0.1
cycler==0.10.0
daemon==1.1
dateparser==0.5.1
decorator==4.0.11
Django==1.10.5
docker-py==1.10.6
docker-pycreds==0.2.1
dogpile.cache==0.6.2
elasticsearch==5.1.0
enum-compat==0.0.2
enum34==1.1.6
enzyme==0.4.1
ephem==3.7.6.0
first==2.0.1
functools32==3.2.3.post2
future==0.16.0
futures==3.0.5
Glances==2.8.2
gntp==1.0.3
guessit==2.1.1
hgapi==1.7.2
html5lib==0.999999999
httplib2==0.10.2
idna==2.2
incremental==16.10.1
influxdb==4.0.0
ipaddress==1.0.18
ipython==5.2.2
ipython-genutils==0.1.0
jdatetime==1.8.1
Logr==0.2.2
Markdown==2.6.8
MarkupSafe==0.23
matplotlib==2.0.0
mysql-connector-python==2.1.5
ndg-httpsclient==0.4.2
netifaces==0.10.5
numpy==1.12.0
nvidia-ml-py==7.352.0
oauth2==1.9.0.post1
packaging==16.8
parse==1.6.6
parsedatetime==2.2
path.py==10.1
pathlib==1.0.1
pathlib2==2.2.1
pbkdf2==1.3
pbr==1.10.0
peewee==2.8.5
pefile==2016.3.28
pickleshare==0.7.4
pika==0.10.0
pip-init==0.3.0
pip-review==0.5.2
pip-tools==1.8.0
plumbum==1.6.3
ply==3.10
potsdb==1.0.3
prompt-toolkit==1.0.13
protobuf==3.2.0
psutil==5.1.2
pur==4.0.1
py-cpuinfo==0.2.3
pyasn1==0.2.1
pycparser==2.17
pycrypto==2.6.1
pycryptodome==3.4.4
PyDispatcher==2.0.5
Pygments==2.2.0
PyInstaller==3.2.1
pymdstat==0.4.2
PyNMA==1.0
pynzb==0.1.0
pyOpenSSL==16.2.0
pyparsing==2.1.10
pypiwin32==219
pyrencode==0.0.4
PyRSS2Gen==1.1
pysmi==0.0.7
pysnmp==4.3.3
pysrt==1.1.1
pystache==0.5.4
python-dateutil==2.6.0
pytz==2016.10
pyutil==2.0.0
pywin==0.3.1
pywin32==220
PyYAML==3.12
pyzmq==16.0.2
qtfaststart==1.8
queuelib==1.4.2
rarfile==3.0
rebulk==0.8.2
regex==2017.1.17
requests==2.13.0
requests-cache==0.4.13
rpyc==3.3.0
ruamel.ordereddict==0.4.9
ruamel.yaml==0.13.13
scandir==1.4
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
speedtest-cli==1.0.2
SQLAlchemy==1.1.5
statsd==3.2.1
stevedore==1.20.0
subliminal==2.0.5
suds==0.4
terminaltables==3.1.0
tornado==4.4.2
torrt==0.4.2
traitlets==4.3.1
tus==1.0a1
Twisted==16.6.0
typing==3.5.3.0
tzlocal==1.3
umalqurra==0.2
unrar==0.3
urllib3==1.20
vboxapi==1.0
virtualenv==15.1.0
w3lib==1.16.0
wcwidth==0.1.7
webencodings==0.5
websocket-client==0.40.0
Werkzeug==0.11.15
wget==3.2
wifi==0.3.8
win-unicode-console==0.5
zbase32==1.1.5
zeroconf==0.18.0
zope.interface==4.3.3
zxcvbn-python==4.4.14

[pfmoore]

@seanvree OK. I wasn't implying that you didn't know what you were doing, just trying to follow what's happening. The output you gave is useful. From the python -m pip freeze output, pip is clearly working, which is good.

You say you try python2.7 -m pip freeze > requirements.txt. That should fail immediately, with "Command not found", because Python doesn't supply a versioned executable on Windows. The fact that it doesn't hang implies that somewhere on your PATH you have a python2.7 command, probably python2.7.bat Whatever's in that is likely what is causing your issues - you could try to diagnose why it's not working but honestly I'd recommend just getting rid of it and using the unversioned python command. If you do want to diagnose, then I'd need to know the contents of python2.7.bat and the contents of the requirements.txt file you say is created.

The one puzzle I'm still having trouble with is why python -m pip freeze | % { ($_ -split '==')[0] } would not work in Powershell. I don't know if you know Powershell at all, but the first part is just the pip command you showed working above. You're just piping that output into a Powershell cmdlet % which is an alias for ForEach-Object that is splitting each line on == and returning the first component. For that to not work if the pip freeze part is working is weird. I assume your Powershell environment is the same as your CMD environment, so it's the same python being executed in both cases (no weird aliasing or path changes in your powershell profile)? And python -m pip freeze gives the same output in powershell as in cmd?

[seanvree]

@pfmoore - if you have python for windows installed and add it to your PATH, then you can run all py commands natively by simply typing "PYTHON"

Again, it's NOT a bat file sir. I can simply type "PYTHON" in CMD window/PS and I get the output as follows:

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\WINDOWS\system32>python
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

Yes, I get the same output in PS as in CMD.

[pfmoore]

@seanvree I understand that (I've used Python on Windows for many years now). But you said you did python2.7 -m pip freeze. There is no python2.7 command supplied with the Python 2.7 Windows distribution. I say this with certainty - even if I wasn't already sure, I've checked my own Python 2.7 installation to confirm.

So the question remains, where did you get that python2.7 command from? You need to check each directory in PATH, plus your current directory to see if there's a python2.7.{exe, com, cmd, bat} file in there. If the python2.7 command also works in Powershell, you can use

get-command python2.7 | Select-Object Source

to show the location of the file that the shell is using for python2.7.

[seanvree]

@pfmoore there is if you install this:

https://github.com/davidmarble/pywin

The "Python2.7" command simply allows you to run diff python commands for diff python versions installed on your box. which I do.

So, I can run "python2.7 -m pip ext etc" and that would run pip under 2.7

"Python3.6 -m pip etc etc" that would invoke the command for the 3.6 install.

If I run If I run get-command python2.7 | Select-Object Source I get the following output:

Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\Users\Administrator> get-command python2.7 | Select-Object Source

Source

C:\Python27\python2.7.bat

PS C:\Users\Administrator>

That .bat file shows:

@C:\Python27\python.exe %*

[pfmoore]

Ah, OK. That's your problem then. You do know that as python2.7.bat is a bat file, you can't call it from another bat file unless you do call python2.7? Because by default bat files chain, they don't nest.

[seanvree]

Right, that's actually what I was doing originally when it stopped working.

So, I should be able to do:

call python2.7 -m pip freeze --local | grep -v '^-e' | cut -d = -f 1 | xargs -n1 pip install -U

Right? Because it still hangs.

[pfmoore]

Well, grep, cut and xargs still aren't Windows programs. And I know xargs tends to be flaky (it depends on the build - talk to whoever supplied your port of xargs). And The single quotes in the grep command arguments I already commented on.

I think I've gone as far as I can here. It's still clear to me that this isn't a pip issue, so you should look for help from a Windows forum, or from whoever supplied the various utilities you are using. Hope I've been of some help, at least.

[seanvree]

@pfmoore -

Thanks so much for your help. I respectfully disagree with you, however. I know it's not an input/command issue. Been using the exact same commands for 2 years with no issues and as soon as I updated the entire python/pip package, something broke.

Cheers sir.

[pfmoore]

I respectfully disagree with you, however.

OK, well if you can demonstrate the pip issue without side issues of non-standard batch launchers and 3rd party commands, please feel free to reopen the issue with a reproducible test case. At the moment, I can't see from your report any evidence of a pip issue that we can action.

[seanvree]

@pfmoore -

Okay, fair. So let me simply ask you this:

How do you update all pip packages on windows with one or two commands?

[pfmoore]

Mostly I don't, because some are hard to do (need compilers, come from Christoph Gohlke's site). And there's an ongoing debate over possibly adding a pip update-all type of command. But right now, if I did do a bulk update, I'd do something like (in powershell)

pip install -U $(pip list --outdated --format=freeze | % { $_ -replace '=.*','' })

I just did precisely that now, and it worked perfectly. In practice, I mostly just do pip list --outdated and manually grab the list of packages I care about updating.

Note though that blindly updating like this risks breaking dependencies, as pip doesn't do a full recursive dependency analysis/check when doing this sort of update (there's open issues around this under titles like "pip needs a dependency resolver" - a quick search found #988 and #2716, but there may be more - work is ongoing). You should test your installation after doing this.

[seanvree]

@pfmoore - Yup. got it. That command does not work from PS. Same thing, just hangs.

[pfmoore]

Weird. It works for me so I guess there's something up with your environment. Sorry I've no idea what it might be.