Crashes if pip-upgrader exists in requirements.txt #4

eddie-dunn · 2017-06-16T13:53:21Z

% pip-upgrader
<snip>
Traceback (most recent call last):
  File "/home/username/.virtualenvs/venv/bin/pip-upgrade", line 11, in <module>
    sys.exit(main())
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/cli.py", line 70, in main
    packages, options.get('--use-default-index')).detect_available_upgrades(options)
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 126, in detect_available_upgrades
    package_status, reason = self._fetch_index_package_info(package_name, current_version)
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 167, in _fetch_index_package_info
    return self._parse_simple_html_package_info(package_name, current_version, response)
  File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 228, in _parse_simple_html_package_info
    latest_version = max([vers for vers in all_versions if not vers.is_prerelease and not vers.is_postrelease])
ValueError: max() arg is an empty sequence

% pip-upgrade --version
1.4.0
% python --version
Python 3.5.3

You might not want to support running pip-upgrader on itself, but I don't think the app should crash because someone has put it in requirements.txt.

The text was updated successfully, but these errors were encountered:

simion · 2017-06-16T14:15:54Z

I have it in the requirements and it works.
Please paste yoyr requirements.txt here or email it to me at simion.agv@gmail.com and i will check which package crashes. Also, this is a good opportunity to catch all upgrade errors for a single package, print them to console and continue upgrading with the rest of the packages.
This way, if any other strange package is added, you won't be blocked.

eddie-dunn · 2017-06-16T18:11:30Z

Ah, the problem was that I generated the file with pip freeze > requirements.txt, which locked pip-upgrader to a specific version.

requirements.txt:

...
pip-upgrader==1.4.0
...

crashes.

requirements.txt:

...
pip-upgrader
...

works.

simion · 2017-06-16T19:01:07Z

Still wierd, it shouldn't crash.

simion · 2017-06-16T19:01:29Z

I'm going to release a fix for it anyway.

eddie-dunn · 2017-06-16T19:22:43Z

Could you reproduce the bug? For some reason

[vers for vers in all_versions if not vers.is_prerelease and not vers.is_postrelease]

produces an empty list. Adding a guard clause before running max() on it should work, but perhaps the underlying issue is triggered by the filtering done on the previous lines.

simion · 2017-06-16T21:34:09Z

Released version v1.4.1 with some error catching, please check it out.

simion · 2017-06-16T21:34:44Z

I cannot reproduce it 😞

eddie-dunn · 2017-06-17T17:10:50Z

I can't reproduce it on my laptop at home either :/

I.e., both versions worked fine with pip-upgrader==1.4.0 in requirements.txt. I will see if I can find the root cause once I get back to work on Monday.

simion · 2017-06-17T20:16:57Z

Cool, thanks. If it breaks at work, it should print the problematic package and continue.
For this, you'll have to upgrade pip-upgrader first.
pip-upgrade -p pip-upgrader

eddie-dunn · 2017-06-19T10:17:52Z

The problematic package is pip-upgrade, in my venv at work. I've done some digging and it seems the line

versions_match = re.findall(pattern, response.content.decode('utf-8'), flags=re.IGNORECASE)

produces an empty list for the following response.content string

<html><head><title>Simple Index</title><meta name="api-version" value="2" /></head><body>
<a href="../../packages/e7/1c/5621b74ed130a6c6331865d4e05c6be69e3a703dd8a2a6e672af15156120/pip_upgrader-0.1.0-py2.py3-none-any.whl#md5=13cd80dd4e0abfbf711626cdef323b08" rel="internal" >pip_upgrader-0.1.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/7f/95/2a8f3fbad56f0c044fab9d196b4d53906c892239223d0c0d271561993305/pip_upgrader-0.1.0.tar.gz#md5=9c9303e83734d99dc692a496217296a8" rel="internal" >pip_upgrader-0.1.0.tar.gz</a><br/>
<a href="../../packages/8b/29/2b7839745e5c977fe3644cb2ac0c98e1bec546677587cfbaa96951dd1eb7/pip_upgrader-0.1.1-py2.py3-none-any.whl#md5=b97e169e6b9c03b222837a98655ec9f2" rel="internal" >pip_upgrader-0.1.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/30/a7/88ca8de3f2cbb2a3f53d6bb7d5c83d470eb205c0900f104c420be0911e0f/pip_upgrader-0.1.1.tar.gz#md5=e2b386ea95010de240625c4b63a94ae5" rel="internal" >pip_upgrader-0.1.1.tar.gz</a><br/>
<a href="../../packages/18/86/c34e458362282dfcd704c70d54f4432dc84779b706c29a4413128aa8736b/pip_upgrader-1.0.0-py2.py3-none-any.whl#md5=cd43d3fe7e5ef64e0a084e086360b87f" rel="internal" >pip_upgrader-1.0.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a8/4c/1b0fd5da99ff288f9954c8abc5ea0b08496bbdf361619fd15ae6cdd004bd/pip_upgrader-1.0.0.tar.gz#md5=75b70da482de63fc6dd0fe9e5fabc3f8" rel="internal" >pip_upgrader-1.0.0.tar.gz</a><br/>
<a href="../../packages/4b/a2/bd86365aff1f1ab5c1c47bc5f0311b3a94d151395314cc192acb80918a82/pip_upgrader-1.0.1-py2.py3-none-any.whl#md5=ce7eec58c7cde9f9c708c74ebd77447a" rel="internal" >pip_upgrader-1.0.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a2/e9/8cc706ef5f97ebbaed3074f1eb87bd453ec6e03953135f7d8887243ee711/pip_upgrader-1.0.1.tar.gz#md5=fb3ee8cb0c214e19add0e017b08864d0" rel="internal" >pip_upgrader-1.0.1.tar.gz</a><br/>
<a href="../../packages/d5/54/73e845a3fd5eaa886bb514420e49c819e06b6dde965fd44640c48f5fe014/pip_upgrader-1.1.0-py2.py3-none-any.whl#md5=6c954879e2bc68bcd3291fbef0041f7c" rel="internal" >pip_upgrader-1.1.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/33/bd/f89d891fdb4cf0bd1ea9403cfd50735f8d51d89d4fbbea122fa8b7771990/pip_upgrader-1.1.0.tar.gz#md5=2fcb2ffa4b681ce99e4779ae5abe0d20" rel="internal" >pip_upgrader-1.1.0.tar.gz</a><br/>
<a href="../../packages/69/4f/b0fa5f234b7be7a9bfb081713101e50db8f9353a7b6aec81601f65751b0d/pip_upgrader-1.2.0-py2.py3-none-any.whl#md5=27a85b7027de08fbdf19dfc8f97a45f3" rel="internal" >pip_upgrader-1.2.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/82/03/e8992bb500bdd9e228b898315fde69b6e93ef6182a54b89deb5a908d6985/pip_upgrader-1.2.0.tar.gz#md5=e7da3594fc6f79a65ea09186a71c411e" rel="internal" >pip_upgrader-1.2.0.tar.gz</a><br/>
<a href="../../packages/87/b7/47293388a134d651bfb00983a7ac6bd57093f6513dc77a1d0b4abb9761cf/pip_upgrader-1.3.0-py2.py3-none-any.whl#md5=27fb3f805ec291336379817192c673e0" rel="internal" >pip_upgrader-1.3.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/5c/ef/e00b75c4d933ddfd80319f788f3a266a96f474983e237ae150cd234fcdc0/pip_upgrader-1.3.0.tar.gz#md5=c1c0c8388a3419859fb7b152adfd005b" rel="internal" >pip_upgrader-1.3.0.tar.gz</a><br/>
<a href="../../packages/c4/b2/f4b3cf8b93ea34972a181855026021ae7ab41985280c8873baa0cc31965f/pip_upgrader-1.3.3-py2.py3-none-any.whl#md5=6a72e786c29e782b64b6d3b90554df4e" rel="internal" >pip_upgrader-1.3.3-py2.py3-none-any.whl</a><br/>
<a href="../../packages/90/b9/32bc955ab0b9ef7287db5d3422baac0019ec1a3023dd84506c9aa21524a1/pip_upgrader-1.3.3.tar.gz#md5=7f31f9f1184eceac90d65d3134d877cd" rel="internal" >pip_upgrader-1.3.3.tar.gz</a><br/>
<a href="../../packages/62/f7/4d9609f1b4b98d7f4e7edaabc57e0d812985c198f8c90c45c46197871067/pip_upgrader-1.3.5-py2.py3-none-any.whl#md5=ab83e627298a0a16f4ade8b0aa4024c2" rel="internal" >pip_upgrader-1.3.5-py2.py3-none-any.whl</a><br/>
<a href="../../packages/24/f2/4f6262ccd26692cda30e41aa40f44d338fd7f6744a699260ae62692bd99f/pip_upgrader-1.3.5.tar.gz#md5=f1c329fdcc5f3129dd4b2306c6615ffd" rel="internal" >pip_upgrader-1.3.5.tar.gz</a><br/>
<a href="../../packages/90/41/55fd034f13780c9ced6220e23a8b286b8577e77c323cfc7502c39d210525/pip_upgrader-1.4.0-py2.py3-none-any.whl#md5=ab7d4df56b1dc98dd00926793f227511" rel="internal" >pip_upgrader-1.4.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a2/aa/bbec7df19a29c2d3ded829c136116e50d0c2f86287b24fb4162aa9521db8/pip_upgrader-1.4.0.tar.gz#md5=1b401ef39ba8df949427fc422518bd55" rel="internal" >pip_upgrader-1.4.0.tar.gz</a><br/>
<a href="../../packages/1f/44/9135c9e4434ffd057e4b5fbed7af57dafd2dcbcc801322f50a2e85d83fe5/pip_upgrader-1.4.1-py2.py3-none-any.whl#md5=ecc1c675db23044b9ba6c9891ed30ad3" rel="internal" >pip_upgrader-1.4.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/62/aa/024a4613dca987ca5b2fc00b99343893ccb40e07478945162b4992b81917/pip_upgrader-1.4.1.tar.gz#md5=71e46108739d6e7a15d91844707e4edf" rel="internal" >pip_upgrader-1.4.1.tar.gz</a><br/>
<a href="../../packages/8d/dc/db9b7612d7df5805c440a5a01c66d902e3f8e0f4ed884d2da0b46099a10e/pip_upgrader-1.4.dev1-py2.py3-none-any.whl#md5=63df490e7dd62c6fb1e2257c70519832" rel="internal" >pip_upgrader-1.4.dev1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/0b/84/daab498858f39b0ddbde46c1744137c53c4b78ccc0a52ad0d091acd096c2/pip_upgrader-1.4.dev1.tar.gz#md5=4c8a906b77315fda56b42cd2effdc5b7" rel="internal" >pip_upgrader-1.4.dev1.tar.gz</a><br/>
</body></html>

I'm no regex guru, but I think the problem is that the html blob only refers to pip_upgrader, when the package_name variable is pip-upgrader. I.e., one has a dash, the other an underscore.

Here's my requirement.txt

apipkg==1.4
astroid==1.5.3
certifi==2017.4.17
chardet==3.0.4
colorclass==2.2.0
coverage==4.4.1
decorator==4.0.11
docopt==0.6.2
execnet==1.4.1
idna==2.5
isort==4.2.15
lazy-object-proxy==1.3.1
mccabe==0.6.1
packaging==16.8
pbr==3.0.1
pep8==1.7.0
pika==0.10.0
pip-upgrader==1.4.1
py==1.4.34
pygerrit2==2.0.2
pylint==1.7.1
pylint-mccabe==0.1.3
pyparsing==2.2.0
pytest==3.1.2
pytest-cache==1.0
pytest-cov==2.5.1
pytest-datadir-ng==1.1.0
pytest-mccabe==0.1
pytest-pep8==1.0.6
pytest-pylint==0.7.1
requests==2.18.1
requests-mock==1.3.0
retry==0.9.2
six==1.10.0
terminaltables==3.1.0
urllib3==1.21.1
wrapt==1.10.10

I can reproduce it by running the following commands

$ mkvirtualenv foo
<snip>
$ pip install -r requirements.txt  # the requirements.txt from above
<snip>
$ pip-upgrade
Found valid requirements file(s): 
requirements.txt
Setting API url to https://artifacts.domain.com/artifactory/api/pypi/all-pypi/simple/{package} as found in /home/username/.config/pip/pip.conf. Use --default-index-url to use pypi default index
1/37: apipkg ... up to date: 1.4
2/37: astroid ... up to date: 1.5.3
3/37: certifi ... up to date: 2017.4.17
4/37: chardet ... up to date: 3.0.4
5/37: colorclass ... up to date: 2.2.0
6/37: coverage ... up to date: 4.4.1
7/37: decorator ... up to date: 4.0.11
8/37: docopt ... up to date: 0.6.2
9/37: execnet ... up to date: 1.4.1
10/37: idna ... up to date: 2.5
11/37: isort ... up to date: 4.2.15
12/37: lazy-object-proxy ... up to date: 1.3.1
13/37: mccabe ... up to date: 0.6.1
14/37: packaging ... up to date: 16.8
15/37: pbr ... up to date: 3.0.1
16/37: pep8 ... up to date: 1.7.0
17/37: pika ... up to date: 0.10.0
error while parsing version
19/37: py ... up to date: 1.4.34
20/37: pygerrit2 ... up to date: 2.0.2
21/37: pylint ... up to date: 1.7.1
22/37: pylint-mccabe ... up to date: 0.1.3
23/37: pyparsing ... up to date: 2.2.0
24/37: pytest ... up to date: 3.1.2
25/37: pytest-cache ... up to date: 1.0
26/37: pytest-cov ... up to date: 2.5.1
27/37: pytest-datadir-ng ... up to date: 1.1.0
28/37: pytest-mccabe ... up to date: 0.1
29/37: pytest-pep8 ... up to date: 1.0.6
30/37: pytest-pylint ... up to date: 0.7.1
31/37: requests ... up to date: 2.18.1
32/37: requests-mock ... up to date: 1.3.0
33/37: retry ... up to date: 0.9.2
34/37: six ... up to date: 1.10.0
35/37: terminaltables ... up to date: 3.1.0
36/37: urllib3 ... up to date: 1.21.1
37/37: wrapt ... up to date: 1.10.10
All packages are up-to-date.

Upgrade interrupted.
$ python -V
Python 3.5.3
$ pip-upgrade --version
1.4.1

eddie-dunn · 2017-06-19T10:19:56Z

Why this doesn't happen on my laptop, or for you at all, I cannot say.

simion · 2017-06-19T19:59:59Z

I think it's because at work you have a custom package index (artifact) configured in ~/.config/pip/pip.conf.

After upgrading pip-upgrade (use --default-index-url if you still have errors), does it work? It should print the error in console, if any.

eddie-dunn · 2017-06-20T09:19:28Z

Yes, you are correct; that worked with no issue.

At work we should generally go through the custom index (which should pull requested packages from PyPI if they don't exist). In other words, do you have any idea why the custom index might cause the issue?

If not, that's OK. The workaround is simple enough that I can use it instead. But a fix would be nice, because unless it's some kind of misconfiguration of the index on our side, others might encounter the problem as well.

simion · 2017-06-20T18:36:08Z

You're right, right now the upgrader does not fallback to pypi when a package is not found.

To be fixed in next release 😉

simion · 2017-06-22T20:36:03Z

The solution for this issue is not complicated, but I need to install artifactory to test it properly :)

eddie-dunn · 2017-07-04T09:18:01Z

Do you know what Artifactory is? It's a commercial paid for product. I'm not sure you need Artifactory to test the issue, but there's a free trial here:
https://www.jfrog.com/artifactory/free-trial/

Maybe a regular private package index would be enough to test it, without having to deal with Artifactory.

simion added the enhancement label Jun 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crashes if pip-upgrader exists in requirements.txt #4

Crashes if pip-upgrader exists in requirements.txt #4

eddie-dunn commented Jun 16, 2017

simion commented Jun 16, 2017

eddie-dunn commented Jun 16, 2017

simion commented Jun 16, 2017

simion commented Jun 16, 2017

eddie-dunn commented Jun 16, 2017

simion commented Jun 16, 2017

simion commented Jun 16, 2017

eddie-dunn commented Jun 17, 2017

simion commented Jun 17, 2017

eddie-dunn commented Jun 19, 2017 •

edited

eddie-dunn commented Jun 19, 2017

simion commented Jun 19, 2017

eddie-dunn commented Jun 20, 2017

simion commented Jun 20, 2017

simion commented Jun 22, 2017

eddie-dunn commented Jul 4, 2017

Crashes if pip-upgrader exists in requirements.txt #4

Crashes if pip-upgrader exists in requirements.txt #4

Comments

eddie-dunn commented Jun 16, 2017

simion commented Jun 16, 2017

eddie-dunn commented Jun 16, 2017

simion commented Jun 16, 2017

simion commented Jun 16, 2017

eddie-dunn commented Jun 16, 2017

simion commented Jun 16, 2017

simion commented Jun 16, 2017

eddie-dunn commented Jun 17, 2017

simion commented Jun 17, 2017

eddie-dunn commented Jun 19, 2017 • edited

eddie-dunn commented Jun 19, 2017

simion commented Jun 19, 2017

eddie-dunn commented Jun 20, 2017

simion commented Jun 20, 2017

simion commented Jun 22, 2017

eddie-dunn commented Jul 4, 2017

eddie-dunn commented Jun 19, 2017 •

edited