requests 2.14.0 cannot be installed using pip < 8.1.2

[lmazuel]

Example with pip 6.1.1 (but same with pip 8.1.1):

> pip install requests
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting requests
  Using cached requests-2.14.0-py2.py3-none-any.whl
  Exception:
  Traceback (most recent call last):
    File "D:\VEnvs\testpip\lib\site-packages\pip\basecommand.py", line 246, in main
      status = self.run(options, args)
    File "D:\VEnvs\testpip\lib\site-packages\pip\commands\install.py", line 342, in run
      requirement_set.prepare_files(finder)
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 345, in prepare_files
      functools.partial(self._prepare_file, finder))
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 290, in _walk_req_to_install
      more_reqs = handler(req_to_install)
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 557, in _prepare_file
      set(req_to_install.extras) - set(dist.extras)
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2758, in extras
      return [dep for dep in self._dep_map if dep]
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2781, in _dep_map
      self.__dep_map = self._compute_dependencies()
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2814, in _compute_dependencies
      common = frozenset(reqs_for_extra(None))
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2811, in reqs_for_extra
      if req.marker_fn(override={'extra':extra}):
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\_markerlib\markers.py", line 113, in marker_fn
      return eval(compiled_marker, environment)
    File "<environment marker>", line 1, in <module>
  NameError: name 'platform_system' is not defined

So, installing requests 2.14.0 implicitly requires pip >= 9.x. Should be at least in the release note as a disclaimer, or be fixed if it's not on purpose.

[Lukasa]

Yeah, we can add that as a release note. That said, pip 9 is a bit newer than I thought we were going to need: @dstufft, is the platform_system marker not provided before then?

[lmazuel]

I just bisect, and it works starting 8.1.2 (broken in 8.1.1).

[Lukasa]

Ok, so that means it works with a version of pip that's a year old. I think I'm ok with this, though naturally the rest of the dev team should express opinions here. Generally, I'm a bit loathe to be tied down by older pips: generally speaking for most people it's easy to upgrade pip, and if it's hard to upgrade pip they're usually using a distro and so should use a virtualenv to install Requests, at which point they should just install a new pip inside the virtualenv.

Any thoughts on that team? @nateprewitt @sigmavirus24 @kennethreitz?

[Lukasa]

(For those who want the data point, pip 8.1.2 was released one year ago on Thursday).

[acdha]

I just found this issue because the default pip on Travis CI is 6.x (easily fixed with the addition of pip install --upgrade pip but that'll be in every project until they update the base image)

[Lukasa]

Hrm. That's a bit hard for Travis CI to justify: the most recent 6.x release of pip is two years old, which is really a bit much!

[acdha]

@Lukasa I'm opening an issue now

[dstufft]

I'm pretty sure the justifcation is they're Rubyists and don't think much about pip :)

[Lukasa]

@dstufft Do you have an opinion on this, btw?

[dstufft]

I think everyone should always upgrade to the latest pip! Whether enough people have done that for requests to consider it reasonable to depend on it or not, I dunno.

[Lukasa]

Ok, so I decided to get some data. I ran this query over the pip BigQuery data to see what pip version was used to install Requests over the past 30 days. Here are our answers, down to 5k downloads (I don't really care about any number lower than that):

Row	details_installer_name	details_installer_version	total_downloads
1	pip	9.0.1	5020104
2	pip	1.5.4	1271017
3	pip	8.1.1	860806
4	pip	8.1.2	605871
5	pip	8.0.3	500970
6	pip	7.1.2	471446
7	pip	6.1.1	370775
8	pip	1.5.6	317236
9	pip	6.0.8	153005
10	pip	6.0.7	149881
11	pip	7.1.0	78024
12	pip	9.0.0	41436
13	pip	1.4.1	38936
14	pip	8.0.2	34807
15	pip	1.5.5	30253
16	pip	1.5	28426
17	pip	7.0.1	9441
18	pip	7.0.3	8717
19	pip	8.1.0	7126
20	pip	1.5.1	4979

[Lukasa]

So the versions that support these markers are 8.1.2, 9.0.0, and 9.0.1. Together they account for 5,667,411 of our 10,015,055 pip downloads in that time frame. Put another way, 56.6% of our downloads last month came from pip installs that are compatible with the markers.

[Lukasa]

For some extra data points, some of the other large numbers are people using the ancient pips their distros provided. For example, pip 1.5.4 is almost entirely going to be Ubuntu Trusty, with their 1,271,017 downloads. I'm inclined not to worry about them too much: if they're installing Requests with the distro-provided pip all they're going do is break their pip. Same note with 1.5.6 (Debian Jessie), which together provide another 17% or so of downloads that are basically beyond us.

[oberstet]

This also broke our CI a minute ago: so is Travis gonna fix their env or are projects supposed to do an update within their Travis scripts?

[Lukasa]

Ok, so right now with 56% of our downloaders already safe and most of the rest one pip install -U pip away from being ok, I'm pretty tempted to say that we should consider toughing this out. It's going to annoy a whole bunch of people, but if we can drag the ecosystem forward a bit to the point that people can actually meaningfully use environment markers for conditional dependencies that would be one hell of a public service.

[Lukasa]

@oberstet If it were me, I'd add a pip install -U pip to the start of my travis scripts. Travis has been known to be slow with their Python environments at the best of time, and that way you guarantee you'll never encounter a problem with a dependency that does this again in the future. Consider it a forward looking defense against this kind of pain happening again.

[alex]

As a fellow maintainer of OSS projects who wants to use environment markers, <3. It seems like we should also engage with the travis folks to fix this, as that'll reduce a lot of the pain.

[nateprewitt]

@Lukasa, 8.1.1 seems to likely be coming from Ubuntu Xenial too. It seems reasonable for people to be upgrading a version of pip that in some cases is more than 3 years old. I'm on board with waiting this one out as well with the current information. We can't support legacy versions of pip forever.

[oberstet]

@Lukasa I'm fine with that. Travis should fix that, instead of myriads of projects, but what the hell: your advice is pragmatic! =)

[Lukasa]

@oberstet Very much agreed, I'd prefer a world where Travis has some kind of rolling approach to pip versions, and I'm happy to work with them to make that possible if they need outside help. In the meantime, though, we shall do what we can in their absence! 😁

[adamjstewart]

the rest are one pip install -U pip away from being ok

Just to point out, they are also one pip install requests==2.13.0 away from being ok. Seriously, this is going to impact a lot of people who use Travis. Before today I had never even heard of requests, but all of a sudden every unit test for our project is failing because codecov depends on requests. I agree that Travis should install a newer version of pip, but that doesn't mean this is the right way to do it. I would prefer rolling back whatever change requires environment markers until Travis updates pip.

[Lukasa]

@adamjstewart Yeah, that's absolutely true, and some number of projects will probably do that. All we can do is make a decision about at what point we decide to stop supporting the older releases of pip: at what level of usage do we give up on them?

I should note that Travis-CI is included in the above numbers, which means even with Travis-CI less than half of our downloads are currently affected. I don't consider that number to be at a level where I want to immediately revert this.

I'm still keeping this open for discussion: I want the community and the other maintainers to weigh in. @nateprewitt and I have had our say, two maintainers have yet to get involved, and there are many more community folk who should feel free to drop in and tell us what they think. All I am saying is that right now I am open to leaving this as-is.

[sethmlarson]

Let's see how many people's computers we can break.

Seeing this issue made me smile. :) I'm 👍 on keeping it how it is, on with progress!

[blamarvt]

Just to point out, they are also one pip install requests==2.13.0 away from being ok.

To be fair, we ran into this issue when installing a package which depends on an unbounded requests version but I agree that a popular library like this forcing a pip upgrade is probably a good thing but will mess up a lot of people's day (like mine!).

[Lukasa]

This is a common feature of shipping Requests releases: when you have a userbase as large as ours it's basically impossible to avoid breaking someone. This was a little bit bigger than I expected, I admit, but still. 😁

[origamee]

Folks, I upgraded requests using sudo pip install --upgrade requests and now I can't import requests any more. It says:

ImportError: No module named requests

So I've removed the requests module multiple times and re-installed, but nothing works. I'm using python2.7 on Centos release 6.9

[Lukasa]

@ssmasud You must not sudo pip install --upgrade requests on a Linux machine. Centos already installs Requests as a system package, and doing a sudo pip install --upgrade requests will overwrite that system package with a newer version that is not compatible with the other packages you install.

On Linux, if you are pip installing Requests to get a new version you must use a virtualenv.

[origamee]

ok @Lukasa so how do I make the requests module functional again?

[origamee]

or in short, how do i repair python2.7 installation on centos?

[ncoghlan]

@ssmasud This is probably a more appropriate question for Ask Fedora or Server Fault, but the key step is to ask yum to restore the system version:

sudo yum reinstall -y python-requests

If that doesn't fix the problem, you may also need to reinstall the system versions of request's dependencies (it's probably a good idea to do that anyway, just to be on the safe side):

python-backports
python-backports-ssl_match_hostname
python-six
python-urllib3

[rfleschenberg]

I am not very familiar with centos, but it might be a good idea to have a look at your /usr/local/lib/python* tree, or check where pip installed requests to and clean that up.

[origamee]

Thanks guys. Looks like pip2.7 install <whatever> did the trick