New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jython & latest pip doesn't work #5121

Closed
frenzymadness opened this Issue Mar 28, 2018 · 10 comments

Comments

Projects
None yet
5 participants
@frenzymadness

frenzymadness commented Mar 28, 2018

  • Pip version: 9.0.2, 9.0.3
  • Python version: Jython 2.7.1 (latest)
  • Operating system: Fedora 27 in Docker container

Description:

I am trying to use the latest Jython with the latest pip but I cannot install packages.

What I've run:

In the fresh Docker container with Fedora 27:

# Install pip by ensurepip Jython module
[root@8081f73a9c48 /]# jython -m ensurepip
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-9.0.1 setuptools-28.8.0
# Try to install package - pytest just for example
[root@8081f73a9c48 /]# jython -m pip install pytest
Collecting pytest
  Downloading pytest-3.5.0-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 474kB/s 
Collecting attrs>=17.4.0 (from pytest)
  Downloading attrs-17.4.0-py2.py3-none-any.whl
Collecting pluggy<0.7,>=0.5 (from pytest)
  Downloading pluggy-0.6.0.tar.gz
Collecting py>=1.5.0 (from pytest)
  Downloading py-1.5.3-py2.py3-none-any.whl (84kB)
    100% |████████████████████████████████| 92kB 1.1MB/s 
Collecting six>=1.10.0 (from pytest)
  Downloading six-1.11.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /usr/share/jython/Lib/site-packages (from pytest)
Collecting more-itertools>=4.0.0 (from pytest)
  Downloading more_itertools-4.1.0-py2-none-any.whl (47kB)
    100% |████████████████████████████████| 51kB 867kB/s 
Collecting funcsigs; python_version < "3.0" (from pytest)
  Downloading funcsigs-1.0.2-py2.py3-none-any.whl
Installing collected packages: attrs, pluggy, py, six, more-itertools, funcsigs, pytest
  Running setup.py install for pluggy ... done
Successfully installed attrs-17.4.0 funcsigs-1.0.2 more-itertools-4.1.0 pluggy-0.6.0 py-1.5.3 pytest-3.5.0 six-1.11.0
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

It works! Let's do it again in a new container but upgrade pip before install step:

# Install default version 9.0.1
[root@b5718b55e70d /]# jython -m ensurepip
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-9.0.1 setuptools-28.8.0
# Upgrade to 9.0.3
[root@b5718b55e70d /]# jython -m pip install --upgrade pip
Collecting pip
  Downloading pip-9.0.3-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 379kB/s 
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-9.0.3
# Install pytest
[root@b5718b55e70d /]# jython -m pip install pytest
Collecting pytest
Exception:
Traceback (most recent call last):
  File "/usr/share/jython/Lib/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/share/jython/Lib/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/usr/share/jython/Lib/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/usr/share/jython/Lib/site-packages/pip/req/req_set.py", line 376, in prepare_files
    discovered_reqs.extend(self._prepare_file(
  File "/usr/share/jython/Lib/site-packages/pip/req/req_set.py", line 376, in prepare_files
    discovered_reqs.extend(self._prepare_file(
  File "/usr/share/jython/Lib/site-packages/pip/req/req_set.py", line 617, in _prepare_file
    unpack_url(
  File "/usr/share/jython/Lib/site-packages/pip/req/req_set.py", line 617, in _prepare_file
    unpack_url(
  File "/usr/share/jython/Lib/site-packages/pip/download.py", line 816, in unpack_url
    unpack_http_url(
  File "/usr/share/jython/Lib/site-packages/pip/download.py", line 656, in unpack_http_url
    from_path, content_type = _download_http_url(link,
  File "/usr/share/jython/Lib/site-packages/pip/download.py", line 831, in _download_http_url
    resp = session.get(
  File "/usr/share/jython/Lib/site-packages/pip/download.py", line 831, in _download_http_url
    resp = session.get(
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/adapters.py", line 430, in send
    resp = conn.urlopen(
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/adapters.py", line 430, in send
    resp = conn.urlopen(
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/connectionpool.py", line 589, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/connectionpool.py", line 589, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/connectionpool.py", line 242, in _get_conn
    if conn and is_connection_dropped(conn):
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/connection.py", line 27, in is_connection_dropped
    return bool(wait_for_read(sock, timeout=0.0))
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/connection.py", line 27, in is_connection_dropped
    return bool(wait_for_read(sock, timeout=0.0))
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/wait.py", line 33, in wait_for_read
    return _wait_for_io_events(socks, EVENT_READ, timeout)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/wait.py", line 24, in _wait_for_io_events
    selector.register(sock, events)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 344, in register
    key = super(PollSelector, self).register(fileobj, events, data)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 197, in register
    key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 181, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 50, in _fileobj_to_fd
    raise ValueError("Invalid file object: {0!r}".format(fileobj))
ValueError: Invalid file object: <ssl.SSLSocket object at 0x1a4>

I've tested the same scenario with pip 9.0.2 and it doesn't work as well.

I don't know where the problem comes from so I cannot provide any solution proposal. Let me know if I can help with anything or provide something more.

Jython seems to be supported platform for pip so it would be nice to have it in pip's tox.ini and .travis.yml files to run CI test on them. Bug for this is here from 2015 #3044.

@pfmoore

This comment has been minimized.

Member

pfmoore commented Mar 28, 2018

Hi @frenzymadness - pip version 10 will be getting released in a few weeks' time. Could you check if this is still an issue on the current master branch from github? pip install --upgrade https://github.com/pypa/pip should get the latest development version for you to test.

@frenzymadness

This comment has been minimized.

frenzymadness commented Mar 28, 2018

@pfmoore I tried but unfortunately, the result is the same.

# Install default pip 90.0.1
[root@0848f0877e10 /]# jython -m ensurepip
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-9.0.1 setuptools-28.8.0
# Install upgrade from git repository
[root@0848f0877e10 /]# jython -m pip install --upgrade git+https://github.com/pypa/pip
Collecting git+https://github.com/pypa/pip
  Cloning https://github.com/pypa/pip to /tmp/pip-sffB_Q-build
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Running setup.py install for pip ... done
Successfully installed pip-10.0.0.dev0
# Install pytest
[root@0848f0877e10 /]# jython -m pip install pytest
Collecting pytest
Exception:
Traceback (most recent call last):
  File "/usr/share/jython/Lib/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/usr/share/jython/Lib/site-packages/pip/_internal/commands/install.py", line 286, in run
    resolver.resolve(requirement_set)
  File "/usr/share/jython/Lib/site-packages/pip/_internal/commands/install.py", line 286, in run
    resolver.resolve(requirement_set)
  File "/usr/share/jython/Lib/site-packages/pip/_internal/resolve.py", line 98, in resolve
    discovered_reqs.extend(
  File "/usr/share/jython/Lib/site-packages/pip/_internal/resolve.py", line 98, in resolve
    discovered_reqs.extend(
  File "/usr/share/jython/Lib/site-packages/pip/_internal/resolve.py", line 255, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/share/jython/Lib/site-packages/pip/_internal/resolve.py", line 206, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(
  File "/usr/share/jython/Lib/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement
    unpack_url(
  File "/usr/share/jython/Lib/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement
    unpack_url(
  File "/usr/share/jython/Lib/site-packages/pip/_internal/download.py", line 828, in unpack_url
    unpack_http_url(
  File "/usr/share/jython/Lib/site-packages/pip/_internal/download.py", line 667, in unpack_http_url
    from_path, content_type = _download_http_url(link,
  File "/usr/share/jython/Lib/site-packages/pip/_internal/download.py", line 844, in _download_http_url
    resp = session.get(
  File "/usr/share/jython/Lib/site-packages/pip/_internal/download.py", line 844, in _download_http_url
    resp = session.get(
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/_internal/download.py", line 394, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/cachecontrol/adapter.py", line 54, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/adapters.py", line 430, in send
    resp = conn.urlopen(
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/requests/adapters.py", line 430, in send
    resp = conn.urlopen(
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/connectionpool.py", line 589, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/connectionpool.py", line 589, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/connectionpool.py", line 242, in _get_conn
    if conn and is_connection_dropped(conn):
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/connection.py", line 27, in is_connection_dropped
    return bool(wait_for_read(sock, timeout=0.0))
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/connection.py", line 27, in is_connection_dropped
    return bool(wait_for_read(sock, timeout=0.0))
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/wait.py", line 33, in wait_for_read
    return _wait_for_io_events(socks, EVENT_READ, timeout)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/wait.py", line 24, in _wait_for_io_events
    selector.register(sock, events)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 344, in register
    key = super(PollSelector, self).register(fileobj, events, data)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 197, in register
    key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 181, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/share/jython/Lib/site-packages/pip/_vendor/urllib3/util/selectors.py", line 50, in _fileobj_to_fd
    raise ValueError("Invalid file object: {0!r}".format(fileobj))
ValueError: Invalid file object: <ssl.SSLSocket object at 0x1a0>
@pfmoore

This comment has been minimized.

Member

pfmoore commented Mar 28, 2018

Hmm, this looks like it might be an issue with urllib3, which is vendored in pip. I guess in the first instance it would need to be reported to them.

The following issue seems relevant: requests/requests#3992

@dstufft @pradyunsg Do either of you have any insights here? Is broken Jython support significant enough to block the pip 10 release (I wasn't even aware that we supported Jython :-() I've marked this as a release blocker for now, but the requests bug referenced above has been round for nearly a year and implies that the behaviour of Jython is intended. My feeling is that our only practical option may have to be to accept that we can't support Jython until urllib3/requests finds a workaround for this...

(FWIW, I couldn't find any formal statement from urllib3/requests that they support Jython. Our support for platforms should be conditional on support from our dependencies, but it's hard to make that explicit if they don't state what they support...)

@pradyunsg

This comment has been minimized.

Member

pradyunsg commented Mar 28, 2018

Looking at hg.python.org/jython, I see there's been activity from @jamesmudd, @jeff5 and @Stewori on Jython. Maybe one of them could to provide some insight into this.

As for support from urllib3/requests, there's one mention of Jython in the changelog of urllib3. Same goes for the code though. They seem to support it, sort of. Some inputs from @Lukasa or @shazow would be useful here.


I wasn't even aware that we supported Jython :-(

Same.

Idk how many pip users use Jython. And if they do, I think asking them to pin to pip 9.0.1 is okay, since they're probably using a 2015 release of Jython or bleeding edge, which are both cases which should have dedicated support, which we can't provide.

My feeling is that our only practical option may have to be to accept that we can't support Jython until urllib3/requests finds a workaround for this...

Additionally, I'd say, we'd need to set up the CI and find someone to provide some help with Jython in general, to make it feasible for us to properly support Jython. Until that happens, it's probably gonna be a "best effort" -- if things break, we're sorry and we'd take trivial fixes that un-break your installation; pin to an older version until that fix comes out.

@pradyunsg pradyunsg added the P: jython label Mar 28, 2018

@pfmoore

This comment has been minimized.

Member

pfmoore commented Mar 28, 2018

Yep, #3044 has been waiting for someone to supply a PR for 2 1/2 years. That suggests to me that there's little interest (at least from anyone who'd be around to support and fix Jython issues).

@pfmoore

This comment has been minimized.

Member

pfmoore commented Mar 28, 2018

I wasn't even aware that we supported Jython :-(

Actually, checking the docs, we don't claim support for Jython: https://pip.pypa.io/en/latest/installing/#python-and-os-compatibility

So I guess the answer here is:

Sorry, you should pin to 9.0.1 for now. If our dependencies get a fix for this, we'll incorporate it and subsequent releases of pip should work again with Jython. For long-term support, we'd need someone to step up and offer to work on maintaining Jython support, the first step of which would be to address #3044.

I'm going to remove the "release blocker" label on that basis.

@dstufft

This comment has been minimized.

Member

dstufft commented Mar 28, 2018

I don't think any of our versions work on Jython, I think Jython has ensurepip which they've patched to install a patched version of pip that work on Jython.

@jeff5

This comment has been minimized.

jeff5 commented Mar 28, 2018

I wasn't even aware that we supported Jython :-(

It's probably more accurate to say that Jython supports pip :-) I think you've made the right call (stick with the original version). We'll see if we can do better, but file descriptors are tricky in Jython because of the distance Java puts between us and this low-level OS concept.

Logged at: http://bugs.jython.org/issue2660

@jimbaker knows most about pip in Jython.

@pfmoore

This comment has been minimized.

Member

pfmoore commented Mar 28, 2018

Cool thanks for the clarification. As noted above, we'd be OK with accepting PRs that make Jython support easier, but in this case it's a vendored dependency so I doubt there's much we can do from pip's side.

@frenzymadness

This comment has been minimized.

frenzymadness commented Apr 3, 2018

Thanks a lot for all your comments and clarifications. Everything makes sense. Because the latest (and incompatible) pip is automatically installed into tox virtual environment, I've created an issue also there. I'm gonna create a small reproducer to Jython bug.

I think that this issue can be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment