Cannot install pexpect 4.0 with python 2.7

[martinsmid]

Maybe this is just an docs issue (http://pexpect.readthedocs.org/en/stable/install.html#requirements), however pexpect >=4.0 does not work with python 2.7, because it uses a python 3.3 syntax (yield from).
It is not clearly stated in docs and the install fails with a syntax error.

Installing collected packages: pexpect
  Running setup.py install for pexpect
      File "/home/smido/projects/pexeso-testing-indexer/env/lib/python2.7/site-packages/pexpect/async.py", line 16
        transport, pw = yield from asyncio.get_event_loop()\
                                 ^
    SyntaxError: invalid syntax

https://github.com/pexpect/pexpect/blob/master/pexpect/async.py#L16

[takluyver]

It shows the error, but the install should have actually completed (at least, this is what I've found before). The asyncio integration is only usable on Python 3.3 and above, but it shouldn't try to load that module unless you're using it, so it shouldn't be a problem.

I don't know of any way to prevent Python 2 from complaining on installation, but as far as I know it doesn't have any actual effect.

[jquast]

@martinsmid upgrade pip and the error goes away. pypa/pip#1984

[jquast]

(also -- not an error! your installation is fine, enjoy !)

[ygbourhis]

It does install OK in pip but only in pip and not in deb/rpm packages.
rpm/deb pcakges need to build the pyc files in post install scripts.
so

python -m compileall $DESTDIR  # destination dir in site-packages

Fails

it is compulsory for distribution packages to have python -m compileall succeed without errors.

[JaviMerino]

For packagers of pexpect, this is a real bug, so please reopen it until it is properly fixed.

[takluyver]

I'm sorry, but we don't consider this a bug. There is a module that can, by design, only be loaded in Python 3. This is not a problem for the primary mechanisms used to distribute and install Python modules. Where other packaging systems have a problem with this, packagers will need to work around it - for instance, by removing async.py from the Python 2 package before byte-compilation.

We distribute a single package for Python 2 and 3 (and we don't want to change that), so we can't remove the file from our packages.

[tomspur]

@ignatenkobrain did exactly that in Fedora:
Remove async.py after installing the module but before byte-compiling.

[JaviMerino]

Ok, I've done the same for Debian. What I don't like is that if somebody passes async=True to expect() they get a "File not Found" exception. Can the error be improved to something like NotImplementedError with some text saying that this feature is only available for python 3.3+?

[takluyver]

Yeah, making that something like a not implemented error makes sense.

[swarren]

takiuyver, FWIW, "pip install" is NOT the "primary mechanism used to distribute and install Python modules" on any system with a system package manager (and perhaps even other systems too). Being hostile to the requirements of package managers is not a useful attitude.

[takluyver]

Actually, on my Linux machines, pip is the primary way that I install Python packages ;-).

I'm not hostile to other packaging systems, but there's nothing we can easily do to solve this on our end, whereas it should be easy to for packagers to workaround - by excluding that file from the Python 2 package, or ignoring failures in byte compilation like pip does (since there's no guarantee that every file with a .py extension is valid Python syntax on a particular version of Python).

I'm happy to work with packagers to find constructive solutions, but I've got a bit irate with this particular issue because we've had several duplicate reports of it, and I somewhat resent the suggestion that we should change how we code to accommodate packaging mechanisms that can't cope with one module in a package being Python 3 only.

[swarren]

Actually, on my Linux machines, pip is the primary way that I install Python packages ;-).

Sure, but it's not the primary way of the OS; your choice to use pip is non-standard (at least for system-level installations; mixing distro packages and pip doesn't seem like a good idea to me. Perhaps you use virtualenv for everything though, so that objection isn't relevant).

Besides (and this is the reason I came here) pexpect doesn't (at least appear to) install correctly even using pip. There's a big failure message printed during installation. Sure as per the comments above it turns out you can just ignore that, but I'd hardly call that a good user experience. (BTW "ignoring failures in byte compilation like pip does" is a bit economical with the truth; ignoring the issue would imply that it doesn't spit out error messages that users can see.)

Surely the fact that there are duplicated reports of this issue shows that multiple users are affected by this problem Surely it would be better to solve it than getting irate about people (rightfully) reporting issues they experience?

Finally, if I were a distro packager, after having reported the issue, I would find "I'm sorry, but we don't consider this a bug." to be quite hostile.

Two simple things you could do to avoid users seeing this problem:

a) Mark the latest pexpect as being compatible with Python3 only.

b) Producing two separate packages for Python2 and Python3 installations. I believe I've seen other Python-based projects do this. Assuming you have a script to generate release packages, this should be as simple as doing the same thing for both packages, and then deleting a single file in one case. If you do this, you save every distro maintainer (and person installing using pip on Python2) from having to duplicate that same work.

[takluyver]

pexpect doesn't (at least appear to) install correctly even using pip. There's a big failure message printed during installation.

Recent versions of pip do not show the error message.

a) Mark the latest pexpect as being compatible with Python3 only.

That would be inaccurate. We write and test it to work on Python 2.7, so we're certainly not going to claim that it doesn't. Only the asyncio integration requires Python 3.

b) Producing two separate packages for Python2 and Python3 installations.

1. To create separate tarballs, I believe we would have to upload them as separate names on PyPI. That's possible, but it's extra hassle for no good reason.
2. Distros would have to maintain two separate source packages, rather than building two binary packages from one source, so the benefit is questionable anyway.
3. I want people to be able to use this technique - having some modules only loadable on Python 3 - in other packages. Making some features Python 3 only is a sensible stepping stone between fully supporting Python 2 and completely dropping support. I don't want to set a precedent that developers need to faff around with packaging when they do this. I think it's something reasonable that downstream packaging mechanisms should be able to deal with.

If anyone has constructive thoughts to add to this, please do so. But I am not interested in repeatedly justifying the decision we've made against the same complaints, and if this thread goes on like this, I intend to lock it.

[swarren]

I think it's something reasonable that downstream packaging mechanisms should be
able to deal with.

Please do not that it's not only distro packagers that have to deal with this issue, but every single user who installs the package via pip (apparently only if they're not using the latest version of pip). On Ubuntu 14.04 for example, people will see this issue.

Other than that, OK sure, I'll stop attempting to get this legitimate issue fixed so that users aren't affected by it.

[takluyver]

it's not only distro packagers that have to deal with this issue, but every single user who installs the package via pip (apparently only if they're not using the latest version of pip)

They have to deal with it by ignoring an error message on installation, yes. There is nothing extra for them to do, and they can use Pexpect as normal. The pip developers seemingly considered it a bug that the error was shown when install succeeded, which is why the latest version of pip does not show it.

[tpetazzoni]

Just to mention that this issue is also annoying for the Buildroot project, where we have to remove async.py manually to make the byte-compilation process work with error with Python 2.x.

[ocefpaf]

It would be nice to get an error when trying to use that on Python 2.7. Such a feature would also remove, as free bonus 😉, the packaging headache.