Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yield from statement is not python 2 compatible #295

Closed
ygbourhis opened this issue Oct 19, 2015 · 14 comments
Closed

yield from statement is not python 2 compatible #295

ygbourhis opened this issue Oct 19, 2015 · 14 comments

Comments

@ygbourhis
Copy link

During a pip install of the package, it fails with this error:

Listing ./.venv/lib/python2.7/site-packages/pexpect ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/ANSI.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/FSM.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/__init__.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/async.py ...
  File "./.venv/lib/python2.7/site-packages/pexpect/async.py", line 16
    transport, pw = yield from asyncio.get_event_loop()\
                             ^
SyntaxError: invalid syntax

Compiling ./.venv/lib/python2.7/site-packages/pexpect/exceptions.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/expect.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/fdpexpect.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/popen_spawn.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/pty_spawn.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/pxssh.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/replwrap.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/run.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/screen.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/spawnbase.py ...
Compiling ./.venv/lib/python2.7/site-packages/pexpect/utils.py ...

This is because the yield from statement is specific to python 3k.
There are two solutions, either remove python 2 compatibility in setup.py, or not using this python 3 specific statement.

@takluyver
Copy link
Member

Duplicate of #290. It installs OK, even though you see an error, and that module is only loaded if you're using the asyncio integration, so it's not a problem. Apparently if you upgrade pip, you'll no longer see the error.

@ygbourhis
Copy link
Author

It does install OK in pip but only in pip and not in deb/rpm packages.
rpm/deb pcakges need to build the pyc files in post install scripts.
so

python -m compileall $DESTDIR  # destination dir in site-packages

Fails

it is compulsory for distribution packages to have python -m compileall succeed without errors.

@takluyver
Copy link
Member

Perhaps the distro packagers should remove that file for the Python 2 package?

@ygbourhis
Copy link
Author

distro packages use the setup.py installer to build packages. Dsitribution packages do not have the capacity to inspect ALL python packages that exist to strip non compatible files. this should be handled in the setup.py

@takluyver
Copy link
Member

For Python packaging it's fine to have a module which is only valid syntax on some versions of Python. If distros have additional requirements, I think its up to them to find ways to deal with that.

@jquast
Copy link
Member

jquast commented Oct 19, 2015

pylint has the same "problem", recommend you investigate what your distribution does with pylint.

(zz)$ python -m compileall `python -c 'import os, pylint; print(os.path.dirname(pylint.__file__))'`
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/checkers ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/pyreverse ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/reporters ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/data ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional ...
Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_abc_methods.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_abc_methods.py", line 6
    class Parent(object, metaclass=abc.ABCMeta):
                                  ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_class_instantiated_py2.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_class_instantiated_py2.py", line 15
    class GoodClass(object, metaclass=abc.ABCMeta):
                                     ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_class_instantiated_py3.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_class_instantiated_py3.py", line 14
    class GoodClass(object, metaclass=abc.ABCMeta):
                                     ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_method_py2.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_method_py2.py", line 35
    class Structure(object, metaclass=abc.ABCMeta):
                                     ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_method_py3.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/abstract_method_py3.py", line 35
    class Structure(object, metaclass=abc.ABCMeta):
                                     ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/class_members_py27.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/class_members_py27.py", line 34
    class TestMetaclass(object, metaclass=ABCMeta):
                                         ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/class_members_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/class_members_py30.py", line 34
    class TestMetaclass(object, metaclass=ABCMeta):
                                         ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/no_name_in_module.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/no_name_in_module.py", line 15
    print('hello world', file=sys.stdout)
                             ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/old_style_class_py27.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/old_style_class_py27.py", line 10
    class NotOldStyle2(metaclass=type):
                                ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/raising_non_exception_py3.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/raising_non_exception_py3.py", line 13
    raise Exc from missing # [raising-non-exception]
                 ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/unbalanced_tuple_unpacking_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/unbalanced_tuple_unpacking_py30.py", line 9
    first, second, *last = (1, 2, 3, 4)
                   ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/undefined_variable_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/undefined_variable_py30.py", line 8
    def test(self)->Undefined: # [undefined-variable]
                  ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/yield_outside_func.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/functional/yield_outside_func.py", line 2
    yield 1  # [yield-outside-function]
SyntaxError: 'yield' outside function

Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input ...
Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_bad_exception_context_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_bad_exception_context_py30.py", line 14
    raise IndexError from 1
                        ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_continue_not_in_loop.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_continue_not_in_loop.py", line 8
    continue
SyntaxError: 'continue' not properly in loop

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_e0108.py ...
SyntaxError: duplicate argument '_' in function definition (func_e0108.py, line 4)

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_exec_used_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_exec_used_py30.py", line 6
    exec('a = 1', globals={})
                         ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_keyword_repeat.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_keyword_repeat.py", line 8
    function_default_arg(two=5, two=7)
SyntaxError: keyword argument repeated

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_kwoa_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_kwoa_py30.py", line 3
    def function(*, foo):
                  ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_noerror_mcs_attr_access.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_noerror_mcs_attr_access.py", line 14
    class Test(object, metaclass=Meta):
                                ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_noerror_unused_variable_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_noerror_unused_variable_py30.py", line 12
    nonlocal attr
                ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_return_outside_func.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_return_outside_func.py", line 3
    return
SyntaxError: 'return' outside function

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_return_yield_mix_py_33.py ...
SyntaxError: 'return' with argument inside generator (func_return_yield_mix_py_33.py, line 8)

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_syntax_error.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_syntax_error.py", line 1
    def toto
           ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_undefined_metaclass_var_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_undefined_metaclass_var_py30.py", line 8
    class Bad(metaclass=ABCMet):
                       ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_unused_import_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_unused_import_py30.py", line 10
    class Meta(metaclass=abc.ABCMeta):
                        ^
SyntaxError: invalid syntax

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_used_before_assignment_py30.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_used_before_assignment_py30.py", line 10
    nonlocal cnt
               ^
SyntaxError: invalid syntax

Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_w0401_package ...
Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_w0705.py ...
  File "/Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/func_w0705.py", line 28
    __revision__ += 1
SyntaxError: default 'except:' must be last

Compiling /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/input/syntax_error.py ...
Sorry: IndentationError: expected an indented block (syntax_error.py, line 2)
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/messages ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/regrtest_data ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/regrtest_data/absimp ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/regrtest_data/package ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/regrtest_data/package/subpackage ...
Listing /Users/docker/.virtualenvs/zz/lib/python2.7/site-packages/pylint/test/regrtest_data/package_all ...

@ygbourhis
Copy link
Author

I'm sorry, but mixing two types of code is wrong if distutils tools don't work.

The following command should work:

python setup.py bdist --format=rpm

But it fails with:
http://paste.openstack.org/show/476714/
Python packages should handle the bdist commands.

please check:
http://pythonhosted.org/setuptools/python3.html
https://docs.python.org/3/howto/pyporting.html

@jquast
Copy link
Member

jquast commented Oct 19, 2015

I am concerned that any accommodations made upstream to allow people to continue to build rpm's for python 2 distutils is not fruitful on this end. This is a battle you will continue with many projects going forward (what do you do about pylint, for example?), and you will have to find a general solution, just as pip has.

Here's what I recommend, downstream, echo into MANIFEST.in during your build,

echo 'exclude pexpect/async.py' >> MANIFEST.in

Just please make sure to exclude such phrase when you finally switch to python3. Python 2to3 tool as suggested by your hyperlinks, and any of the other topics covered are for the reverse direction, how to switch to python3. They do not address the issue of using compileall module on python 3 code that is bi-compatible (we use the six module as suggested, for example -- "yield from" is a statement keyword that cannot be compiled by simple use of six)

@ygbourhis
Copy link
Author

Thanks.

Changing the MANIFEST.in may work but can not be automatized for every package in the wild.
Your solution works like a charm for the package individually, but not for a whole project pulling the package in a venv via pip.
The issue is that many companies like us face the fact that the modules provided by the distro do not match the requirements of our developments, so we have a hurd of Jenkins servers automatically building packages with https://github.com/guilhem/dh-venv and this provides packages with a python virtualenv in them (for deb distros this time instead of rpm) which do not depend on the distro's own version (apart from the python version "only").
Currently it's not possible to inspect each requirement pulled by pip in the venv of the package, so when a package does not compile, we freeze it's version to a lower one which compiles in the requirements.txt of our project. so this makes tons of dh-venv users stop using the evolution of packages (such as pexpect) which do not compile.
In fact the setup.py itself is totally able to exclude the undesired files which will not compile and this should be the path to go. The setup.py is able to, and should, automatically exclude or not files depending on the python version.
And buidling an rpm with bdist is a simple and easy way to check if building a package works, and if building an rpm works, building a deb package with dh-venv will work too .

@jquast
Copy link
Member

jquast commented Oct 20, 2015

I know that setup.py file is capable. In technology, EVERY problem may be solved with code.

I don't feel such code belongs in pexpect. pexpect is meant to solve a specific issue -- I don't feel it is right to join extra code with custom setup.py find_files functions and conditionals for an issue completely unrelated to pexpect (building rpm packages).

I suggest distutils bdist should ignore non-zero exit for compileall error for some files, in the same way pip does. Show me a distutils developer who says "I suggest pexpect project develop a custom find files function to workaround this issue" and I would accept a pull request to do the same.

@jquast
Copy link
Member

jquast commented Oct 20, 2015

And buidling an rpm with bdist is a simple and easy way to check if building a package works

I intended to use tox to test packaging. Tox uses sdist and wheel last I checked. Pretty sure wheel is the future. Producing RPM's is only half the test -- does the rpm work? This raises volunteer costs significantly.

@jquast
Copy link
Member

jquast commented Oct 20, 2015

You're also going to need to find another project and demonstrate how debian and rpm packages are made for such projects (I suggested you inspect the pylint package twice now). You might be suggesting a solution for something where precedent is already determined.

@ygbourhis
Copy link
Author

To answer to your question about "does the rpm work", the answer is "No because it doesn't even build at all"... "python setup.py bdist --format=rpm" gives the traceback I pasted above, and doesn't even produce any rpm. So building an rpm is not possible automatically with distutils and has to be done by manual rpm packaging (meaning that non rpm packagers will not know at all how to do that).

@jquast
Copy link
Member

jquast commented Oct 29, 2015

@ygbourhis I'm sorry I think my statement was unclear and misunderstood. I would really like you to understand why I take this stance: building an rpm successfully is outside the scope of pexpect
project

The custom setup.py code you suggest would be owned inside the pexpect project, and thus, it would become our responsibility to test whether such code is correct. So to re-quote myself,

Producing RPM's is only half the test -- does the rpm work?

To expand: It is my position that all new code is tested, and, the only way to test such code is by using "python setup.py rpmbuild" -- but this is only one half of the testing that would satisfy my standard. We must also ensure the rpm contents are correct!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants