Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python 2.6 build failure with 0.27 #1879

Closed
totaam opened this issue Sep 23, 2017 · 16 comments
Closed

python 2.6 build failure with 0.27 #1879

totaam opened this issue Sep 23, 2017 · 16 comments

Comments

@totaam
Copy link

totaam commented Sep 23, 2017

tested on CentOS 6:

running build_ext
Traceback (most recent call last):
  File "setup.py", line 278, in <module>
    **setup_args
  File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.6/distutils/command/build.py", line 134, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python2.6/distutils/cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.6/distutils/dist.py", line 994, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib64/python2.6/distutils/cmd.py", line 117, in ensure_finalized
    self.finalize_options()
  File "/usr/src/tmp/BUILD/cython-0.27/Cython/Distutils/build_ext.py", line 19, in finalize_options
    self.distribution.ext_modules)
  File "/usr/src/tmp/BUILD/cython-0.27/Cython/Build/Dependencies.py", line 915, in cythonize
    aliases=aliases)
  File "/usr/src/tmp/BUILD/cython-0.27/Cython/Build/Dependencies.py", line 854, in create_extension_list
    m.sources.remove(file.rsplit('.')[0] + '.c')
ValueError: list.remove(x): x not in list
error: Bad exit status from /var/tmp/rpm-tmp.6StfGq (%build)
    Bad exit status from /var/tmp/rpm-tmp.6StfGq (%build)
@scoder
Copy link
Contributor

scoder commented Sep 23, 2017 via email

@totaam
Copy link
Author

totaam commented Sep 23, 2017

Installing Cython itself with:

python setup.py build

@adembo
Copy link

adembo commented Sep 23, 2017

The Apache Kudu project is also running into the same failure as part of our Python CI build. It's happening on CentOS 6.6, and I think we're using Python 2.6, setuptools 0.6c11, and pip 1.1 (though I'm not sure).

Here's the full error:

  Running setup.py egg_info for package cython
    Unable to find pgen, not compiling formal grammar.
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/build/cython/setup.py", line 278, in <module>
        **setup_args
      File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
        dist.run_commands()
      File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
        cmd_obj.run()
      File "<string>", line 12, in replacement_run
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 252, in find_sources
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 306, in run
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 330, in add_defaults
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/sdist.py", line 203, in add_defaults
      File "/usr/lib64/python2.6/distutils/cmd.py", line 319, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/lib64/python2.6/distutils/cmd.py", line 117, in ensure_finalized
        self.finalize_options()
      File "Cython/Distutils/build_ext.py", line 19, in finalize_options
        self.distribution.ext_modules)
      File "Cython/Build/Dependencies.py", line 915, in cythonize
        aliases=aliases)
      File "Cython/Build/Dependencies.py", line 854, in create_extension_list
        m.sources.remove(file.rsplit('.')[0] + '.c')
    ValueError: list.remove(x): x not in list
    Complete output from command python setup.py egg_info:
    Unable to find pgen, not compiling formal grammar.

running egg_info
creating pip-egg-info/Cython.egg-info
writing pip-egg-info/Cython.egg-info/PKG-INFO
writing top-level names to pip-egg-info/Cython.egg-info/top_level.txt
writing dependency_links to pip-egg-info/Cython.egg-info/dependency_links.txt
writing entry points to pip-egg-info/Cython.egg-info/entry_points.txt
writing manifest file 'pip-egg-info/Cython.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/build/cython/setup.py", line 278, in <module>
    **setup_args
  File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "<string>", line 12, in replacement_run
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 252, in find_sources
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 306, in run
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 330, in add_defaults
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/sdist.py", line 203, in add_defaults
  File "/usr/lib64/python2.6/distutils/cmd.py", line 319, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib64/python2.6/distutils/cmd.py", line 117, in ensure_finalized
    self.finalize_options()
  File "Cython/Distutils/build_ext.py", line 19, in finalize_options
    self.distribution.ext_modules)
  File "Cython/Build/Dependencies.py", line 915, in cythonize
    aliases=aliases)
  File "Cython/Build/Dependencies.py", line 854, in create_extension_list
    m.sources.remove(file.rsplit('.')[0] + '.c')
ValueError: list.remove(x): x not in list

@scoder
Copy link
Contributor

scoder commented Sep 24, 2017

In the case of Kudu, this might happen because it's compiling in C++ mode with a hugely old setuptools version. I committed a fix for the work-around that we use here, could you retry it?
b5a1abf

scoder added a commit that referenced this issue Sep 25, 2017
…ils to apply, but warn about it since it's unclear how to handle such a case.
@scoder scoder added this to the 0.27.1 milestone Sep 26, 2017
@scoder
Copy link
Contributor

scoder commented Sep 26, 2017

I couldn't reproduce the first issue in any way, but I hope that my fixes resolve it, too. Please reopen if not.

@scoder scoder closed this as completed Sep 26, 2017
@scoder
Copy link
Contributor

scoder commented Sep 27, 2017

Fixed in 0.27.1b1. Please give it a try.

@adembo
Copy link

adembo commented Sep 27, 2017

I was finally able to reproduce this. It seems to only reproduce in paths containing a period. @totaam's path was /var/tmp/rpm-tmp.6StfGq and mine was /data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env, so this makes sense.

I think the issue is with this:

# Old setuptools unconditionally replaces .pyx with .c/.cpp
target_file = file.rsplit('.')[0] + ('.cpp' if m.language == 'c++' else '.c')

If I do "/tmp/a.b/foo.pyx".rsplit('.') I get ['/tmp/a', 'b/foo', 'pyx']. Considering just the first entry in that list to be the filename means the filename will be incomplete.

To reproduce:

  1. Find a CentOS 6 or RHEL 6 machine.
  2. Make sure you've got an old version of python-virtualenv. Mine is 1.7.2-2 and ships the following files:
/usr/lib/python2.6/site-packages/virtualenv_support/distribute-0.6.27.tar.gz
/usr/lib/python2.6/site-packages/virtualenv_support/pip-1.1.tar.gz
/usr/lib/python2.6/site-packages/virtualenv_support/setuptools-0.6c11-py2.6.egg
  1. virtualenv /tmp/a.b
  2. source /tmp/a.b/bin/activate
  3. pip install 'cython==0.27'

@adembo
Copy link

adembo commented Sep 27, 2017

BTW, in the same environment I also confirmed that pip install 'cython==0.26.1' doesn't fail, so this is definitely a regression in 0.27.

@scoder
Copy link
Contributor

scoder commented Sep 27, 2017

Ah - sure. Stupid bug. :)

@scoder
Copy link
Contributor

scoder commented Sep 27, 2017

Thanks for figuring it out.

@adembo
Copy link

adembo commented Sep 27, 2017

Thanks for figuring it out.

NP. I spent a few minutes trying to figure out what change in 0.27 caused this, but I couldn't figure it out. Any ideas?

Also, you may be able to revert commits 22d7699 and b5a1abf now too.

@scoder
Copy link
Contributor

scoder commented Sep 27, 2017

I'll be happy to remove the entire code block when we stop supporting Py2.6 (and the corresponding setuptools versions) ...

@hugovk
Copy link
Contributor

hugovk commented Oct 12, 2017

Any plans for stopping supporting Py2.6? It's 4 years since its final security release. https://en.wikipedia.org/wiki/CPython#Version_history

@totaam
Copy link
Author

totaam commented Oct 13, 2017

Any plans for stopping supporting Py2.6?

CentOS 6.x uses python 2.6 and will be "supported in production" until 2020, with "extended lifecycle" until 2024. And it is very widely deployed.
https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Product_life_cycle

@pkittenis
Copy link
Contributor

pkittenis commented Oct 13, 2017

RedHat can sponsor development efforts to keep it supported then :)

Fact is, py 2.6 has not been supported by the Python core team since 2013. Maintaining support for it in third party libraries costs development time. This is not free.

RHE is a paid for product, not a community effort. Support for it is paid for by RedHat customers. Third party community supported libraries have no incentive whatsoever no support an enterprise product for free.

@scoder
Copy link
Contributor

scoder commented Oct 13, 2017

We will not drop support for 2.6 as long as it does not substantially get in the way. The code to support it is there and why should we remove it?

Besides, even when Cython (or any other library) will drop support for 2.6 at some point, that does not force others to follow. Users can happily continue to build their own packages with older Cython versions. After all, they are also using an older Python version, and probably already various outdated libraries. That's perfectly normal in many settings. Always integrating the most recent versions of everything is an immense waste of productive developer time.

And with that being said, I would like to consider this ticker closed. There is no point in further discussing potential future developments of something unrelated here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants