python 2.6 build failure with 0.27

[totaam]

tested on CentOS 6:

running build_ext
Traceback (most recent call last):
  File "setup.py", line 278, in <module>
    **setup_args
  File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.6/distutils/command/build.py", line 134, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python2.6/distutils/cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.6/distutils/dist.py", line 994, in run_command
    cmd_obj.ensure_finalized()
  File "/usr/lib64/python2.6/distutils/cmd.py", line 117, in ensure_finalized
    self.finalize_options()
  File "/usr/src/tmp/BUILD/cython-0.27/Cython/Distutils/build_ext.py", line 19, in finalize_options
    self.distribution.ext_modules)
  File "/usr/src/tmp/BUILD/cython-0.27/Cython/Build/Dependencies.py", line 915, in cythonize
    aliases=aliases)
  File "/usr/src/tmp/BUILD/cython-0.27/Cython/Build/Dependencies.py", line 854, in create_extension_list
    m.sources.remove(file.rsplit('.')[0] + '.c')
ValueError: list.remove(x): x not in list
error: Bad exit status from /var/tmp/rpm-tmp.6StfGq (%build)
    Bad exit status from /var/tmp/rpm-tmp.6StfGq (%build)

[scoder]

Is this while installing Cython itself, or while compiling some other package with Cython? How did you run the setup script?

[totaam]

Installing Cython itself with:

python setup.py build

[adembo]

The Apache Kudu project is also running into the same failure as part of our Python CI build. It's happening on CentOS 6.6, and I think we're using Python 2.6, setuptools 0.6c11, and pip 1.1 (though I'm not sure).

Here's the full error:

  Running setup.py egg_info for package cython
    Unable to find pgen, not compiling formal grammar.
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/build/cython/setup.py", line 278, in <module>
        **setup_args
      File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
        dist.run_commands()
      File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
        cmd_obj.run()
      File "<string>", line 12, in replacement_run
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 252, in find_sources
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 306, in run
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 330, in add_defaults
      File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/sdist.py", line 203, in add_defaults
      File "/usr/lib64/python2.6/distutils/cmd.py", line 319, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/usr/lib64/python2.6/distutils/cmd.py", line 117, in ensure_finalized
        self.finalize_options()
      File "Cython/Distutils/build_ext.py", line 19, in finalize_options
        self.distribution.ext_modules)
      File "Cython/Build/Dependencies.py", line 915, in cythonize
        aliases=aliases)
      File "Cython/Build/Dependencies.py", line 854, in create_extension_list
        m.sources.remove(file.rsplit('.')[0] + '.c')
    ValueError: list.remove(x): x not in list
    Complete output from command python setup.py egg_info:
    Unable to find pgen, not compiling formal grammar.

running egg_info
creating pip-egg-info/Cython.egg-info
writing pip-egg-info/Cython.egg-info/PKG-INFO
writing top-level names to pip-egg-info/Cython.egg-info/top_level.txt
writing dependency_links to pip-egg-info/Cython.egg-info/dependency_links.txt
writing entry points to pip-egg-info/Cython.egg-info/entry_points.txt
writing manifest file 'pip-egg-info/Cython.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/build/cython/setup.py", line 278, in <module>
    **setup_args
  File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "<string>", line 12, in replacement_run
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 252, in find_sources
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 306, in run
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/egg_info.py", line 330, in add_defaults
  File "/data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg/setuptools/command/sdist.py", line 203, in add_defaults
  File "/usr/lib64/python2.6/distutils/cmd.py", line 319, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/usr/lib64/python2.6/distutils/cmd.py", line 117, in ensure_finalized
    self.finalize_options()
  File "Cython/Distutils/build_ext.py", line 19, in finalize_options
    self.distribution.ext_modules)
  File "Cython/Build/Dependencies.py", line 915, in cythonize
    aliases=aliases)
  File "Cython/Build/Dependencies.py", line 854, in create_extension_list
    m.sources.remove(file.rsplit('.')[0] + '.c')
ValueError: list.remove(x): x not in list

[scoder]

In the case of Kudu, this might happen because it's compiling in C++ mode with a hugely old setuptools version. I committed a fix for the work-around that we use here, could you retry it?
b5a1abf

[scoder]

I couldn't reproduce the first issue in any way, but I hope that my fixes resolve it, too. Please reopen if not.

[scoder]

Fixed in 0.27.1b1. Please give it a try.

[adembo]

I was finally able to reproduce this. It seems to only reproduce in paths containing a period. @totaam's path was /var/tmp/rpm-tmp.6StfGq and mine was /data/jenkins-workspace/kudu-test-cdh5.x/build/debug/py_env, so this makes sense.

I think the issue is with this:

# Old setuptools unconditionally replaces .pyx with .c/.cpp
target_file = file.rsplit('.')[0] + ('.cpp' if m.language == 'c++' else '.c')

If I do "/tmp/a.b/foo.pyx".rsplit('.') I get ['/tmp/a', 'b/foo', 'pyx']. Considering just the first entry in that list to be the filename means the filename will be incomplete.

To reproduce:

1. Find a CentOS 6 or RHEL 6 machine.
2. Make sure you've got an old version of python-virtualenv. Mine is 1.7.2-2 and ships the following files:

/usr/lib/python2.6/site-packages/virtualenv_support/distribute-0.6.27.tar.gz
/usr/lib/python2.6/site-packages/virtualenv_support/pip-1.1.tar.gz
/usr/lib/python2.6/site-packages/virtualenv_support/setuptools-0.6c11-py2.6.egg

3. virtualenv /tmp/a.b
4. source /tmp/a.b/bin/activate
5. pip install 'cython==0.27'

[adembo]

BTW, in the same environment I also confirmed that pip install 'cython==0.26.1' doesn't fail, so this is definitely a regression in 0.27.

[scoder]

Ah - sure. Stupid bug. :)