SciPy fails to build on CentOS 6.6

[ThomasMaxwell]

Get the following error building UVCDAT on CentOS release 6.6:

CMake Error at /usr/local/uvcdat-test/build/SCIPY-prefix/src/SCIPY-stamp/SCIPY-configure.cmake:56 (message):
CTEST_FULL_OUTPUT

Command failed: 1
'/usr/local/scratch/tpmaxwel/cmake/bin/cmake' '-DCONFIG_TYPE=.' '-DCDAT_USE_SYSTEM_LAPACK:STRING=OFF' '-DLAPACK_LIBRARIES:STRING=/usr/lib64/liblapack.so' '/usr/lib64/libblas.so' '-DBLAS_LIBRARIES:STRING=/usr/lib64/libblas.so' '-P' '/usr/local/uvcdat-test/build/CMake/SCIPY_configure_step.cmake'
From /usr/local/uvcdat-test/build/SCIPY-prefix/src/SCIPY-stamp/SCIPY-configure-out.log

---

Processing scipy/cluster/_hierarchy.pyx

Cythonizing sources

---

From /usr/local/uvcdat-test/build/SCIPY-prefix/src/SCIPY-stamp/SCIPY-configure-err.log

---

Configuring SCIPY:

/usr/local/uvcdat-test/build/build/SCIPY

Error compiling Cython file:

---

...

  Mark node i as visited.
  """
  bitset[i >> 3] |= 1 << (i & 7)

cpdef calculate_cluster_sizes(double[:, :] Z, double[:] cs, int n):

                                  ^

---

scipy/cluster/_hierarchy.pyx:47:37: Expected an identifier or literal

Traceback (most recent call last):

File "/usr/local/uvcdat-test/build/build/SCIPY/tools/cythonize.py", line 198, in <module>
  main()
File "/usr/local/uvcdat-test/build/build/SCIPY/tools/cythonize.py", line 194, in main
  find_process_files(root_dir)
File "/usr/local/uvcdat-test/build/build/SCIPY/tools/cythonize.py", line 186, in find_process_files
  process(cur_dir, fromfile, tofile, function, hash_db)
File "/usr/local/uvcdat-test/build/build/SCIPY/tools/cythonize.py", line 160, in process
  processor_function(fromfile, tofile)
File "/usr/local/uvcdat-test/build/build/SCIPY/tools/cythonize.py", line 71, in process_pyx
  raise Exception('Cython failed')

Exception: Cython failed

Traceback (most recent call last):

File "setup.py", line 253, in <module>
  setup_package()
File "setup.py", line 246, in setup_package
  generate_cython()
File "setup.py", line 164, in generate_cython
  raise RuntimeError("Running cythonize failed!")

RuntimeError: Running cythonize failed!

CMake Error at
/usr/local/uvcdat-test/build/CMake/SCIPY_configure_step.cmake:58 (message):

Error in config of SCIPY

---

[ThomasMaxwell]

After hitting this error, I sourced the setup_runtime script and then executed:

cd /usr/local/uvcdat-test/build/build/SCIPY
python setup.py install

It then built without error.

[aashish24]

closing it for now until we can reproduce this.

[aashish24]

I am re-opening this issue as @sankhesh was able to reproduce it. @sankhesh how did you solve the problem?

[sankhesh]

I haven't solved the problem. I was able to work around it using @ThomasMaxwell's instructions.

[aashish24]

Okay, in that case, we will keep this issue open. We should fix it properly when we get a chance.

[doutriaux1]

@aashish24 @sankhesh I wonder if we shouldn't edit/get rid of https://github.com/UV-CDAT/uvcdat/blob/master/CMake/cdat_modules_extra/cdat_common_environment.cmake.in and instead tell our build command to source setup_runtime.sh

[doutriaux1]

@sankhesh so i was looking at this tonight. It's interesting if I build with

make -j8 Cython

No eggs, everything happy.
If I build with

make -j8

egg for Cython...
The difference seems to be that in the first case setuptools is NOT built.
I'm testing the solution at:
http://stackoverflow.com/questions/27964960/pip-why-sometimes-installed-as-egg-sometimes-installed-as-files
As we speak, will see if that fixes things.

[doutriaux1]

the good news that both my RH6 bots with make -j12 seem to trigger this error. So we can test our solutions.

[doutriaux1]

the bad news is ${HOME} solution above does not solve it

[doutriaux1]

ok there's hope the following works:

python setup.py install --old-and-unmanageable

as described at: http://stackoverflow.com/questions/18300017/how-to-prevent-setuptools-install-package-as-an-egg

[doutriaux1]

@sankhesh it's probably worth looking at: pypa/pip#2537
@jbeezley I see you commented on the issues related to this link for girder. Any insight? pypa/pip#271

[jbeezley]

I don't think either of those pip issues are relevant here. My guess to what is happening is you are hitting this condition in cython's setup.py. When setuptools isn't found, the cython entry point isn't configured. This causes scipy's install to go here because cython isn't installed the path, which could very well break other packages despite fixing this issue.

The key is to figure out why this works:

subprocess.call([sys.executable, '-c',
'import sys; from Cython.Compiler.Main import '
'setuptools_main as main; sys.exit(main())'] + flags +
["-o", tofile, fromfile])

but this doesn't

load_entry_point('Cython==0.22.1', 'console_scripts', 'cython')()