netCDF4.c deleted by setup.py?

[AgilentGCMS]

Hello,

I am trying to compile netcdf4-python on a new system. Before I execute 'python setup.py build', I check what files I have in the root directory:

$ ls
build            constants.pyx   docs      MANIFEST.in  netCDF4.pyx       PKG-INFO         setup.cfg           test
Changelog        COPYING         examples  netCDF4.c    netCDF4_utils.py  README.gh-pages  setup.cfg.template  utils
checkversion.py  create_docs.sh  man       netCDF4.pxi  netcdftime        README.md        setup.py            utils.pyx

Then I try to build:

$ python setup.py build
cython version 0.22 found ...
reading from setup.cfg...
using nc-config ...
using netcdf library version 4.3.0
using Cython to compile netCDF4.pyx...
netcdf lib does not have group rename capability
netcdf lib has nc_inq_path function
netcdf lib does not have nc_inq_format_extended function
running build
running build_py
running build_ext
building 'netCDF4' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/apps/netcdf/4.3.0-intel/include -I/apps/hdf5/1.8.14-intel/include -I/home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include -I/usr/include/python2.6 -c netCDF4.c -o build/temp.linux-x86_64-2.6/netCDF4.o
gcc: netCDF4.c: No such file or directory
gcc: no input files
error: command 'gcc' failed with exit status 1

which seems to me to be a weird error, so I do another ls:

$ ls
build            constants.pyx   docs      MANIFEST.in  netCDF4_utils.py  README.gh-pages  setup.cfg.template  utils
Changelog        COPYING         examples  netCDF4.pxi  netcdftime        README.md        setup.py            utils.pyx
checkversion.py  create_docs.sh  man       netCDF4.pyx  PKG-INFO          setup.cfg        test

and indeed, netCDF4.c seems to have disappeared! I have tried just downloading and putting it there by hand, but every time I do a 'python setup.py build', I get the same error about a missing netCDF4.c, and indeed it's gone!

-Sourish

[jswhit]

setup.py is trying to re-create netCDF4.c from netCDF4.pyx using cython. This must be failing for some reason. Try running

cython netCDF4.pyx

in the terminal, and let us know what the error message is.

[AgilentGCMS]

Hmm... there doesn't seem to be an error in that command:

$ cython netCDF4.pyx  && echo $?
0
$ ls
build            constants.pyx   docs      MANIFEST.in  netCDF4.pyx       PKG-INFO         setup.cfg           test
Changelog        COPYING         examples  netCDF4.c    netCDF4_utils.py  README.gh-pages  setup.cfg.template  utils
checkversion.py  create_docs.sh  man       netCDF4.pxi  netcdftime        README.md        setup.py            utils.pyx
$ python setup.py build
cython version 0.22 found ...
reading from setup.cfg...
using nc-config ...
using netcdf library version 4.3.0
using Cython to compile netCDF4.pyx...
netcdf lib does not have group rename capability
netcdf lib has nc_inq_path function
netcdf lib does not have nc_inq_format_extended function
running build
running build_py
running build_ext
building 'netCDF4' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/apps/netcdf/4.3.0-intel/include -I/apps/hdf5/1.8.14-intel/include -I/home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include -I/usr/include/python2.6 -c netCDF4.c -o build/temp.linux-x86_64-2.6/netCDF4.o
gcc: netCDF4.c: No such file or directory
gcc: no input files
error: command 'gcc' failed with exit status 1

[jswhit]

clearly your cython is not working. Worse, it's silently failing and not producing any C code.

You can work around this by editing setup.py and adding has_cython=False after line 19.

[AgilentGCMS]

But when I run cython from the command prompt, it does produce C code:

$ rm netCDF4.c
$ ls
build            constants.pyx   docs      MANIFEST.in  netCDF4_utils.py  README.gh-pages  setup.cfg.template  utils
Changelog        COPYING         examples  netCDF4.pxi  netcdftime        README.md        setup.py            utils.pyx
checkversion.py  create_docs.sh  man       netCDF4.pyx  PKG-INFO          setup.cfg        test
$ cython netCDF4.pyx  && echo $?
0
$ ls
build            constants.pyx   docs      MANIFEST.in  netCDF4.pyx       PKG-INFO         setup.cfg           test
Changelog        COPYING         examples  netCDF4.c    netCDF4_utils.py  README.gh-pages  setup.cfg.template  utils
checkversion.py  create_docs.sh  man       netCDF4.pxi  netcdftime        README.md        setup.py            utils.pyx
$ ls -l netCDF4.c 
-rw-r--r-- 1 Sourish.Basu co2 3339819 2015 Apr 10 19:38:25 netCDF4.c

So it only doesn't generate C code from within setup.py.

[jswhit]

I get

% python setup.py build
cython version 0.22 found ...
reading from setup.cfg...
using nc-config ...
using netcdf library version 4.3.3-rc2
using Cython to compile netCDF4.pyx...
netcdf lib has group rename capability
netcdf lib has nc_inq_path function
netcdf lib has nc_inq_format_extended function
running build
running build_py
running build_ext
cythoning netCDF4.pyx to netCDF4.c
building 'netCDF4' extension

you seem to be missing the "cythoning" step, which is the message output by cython. I don't know why.

[jswhit]

Sourish - could you try the setup.py in the "cythonize" branch I just created?

https://github.com/Unidata/netcdf4-python/tree/cythonize

[AgilentGCMS]

Same problem:

$ python setup.py build
cython version 0.22 found ...
reading from setup.cfg...
using nc-config ...
using netcdf library version 4.3.0
using Cython to compile netCDF4.pyx...
netcdf lib does not have group rename capability
netcdf lib has nc_inq_path function
netcdf lib does not have nc_inq_format_extended function
running build
running build_py
running build_ext
building 'netCDF4' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/apps/netcdf/4.3.0-intel/include -I/apps/hdf5/1.8.14-intel/include -I/home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include -I/usr/include/python2.6 -c netCDF4.c -o build/temp.linux-x86_64-2.6/netCDF4.o
gcc: netCDF4.c: No such file or directory
gcc: no input files
error: command 'gcc' failed with exit status 1

In the original (master) branch, I tried to run the gcc compile line by hand after creating netCDF4.c from the command line:

$ cython netCDF4.pyx
$ gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/apps/netcdf/4.3.0-intel/include -I/apps/hdf5/1.8.14-intel/include -I/home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include -I/usr/include/python2.6 -c netCDF4.c -o build/temp.linux-x86_64-2.6/netCDF4.o
In file included from /home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804,
                 from /home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from netCDF4.c:250:
/home/Sourish.Basu/theia_packages/python/lib64/python2.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
netCDF4.c: In function ‘__pyx_f_7netCDF4__set_att’:
netCDF4.c:18405: warning: passing argument 5 of ‘nc_put_att_string’ from incompatible pointer type
/apps/netcdf/4.3.0-intel/include/netcdf.h:1011: note: expected ‘const char **’ but argument is of type ‘char **’
netCDF4.c: In function ‘__pyx_f_7netCDF4__find_cmptype’:
netCDF4.c:56563: warning: ‘__pyx_v_xtype’ may be used uninitialized in this function
$ ls -l build/temp.linux-x86_64-2.6/netCDF4.o
-rw-r--r-- 1 Sourish.Basu co2 5492848 2015 Apr 10 22:03:14 build/temp.linux-x86_64-2.6/netCDF4.o

So it seems that cython from the command prompt works, but from the python module does not. I also tried setting 'has_cython = False' explicitly after the 'try/except' block which evaluates the cython version, and that seemed to compile a functional netCDF4.so as well, using 'setup.py build'.

[jswhit]

I've never seen this before. Something is wrong with your cython module installation, but I can't imagine what it is. At least you have a workaround.

[jswhit]

Pull request #394 adds use_cython option in setup.cfg that can be used to suppress compilation of netCDF4.pyx (so you won't need to hack setup.py).

[AgilentGCMS]

Jeff,

FYI, I downgraded from Cython-0.22 to Cython-0.19, to see if this problem disappeared. It didn't. Then I downgraded from 1.1.8 of python-netcdf to 1.0.7, and the problem disappeared (with Cython-0.19). I haven't tried the combination of python-netcdf 1.0.7 and Cython-0.22, but given that with Cython-0.19, python-netcdf 1.0.7 didn't have this problem but 1.1.8 did (so did 1.1.7), I'm thinking it has something to do with the way setup.py uses Cython. I will keep digging.

-Sourish

[jswhit]

Sourish - any updates on this?

[AgilentGCMS]

Jeff - On theia I could compile netCDF4-1.1.7.1 using cython 0.22.1. Not sure what the issue was earlier; could have been a buggy cython, for all I know. Since this is not a problem any more for me, I'll close this ticket. Thanks!