Skip to content

Cannot build/install on Ubuntu 12.10 64Bit in Virtual Machine #54

Closed
stefanocrosta opened this Issue Dec 28, 2012 · 23 comments

3 participants

@stefanocrosta

Hello,

I'm not pretending that somebody will track down the cause but I thought I'd share the situation.

I'm trying to install bottleneck on a VirtualBox 64bits Ubuntu 12.10 guest.

python setup.py install

ends with the following lines:

bottleneck/src/func/64bit/func.c: At top level:
/home/scrosta/virtualenv/local/lib/python2.7/site-packages/numpy/core/include/numpy/__ufunc_api.h:226:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.7/README.Bugs> for instructions.
error: command 'gcc' failed with exit status 4

where of course the gcc: internal compiler error: Killed (program cc1) is the ugly bit.

Also when trying a make all from the repository gives exactly the same error:

/home/scrosta/virtualenv/local/lib/python2.7/site-packages/numpy/core/include/numpy/__ufunc_api.h:226:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.7/README.Bugs> for instructions.
error: command 'gcc' failed with exit status 4
make: *** [funcs] Error 1
@stefanocrosta

and just out of curiosity, had the exact same issue on a virtualized 12.04 ubuntu with gcc 4.6

@kwgoodman
Owner

A web search on command 'gcc' failed with exit status 4 gave me this:

pydata/pandas#1880

Don't know if it is helpful.

@stefanocrosta

Thanks @kwgoodman !

Sounds like this part of the solution is the only one that might help (considering that in my case stepping to my GCC 4.6 and older cython did not help)

Setting up a temporary swapfile before building pandas helps.

I managed to compile with 512 MB swap created with the following commands:

sudo dd if=/dev/zero of=/swapfile bs=1024 count=524288
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

..but even this didn't work. I tried with a 1.6Gb swap :-/
Fails quickly and as badly as before.

$ gcc -dumpversion; cython -V
4.7
Cython version 0.17.3

@stefanocrosta

I could not build it on an Ubuntu 12.04 with 32gigs or RAM either, though the error is different and I'm pondering about making this a very separate issue:

$ pip install bottleneck
Downloading/unpacking bottleneck
Downloading Bottleneck-0.6.0.tar.gz (3.2MB): 3.2MB downloaded
Running setup.py egg_info for package bottleneck

package init file 'bottleneck/tests/__init__.py' not found (or not a regular file)
package init file 'bottleneck/src/func/__init__.py' not found (or not a regular file)
package init file 'bottleneck/src/move/__init__.py' not found (or not a regular file)

Installing collected packages: bottleneck
Running setup.py install for bottleneck
package init file 'bottleneck/tests/init.py' not found (or not a regular file)
package init file 'bottleneck/src/func/init.py' not found (or not a regular file)
package init file 'bottleneck/src/move/init.py' not found (or not a regular file)
building 'func' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/home/hct/.virtualenv/local/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c > bottleneck/src/func/64bit/func.c -o build/temp.linux-x86_64-2.7/bottleneck/src/func/64bit/func.o
bottleneck/src/func/64bit/func.c: In function ‘Pyx_GetBuffer’:
bottleneck/src/func/64bit/func.c:350707:15: warning: unused variable ‘getbuffer_cobj’ [-Wunused-variable]
bottleneck/src/func/64bit/func.c: In function ‘__Pyx_ReleaseBuffer’:
bottleneck/src/func/64bit/func.c:350747:15: warning: unused variable ‘releasebuffer_cobj’ [-Wunused-variable]
bottleneck/src/func/64bit/func.c: At top level:
/home/hct/.virtualenv/local/lib/python2.7/site-packages/numpy/core/include/numpy/__ufunc_api.h:226:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
bottleneck/src/func/64bit/func.c: In function ‘initfunc’:
bottleneck/src/func/64bit/func.c:312800:16: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without
Assembler messages:
Fatal error: can't create build/temp.linux-x86_64-2.7/bottleneck/src/func/64bit/func.o: No such file or directory
error: command 'gcc' failed with exit status 1
Complete output from command /home/hct/.virtualenv/bin/python -c "import setuptools;__file
='/home/hct/.virtualenv/build/bottleneck/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-FlX0Th-record/install-record.txt --single-version-externally-managed --install-headers /home/hct/.virtualenv/include/site/python2.7:
running install

were the relevant part seem to be:

bottleneck/src/func/64bit/func.c:312800:16: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without
Assembler messages:
Fatal error: can't create build/temp.linux-x86_64-2.7/bottleneck/src/func/64bit/func.o: No such file or directory
error: command 'gcc' failed with exit status 1

@kwgoodman
Owner

I get the same note:

bottleneck/src/move/64bit/move.c:220450:16: note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without

But my build is successful. (I'm not using a VM though.)

@timmie
timmie commented Apr 13, 2013

I have the same problem using Ubuntu 12.10:

timmie@mybox:/tmp/bottleneck/bn$ source bin/activate
(bn)timmie@mybox:/tmp/bottleneck/bn$ ls
insgesamt 24K
drwxr-xr-x 2 timmie users 4,0K Apr 13 17:04 bin
drwxr-xr-x 8 timmie users 4,0K Apr 13 17:09 bottleneck
drwxr-xr-x 2 timmie users 4,0K Apr 13 17:04 include
drwxr-xr-x 3 timmie users 4,0K Apr 13 17:04 lib
drwxr-xr-x 2 timmie users 4,0K Apr 13 17:04 local
drwxr-xr-x 9 timmie users 4,0K Apr 13 17:04 lp_daily_devel-0.7.0~ppa1~revno-{revno}


(bn)timmie@mybox:/tmp/bottleneck/bn/bottleneck$ ls
insgesamt 64K
drwxr-xr-x 6 timmie users 4,0K Apr 13 17:07 bottleneck
drwxr-xr-x 4 timmie users 4,0K Apr 13 17:09 build
drwxr-xr-x 5 timmie users 4,0K Apr 13 17:07 doc
-rw-r--r-- 1 timmie users   71 Apr 13 17:07 LICENSE
-rw-r--r-- 1 timmie users 1,8K Apr 13 17:07 Makefile
-rw-r--r-- 1 timmie users  205 Apr 13 17:07 MANIFEST.in
-rw-r--r-- 1 timmie users  13K Apr 13 17:07 README.rst
-rw-r--r-- 1 timmie users 6,1K Apr 13 17:07 RELEASE.rst
drwxr-xr-x 2 timmie users 4,0K Apr 13 17:07 sandbox
-rw-r--r-- 1 timmie users 3,3K Apr 13 17:07 setup.py
drwxr-xr-x 2 timmie users 4,0K Apr 13 17:07 tools
-rw-r--r-- 1 timmie users  709 Apr 13 17:07 tox.ini


(bn)timmie@mybox:/tmp/bottleneck/bn/bottleneck$ python -c "import cython"


python ./setup.py build
running build
running build_py
package init file 'bottleneck/tests/__init__.py' not found (or not a regular file)
package init file 'bottleneck/src/func/__init__.py' not found (or not a regular file)
package init file 'bottleneck/src/move/__init__.py' not found (or not a regular file)
package init file 'bottleneck/tests/__init__.py' not found (or not a regular file)
package init file 'bottleneck/src/func/__init__.py' not found (or not a regular file)
package init file 'bottleneck/src/move/__init__.py' not found (or not a regular file)
running build_ext
building 'func' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c bottleneck/src/func/64bit/func.c -o build/temp.linux-x86_64-2.7/bottleneck/src/func/64bit/func.o
gcc: Fehler: bottleneck/src/func/64bit/func.c: Datei oder Verzeichnis nicht gefunden
gcc: Schwerwiegender Fehler: Keine Eingabedateien
Kompilierung beendet.
error: command 'gcc' failed with exit status 4
@kwgoodman
Owner

I guess it's a memory issue. To make sure, could you try compiling with the changes below? With these changes bottleneck only has two functions: bn.func.median() and bn.move_sum(). Make sure to modify func.py and not func.pyx (same goes for move.py).

bottleneck/src/template/func/func.py:

diff --git a/bottleneck/src/template/func/func.py b/bottleneck/src/template/func/func.py
index e13bbe7..307cb92 100644
--- a/bottleneck/src/template/func/func.py
+++ b/bottleneck/src/template/func/func.py
@@ -26,25 +26,6 @@ from .allnan import allnan

 funcs = {}
 funcs['median'] = median
-funcs['nanmedian'] = nanmedian
-funcs['nansum'] = nansum
-funcs['nanmean'] = nanmean
-funcs['nanvar'] = nanvar
-funcs['nanstd'] = nanstd
-funcs['nanmin'] = nanmin
-funcs['nanmax'] = nanmax
-funcs['nanargmin'] = nanargmin
-funcs['nanargmax'] = nanargmax
-funcs['rankdata'] = rankdata
-funcs['nanrankdata'] = nanrankdata
-funcs['ss'] = ss
-funcs['nn'] = nn
-funcs['partsport'] = partsort
-funcs['argpartsort'] = argpartsort
-funcs['replace'] = replace
-funcs['anynan'] = anynan
-funcs['allnan'] = allnan
-
 header = """#cython: embedsignature=True

 import numpy as np
@@ -97,25 +78,7 @@ cdef extern from "math.h":

 PARTSORT_ERR_MSG = "`n` (=%d) must be between 1 and %d, inclusive." 

-include "nanmax.pyx"
-include "nanmin.pyx"
-include "nansum.pyx"
-include "nanmean.pyx"
-include "nanstd.pyx"
-include "nanvar.pyx"
 include "median.pyx"
-include "nanmedian.pyx"
-include "nanargmin.pyx"
-include "nanargmax.pyx"
-include "rankdata.pyx"
-include "nanrankdata.pyx"
-include "ss.pyx"
-include "nn.pyx"
-include "partsort.pyx"
-include "argpartsort.pyx"
-include "replace.pyx"
-include "anynan.pyx"
-include "allnan.pyx"
 """

 def funcpyx(funcs=funcs, bits=None):

bottleneck/src/template/move/move.py

diff --git a/bottleneck/src/template/move/move.py b/bottleneck/src/template/move/move.py
index 88989f7..9c1ef5e 100644
--- a/bottleneck/src/template/move/move.py
+++ b/bottleneck/src/template/move/move.py
@@ -5,29 +5,9 @@ import bottleneck.src.template.template as tempmod
 import os.path

 from .move_sum import move_sum
-from .move_nansum import move_nansum
-from .move_mean import move_mean
-from .move_median import move_median
-from .move_nanmean import move_nanmean
-from .move_std import move_std
-from .move_nanstd import move_nanstd
-from .move_min import move_min
-from .move_max import move_max
-from .move_nanmin import move_nanmin
-from .move_nanmax import move_nanmax

 funcs = {}
 funcs['move_sum'] = move_sum
-funcs['move_nansum'] = move_nansum
-funcs['move_mean'] = move_mean
-funcs['move_median'] = move_median
-funcs['move_nanmean'] = move_nanmean
-funcs['move_std'] = move_std
-funcs['move_nanstd'] = move_nanstd
-funcs['move_min'] = move_min
-funcs['move_max'] = move_max
-funcs['move_nanmin'] = move_nanmin
-funcs['move_nanmax'] = move_nanmax

 header = """#cython: embedsignature=True

@@ -70,16 +50,6 @@ elif np.int_ == np.int64:
 MOVE_WINDOW_ERR_MSG = "Moving window (=%d) must between 1 and %d, inclusive"

 include "move_sum.pyx"
-include "move_nansum.pyx"
-include "move_mean.pyx"
-include "move_median.pyx"
-include "move_nanmean.pyx"
-include "move_std.pyx"
-include "move_nanstd.pyx"
-include "move_min.pyx"
-include "move_max.pyx"
-include "move_nanmin.pyx"
-include "move_nanmax.pyx"
 """

 def movepyx(funcs=funcs, bits=None):
@timmie
timmie commented Apr 14, 2013

Are you sure?
I have 8GB RAM...

Check also the pandas setup.py:
There's a extra cython command:
https://github.com/pydata/pandas/blob/master/setup.py#L320

@kwgoodman
Owner

No, I'm not sure. It was a guess.

Are you saying the problem is that you don't have cython installed? The line you are pointing to ends in this:

raise Exception("""Cython-generated file '%s' not found.
                           Cython is required to compile pandas from a development branch.
                           Please install Cython or download a release package of pandas.
                """ % src)
@timmie
timmie commented Apr 15, 2013

No, I'm not sure. It was a guess.
And I do not work with Cython at all.
I am just trying to package the module on Ubuntu
https://code.launchpad.net/~pythonxy/+recipe/bottleneck-daily

No, I meant:
https://github.com/pydata/pandas/blob/master/setup.py#L304
which points to
https://github.com/pydata/pandas/blob/master/setup.py#L358

@kwgoodman
Owner

Bottleneck's setup.py only supports building from C files, not from pyx files.

@kwgoodman
Owner

I added the C files to the repo. Does that fix your issue?

@timmie
timmie commented Apr 21, 2013

If you do not want to increase the repository size, these files could be put in an extra repo.
But I would complicate things.

Thanks!
People will now be apple to easily use your code on Ubuntu!

@kwgoodman
Owner

That's great news!

Hey I noticed this script contains a couple of mentions of patsy:

http://bazaar.launchpad.net/~pythonxy/pythonxy/bottleneck_debian/view/head:/control

@timmie
timmie commented Apr 21, 2013

Thanks for the pointer.
The last traces from the template were removed and all updated:
http://bazaar.launchpad.net/~pythonxy/pythonxy/bottleneck_debian/revision/2

This will be included in the next package builds

I guess this issue can be closed.
@stefanocrosta are you fine?
did you try the PythonXY repo?

@kwgoodman
Owner

OK, I'll go ahead and close.

@kwgoodman kwgoodman closed this Apr 24, 2013
@kwgoodman
Owner

I'm interested in getting bottleneck ready for packaging in debian/ubuntu. Let's start a new issue: issue #67.

@stefanocrosta

Hello @timmie ! sorry for getting back so late. Actually my issue must be (slighlty?) different, because i got it to build on my "real" server (12.04) but fails as described on my VM ubuntu (now on 13.04). I tried, but same behaviour/error using pythonXY version... Maybe a combined issue with memory also?

@kwgoodman
Owner

@stefanocrosta, I have rewritten bottleneck. I'm curious if it now builds on a VM, so if you get a chance to try, let me know.

@kwgoodman kwgoodman reopened this Feb 6, 2015
@kwgoodman kwgoodman closed this Feb 9, 2015
@stefanocrosta

Hello @kwgoodman yes now it does, no more errors!
(it's a different VM though... my older laptop retired...)

thanks!!

@kwgoodman
Owner

That's great! Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.