High memory usage during compilation

[nicktimko]

I've been having some trouble (pip) installing libsass in a container, seemingly because the compiler runs out of memory (my Docker host VM has about 300 MB free) and gets killed.

$ docker run -i -t python:3.4 pip install libsass
Collecting libsass
  Downloading libsass-0.8.2.tar.gz (229kB)
    100% |████████████████████████████████| 229kB 2.0MB/s
...
    gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./libsass -I/usr/local/include/python3.4m -c libsass/extend.cpp -o build/temp.linux-x86_64-3.4/libsass/extend.o -c -O3 -fPIC -std=c++0x -Wall -Wno-parentheses
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    gcc: internal compiler error: Killed (program cc1plus)
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See <file:///usr/share/doc/gcc-4.9/README.Bugs> for instructions.
    error: command 'gcc' failed with exit status 4

    ----------------------------------------
Command "/usr/local/bin/python3.4 -c "import setuptools, tokenize;__file__='/tmp/pip-build-4ymrmerb/libsass/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record
/tmp/pip-bdkglgjs-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-4ymrmerb/libsass

By way of comparison (not sure if it's a valid comparison or not), Numpy compiles just fine, using less than half the memory when trying to compile libsass. Can any compilation arguments be added to allow building on more meager VMs?

[sephii]

I'm having the same issue even with 450MB free. I'm running my site on a 512MB VM and can't get more RAM. Can we do something about this?

[asottile]

So I looked into this a little bit with #138 and the files that are having difficult compiling (due to memory) are from the upstream (sass/libsass)

I tried twiddling the compile flags a little bit (and profiling with /usr/bin/time --format=%M gcc ... and was only able to shave off ~20MB of max RSS (out of ~250MB) by removing -g and using -O0 -- the first I can't really change since setuptools is getting that from the interpreter, the latter I can but don't really want to go lower than -O2 (and changing -O3 to -O2 only saves ~1MB, and -O2 to -O0 only saves ~2MB).

One thing that might help (if possible) would be to add swap -- I was able to get libsass to compile (very very slowly) when I limited memory to ~50MB (using docker).

The solution I suggested in #138 also seems to work -- prebuild a wheel on a compatible platform and use that in production.

[asottile]

Perhaps may be interested in #151 (build manylinux wheels)

[nicbou]

This bug got closed, but I'm running into that exact issue. Am I missing a detail?

[nicktimko]

Is your pip selecting the pre-built wheel to install (does it download a
.whl instead of a .tar.gz)?

On Sun, Sep 11, 2016, 05:09 Nicolas Bouliane notifications@github.com
wrote:

This bug got closed, but I'm running into that exact issue. Am I missing a
detail?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#75 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFDVlHNVom22595R1oTuOzpAPFP4qGHDks5qo9NWgaJpZM4E64ZB
.

[asottile]

Note that you'll need a sufficiently new version of pip to take advantage of manylinux wheels. (>=8)