Error 137 on gradle downloading after upgrading Android NDK to r10c

[koral--]

Building Android project which requires NDK have started being tricky with NDK r10c.

1. even unpacking NDK archive fails due to log length limit, as a workaround I've filtered its ouptut
2. after successful unpacking I've encountered random 137 error when downloading gradle

I've found #1409 with suggestion to stop some unneeded system services, but I'm not sure if has helped eg. in build 96 mysql service failed to stop. Finally I've got successful build but I had to restart it 3 times because it was failing on gradle downloading step.

How to get rid of such 137 errors?

[ardock]

I had the same problem and stop services didn't help me and I removed it.

I solved my random 137 errors doing changes until VM happy, this remember me something.

Another guy said that solved it adding echo's.
mapbox/mapbox-gl-native@b5021ea

You can try it too. echo "hey VM I'm a friend, don't kill me!" :)

But I suggest:

#branches:
#  except:
#    - dev
language:
  - android
jdk:
  - oraclejdk8
android:
  components:
    - build-tools-21.0.2
    - android-21

before_script:
  - sudo service postgresql stop || true
  - sudo service mysql stop || true
  - sudo service memcached stop || true
  - sudo service bootlogd stop || true
  - sudo service elasticsearch stop || true
  - sudo service mongodb stop || true
  - sudo service neo4j stop || true
  - sudo service cassandra stop || true
  - sudo service riak stop || true
  - sudo service rsync stop || true
  - sudo service x11-common stop || true

before_install:
  - export ANDROID_NDK_HOME=`pwd`/android-ndk-r10c
  - export PATH=${ANDROID_NDK_HOME}:${PATH}

install:
  - 'echo "Download Gradle Wrapper"; chmod +x ./gradlew; ls -l gradlew; ./gradlew -v'
  - wget http://dl.google.com/android/ndk/android-ndk-r10c-linux-x86_64.bin
  - chmod +x android-ndk-r10c-linux-x86_64.bin
  - ./android-ndk-r10c-linux-x86_64.bin | egrep -v ^Extracting

script:
  - ./gradlew build --info

I think I solved moving installations to Install key (and I'm running emulator on background... so I would remove stop services, probably already stopped...), I don't know but perhaps each stage has a different requirements. Pray for it and try it!. Good luck :)

[BanzaiMan]

For now, it seems to me that NDK is just too big to accommodate reliably.

[cernekee]

For now, it seems to me that NDK is just too big to accommodate reliably.

Could you please elaborate on this statement? Understanding the exact limitation we're hitting is important if we're going to develop a workaround.

Here are the possibilities I've looked at:

• Process limits. This was my first guess because exit code 137 could indicate SIGKILL. But if I run ulimit -a inside the VM I don't see any relevant CPU/memory limits.
• Out of disk space. But df -h shows plenty of free space. And that would normally just dump "out of space" errors to the console, not kill the process outright.
• OOM. Unlikely because 7z isn't a huge memory hog. When I run it locally it takes <50MB. Also, I run valgrind as part of my test suite on another project, and valgrind is worse.
• 438MB download is too big. I don't see any evidence that this is being cut short (although maybe it's running up somebody's bill if it isn't cached). My tarball fetcher script checks the SHA1 hash on every download.
• Using 7z x -x@filelist.txt to skip extracting a bunch of stuff I don't need, like gcc 4.6, clang, 64-bit toolchain binaries, L headers, etc. I didn't see a 137 error during this run, but I did see some other weird activity (lost communication with VM).
• Some travis-specific monitoring process. We're not hitting any of the documented time limits, but maybe it is unhappy because there is no console output during the extraction process? In this commit I added a script that prints a '.' character for each 100 files extracted. This worked the first time I tried it, but that doesn't mean anything because the error is intermittent. On the second run a completely different part of the build caught SIGHUP for no apparent reason.

Another option might involve hosting a cut-down version of NDK somewhere, instead of downloading the full package from Google. They used to publish separate packages for legacy vs. modern, but that doesn't seem to be the case anymore.

[BanzaiMan]

NDK's disk requirement makes it impractical to have it pre-installed (for now—we might revisit it at some point in the future). You can try installing it on the fly, but you have seen problems (this, too, might become easier if we can offer bigger VMs).

[cernekee]

You can try installing it on the fly

This always worked for me until NDK r10 or so, when they changed over to a gigantic self extracting 7z file with the 64-bit and L support files.

Any ideas where the mysterious Exit 137 / SIGHUP problems might be coming from?

[BanzaiMan]

My guess is memory shortage.

[cernekee]

/usr/bin/time -v reports a peak RSS of ~180MB (running locally). Too high?

    Command being timed: "./android-ndk-r10d-linux-x86_64.bin"
    User time (seconds): 35.35
    System time (seconds): 4.97
    Percent of CPU this job got: 95%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:42.23
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 179360
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 18579
    Voluntary context switches: 841
    Involuntary context switches: 8369
    Swaps: 0
    File system inputs: 200
    File system outputs: 7011240
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0

Is there anything in Travis that would go haywire after unpacking 3.5GB of data to the filesystem? When I ran df in the VM, it showed more than enough free space.

[koral--]

Error 137 occurs not only when dealing with NDK. In some of the builds in project linked in first post it occurred also during downloading gradle distribution (about 40 MB with progress indicator printed to console). That step is located after unpacking NDK. So it looks like only the last possibility listed by @cernekee matches.
To determine what exactly is causing 137 it should be possible to look at the travis backend source code and/or run travis on localhost.

[ardock]

I agree, error 137 can occurs when you execute sudo apt-get update too...

Is df -h output correct? I would like to know the real available space because it's not possible install new system images without delete other...

Is it possible that a specific folder is limited? /tmp? In the case of SDK, the disk space issue appears when system-images are cloned into this folder...

I don't know to check it or install travis locally.

[cernekee]

Is it possible that a specific folder is limited? /tmp?

FWIW, I've seen Error 137 unpacking NDK into /opt. /opt and /tmp were both on the same filesystem in my test run.

$ df -h
Filesystem             Size  Used Avail Use% Mounted on
/vz/private/197357724  428G  144G  264G  36% /
none                   1.5G  8.0K  1.5G   1% /dev
none                   308M   64K  308M   1% /run
none                   5.0M     0  5.0M   0% /run/lock
none                   1.5G     0  1.5G   0% /run/shm
/dev/null              768M  133M  636M  18% /var/ramfs

[ardock]

They accepted a pull request deleting sys-img-armeabi-v7a-android-wear-20, we gain 790Mb unzipped.

I use a work around to "solve" another issue deleting one sys-image more
or changing the dir used for emulator (replacing /tmp by ${TRAVIS_BUILD_DIR}).
Unresolved disk space issue on docker

And I "solve" the random error code 137 using other workarounds I already commented (and I think is less frequent using container-based infrastructure).

Now I read these links and your test and I think that perhaps the real issue is another:
http://askubuntu.com/questions/296038/what-is-the-difference-between-tmpfs-and-ramfs
Out of space running...
no-space-left-on-device