default open file descriptors limit regression from precise -> trusty images

[kwlzn]

after pex-tool/pex@0ada560 in the pantsbuild/pex repo (which switches from dist: precise to dist: trusty), we've been experiencing a routine issue with running out of file descriptors in our pypy test shard (with <200 tests).

here is an example of a working run prior to the change: https://travis-ci.org/pantsbuild/pex/jobs/286804414

and an example of a failing run after the change: https://travis-ci.org/pantsbuild/pex/jobs/383327951#L1091

bumping the ulimit in our before_script config seems to have no effect as seen here: https://travis-ci.org/pantsbuild/pex/jobs/386076734#L484

is it possible that the trusty image has regressed in the number of default file descriptors?

[kwlzn]

precise ulimit -n: https://travis-ci.org/kwlzn/pex/builds/388132188#L255

open files                      (-n) 64000

trusty ulimit -n: https://travis-ci.org/kwlzn/pex/builds/388132401#L504

open files                      (-n) 30000

• comparing /etc/security/limits.conf on:

trusty: https://travis-ci.org/kwlzn/pex/builds/388130491#L487

travis hard nofile 64000
travis soft nofile 72000

vs

precise: https://travis-ci.org/kwlzn/pex/builds/388131014#L235

travis   hard    nofile          64000
travis   soft    nofile          72000

[kwlzn]

attempts to ulimit -n 64000 in before_script fail here: https://travis-ci.org/kwlzn/pex/builds/388134050#L477

$ ulimit -n 64000
/home/travis/.travis/job_stages: line 57: ulimit: open files: cannot modify limit: Operation not permitted

it's not clear how to get the same open files limit that was the default on precise on the trusty image.

[joepvd]

Could you try again with sudo: required in your configuration?

[kwlzn]

sudo: required and dist: trusty result in the same low ulimit:

https://travis-ci.org/kwlzn/pex/jobs/388371667#L492

and attempts to ulimit -n 64000 also fail:

https://travis-ci.org/kwlzn/pex/builds/388398133#L503

[threesquared]

We are also having issues hitting this 30000 file limit on our builds.

[mkalkbrenner]

We have the same issue for drupal/search_api_solr and the solarium php library. Both run integration tests with a real Solr server using travis. We now get this Warning:

*** [WARN] *** Your open file limit is currently 30000.
 It should be set to 65000 to avoid operational disruption.

And we get random test failures!

[ThomasLabstep]

Same issue here with phpdbg.

[jandom]

With dist: trusty, in interactive debug, it's possible to lower the ulimit below its default value (30000) but once that's done it's impossible to increase the limit again (some ratchet-like config)

[DrTorte]

Hello,

We're moving our community communications to the Travis CI Community Forum and will no longer be supporting GitHub issues. If this hasn't been solved, please go ahead and create a new thread on the Forum and we'll be better able to help you there. We're looking forward to seeing you there.

If you need support, you can reach our support team here: support@travis-ci.com.

Thanks and happy building!