Broken pipe errors from TravisCI

[iilyak]

Expected Behavior

When we do a push of previously tested code we expect the CI to pass.

Current Behavior

TravisCI fails on the test suite stage. Logs reveal errors like the following:

• Error writing to pipe: Broken pipe
• erl_child_setup: failed with error 32 on line 253 - error 32 is "Broken pipe"

Possible Solution

Broken pipe is usually resource exhaustion however it seems to be a known problem for container based environments.

• increase ulimit for travis instances (if possible)
• one tread on internet recommends switching to full virtualization to fix the issue

Steps to Reproduce (for bugs)

1. create branch of master git checkout master && git checkout -b test-ci
2. push it to github.com/apache/couchdb
3. try few times watching for failed TravisCI builds

Context

1. The issue prevents us from merging PRs
2. The presence of the issue decrease our confidence in the CI
3. We have to spent time trying multiple times
4. It adds additional burden for the reviewer to check logs to make sure the failure is an exhaustion issue.

[iilyak]

Additional debugging info for the case.

diff --git a/.travis.yml b/.travis.yml
index 01b2862a3..89e21baf2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -49,6 +49,7 @@ env:

 # Then comment this section out
 before_script:
+  - ulimit -a
   - rm -rf /tmp/couchjslogs
   - mkdir -p /tmp/couchjslogs
   - ./configure -c --disable-docs --disable-fauxton
diff --git a/Makefile b/Makefile
index 0ab7b4eed..09757fe2c 100644
--- a/Makefile
+++ b/Makefile
@@ -149,6 +149,7 @@ check: all
 eunit: export BUILDDIR = $(shell pwd)
 eunit: export ERL_AFLAGS = -config $(shell pwd)/rel/files/eunit.config
 eunit: couch
+       @ulimit -a && exit 1
        @$(REBAR) setup_eunit 2> /dev/null
        @$(REBAR) -r eunit $(EUNIT_OPTS)

https://travis-ci.org/apache/couchdb/jobs/451351684

before_script

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 241238
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 30000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

eunit

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        unlimited
coredump(blocks)     unlimited
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              unlimited
nofiles              30000
vmemory(kbytes)      unlimited
locks                unlimited

[iilyak]

I don't have any more ideas at the moment

[wohali]

@iilyak See JuliaLang/julia#9501 . Perhaps we are doing the same thing?

the original bug was in the test (redirecting stdout, but not reading from it to prevent kernel from blocking on more data)

[wohali]

Also, at one point we were using the alternate form of virtualisation, but our tests took a lot longer to run.

Travis apparently has more backpressure on the full-VM instances because they require more resources to run than the containers we use now. With such interest in running CouchDB in Docker, I think it's probably best if we look deeper and see if we can't find an actual root cause.

[wohali]

@iilyak we are no longer using the containerised virtualisation in Travis (merged with the Python 3 PR).

I think this will improve things drastically. Going to close this out for now, unless you see it again.