Bazel overloads machine: new processes get stuck in call_rwsem_down_read_failed

[yaroslavvb]

Trying to test latest TF build, bazel makes machine unusable. New processes like ps aux get stuck with following under /proc/<pid>/stack which suggests that the system ran out of semaphores

[<ffffffff813e2594>] call_rwsem_down_read_failed+0x14/0x30
[<ffffffff8126a679>] proc_pid_cmdline_read+0x99/0x4f0
[<ffffffff811fd438>] __vfs_read+0x18/0x40
[<ffffffff811fd9ef>] vfs_read+0x7f/0x130
[<ffffffff811fe7c6>] SyS_read+0x46/0xa0
[<ffffffff817f74f6>] entry_SYSCALL_64_fastpath+0x16/0x75
[<ffffffffffffffff>] 0xffffffffffffffff

Linux version: 4.4.0-36-generic
Bazel version:

Build label: 0.4.3                                                                                       │················································································
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer│················································································
_deploy.jar                                                                                              │················································································
Build time: Thu Dec 22 12:31:25 2016 (1482409885)                                                        │················································································
Build timestamp: 1482409885                                                                              │················································································
Build timestamp as int: 1482409885                                                                       │················································································

To reproduce:

32-core Xeon(R) CPU E5-2630 v3 @ 2.40GHz

git clone https://github.com/tensorflow/tensorflow.git
./configure
XLA support? Y
CUDA support? Y
(everything else default)
bazel test --config=cuda --config=opt -k //tensorflow/...

[19,592 / 21,142] Still waiting for 200 jobs to complete:                                                │················································································
      Running (standalone):                                                                              │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 21 of 30), 3433 s                │················································································
[19,782 / 21,142] Still waiting for 10 jobs to complete:                                                 │················································································
      Running (standalone):                                                                              │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 21 of 30), 3496 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 12 of 30), 3479 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 11 of 30), 3476 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 20 of 30), 3471 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 13 of 30), 3468 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 10 of 30), 3468 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 17 of 30), 3467 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 16 of 30), 3467 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 14 of 30), 3465 s                │················································································
        Testing //tensorflow/compiler/xla/tests:reshape_test_gpu (shard 15 of 30), 3464 s                │················································································
^C                                                                                                       │················································································

[yaroslavvb]

It looks like Bazel just exacerbated an existing issue, and I'm able to build fine on fresh machine