test-osuosl-aix61-ppc64_be build failures

[Trott]

test-osuosl-aix61-ppc64_be-2 seems to be reliably failing to build for the last 24 hours or so.

Example failure: https://ci.nodejs.org/job/node-test-commit-aix/936/nodes=aix61-ppc64/console

@jbergstroem said in IRC:

a lot of hanging test failures:
this: /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-child-process-fork-dgram.js child
..and this: /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/sequential/test-child-process-pass-fd.js child
seems to be a few 'wait' processes. I wonder if that syntax behaves differently on aix
been active for four days :( i'm on the phone for a couple of hours still -- could you perhaps file an issue and ping me/mhdawson?)
(we should ping gibfahn too -- he had a few tests that didn't exit either)

@mhdawson @gibfahn

[gibfahn]

I'll take a look, also cc/ @gireeshpunathil

[jbergstroem]

Just a note: I killed all lingering processes on both machines since the test runner was stalling.

[mhdawson]

I don't see any failures today since @jbergstroem comment and I also had to restart the jenkins agent. Since then its seems to be be running ok. We should probably check tomorrow to see if there are any jobs stacking up in the background.

[mhdawson]

Looked today. There were a small number of processes hanging around:

 iojs  8388860        1   0 18:59:37      -  0:00 /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /h
ome/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-child-process-fork-dgram.js child
    iojs 10289232        1   0   Sep 20      -  0:00 /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /h
ome/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-child-process-fork-dgram.js child
    iojs 10748052        1   0 16:24:13      -  0:00 /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /h
ome/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-child-process-fork-dgram.js child
    iojs 11141252        1   0 04:16:07      -  0:00 /home/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/out/Release/node /h
ome/iojs/build/workspace/node-test-commit-aix/nodes/aix61-ppc64/test/parallel/test-child-process-fork-dgram.js child


Which are related to this issue:

https://github.com/nodejs/node/issues/8271

[gibfahn]

@mhdawson That's interesting, I hadn't realised that the child processes weren't being killed in that test failure. So if the test times out, I assume the test runner kills the parent but not any children?

[jbergstroem]

@mhdawson if you check my above paste (well, Rich's) test/sequential/test-child-process-pass-fd.js seems affected too.

[mhdawson]

I've been looking at the fork-dgram failure, will just about to test proposed fix no AIX now. The other one must be much less frequent.

[mhdawson]

PR to address fork-dgram failures nodejs/node#8697

[jbergstroem]

@mhdawson, @gibfahn: log into -1 now and have a look. lots of stuff stalling. Here's a job that's been going for two hours as well (probably as a result of multiple test runners still being active): https://ci.nodejs.org/job/node-test-commit-aix/1040/nodes=aix61-ppc64/console

[gibfahn]

@jbergstroem I'm not sure whether it's because someone's cleaned up the machine (or because ps -ef isn't the right command), but I'm only seeing a couple of test-dgram processes on that machine.

Also when I first clicked your link I saw the still-running job (and I still have it open in a tab):

But now when I click through I don't:

[jbergstroem]

@gibfahn you also have a few python processes:

 ps -ef | grep python
    iojs 4784192 2818506   0   Sep 21      -  0:04 /usr/bin/python tools/test.py -p tap --logfile test.tap --mode=release --flaky-tests=dontcare addons doctool inspector known_issues message parallel pseudo-tty sequential
    iojs 2490818 1901104   0 17:54:10      -  0:04 /usr/bin/python tools/test.py -p tap --logfile test.tap --mode=release --flaky-tests=dontcare addons doctool inspector known_issues message parallel pseudo-tty sequential
    iojs 3408274 3211782   0   Sep 20      -  0:04 /usr/bin/python tools/test.py -p tap --logfile test.tap --mode=release --flaky-tests=dontcare addons doctool inspector known_issues message parallel pseudo-tty sequential

..and a few gmake processes:

 ps -ef | grep make
    iojs 4128770       1   0 17:43:33      -  0:00 gmake run-ci -j 5
    iojs 2621908       1   0   Sep 20      -  0:00 gmake run-ci -j 5
    iojs 2818506 2818606   0   Sep 21      -  0:00 gmake test-ci
    iojs 1901104 4128770   0 17:53:15      -  0:00 gmake test-ci
    iojs 2818606       1   0   Sep 21      -  0:00 gmake run-ci -j 5
    iojs 3211782 2621908   0   Sep 20      -  0:00 gmake test-ci

This is probably not good seeing how each test.py makes the assumption that they will be run exclusively (tmpdir, whatnot).

[jbergstroem]

You can also see the wait invoked as well as as few defunct processes.

[gibfahn]

@jbergstroem I'm not seeing those on test-osuosl-aix61-ppc64_be-1 (a.k.a. power8-nodejs2.osuosl.org). Are you looking at -2?

EDIT: I do see them on -2

[jbergstroem]

@gibfahn if I am then there's a naming thing. I'm talking about 140.211.9.101.

[gibfahn]

I might have them the wrong way round in my .ssh/config, I have:

Anyway, I'll take a look at why the processes are being left behind.

[mhdawson]

Ok, the test that was causing processes to still be running after the test run was fixed so I think we can close this. Please re-open if you feel differently.