Add more aix machines

[BridgeAR]

Is it possible to add a new aix server for the CI? Currently we have a queue of CIs to finish that all only wait on aix.

[Trott]

Co-sign.

[sam-github]

Mine always seem to be blocked on Arm and Windows. Did something change recently? Not that getting more AIX machines wouldn't be great if that's the new bottleneck. Anything that speeds up the CI is fantastic, its pretty painful ATM.

[Trott]

Mine always seem to be blocked on Arm and Windows. Did something change recently? Not that getting more AIX machines wouldn't be great if that's the new bottleneck. Anything that speeds up the CI is fantastic, its pretty painful ATM.

I think AIX becomes a bottleneck when someone runs CITGM because that ties up one of (I think) only two hosts for hours. Tying up one Windows host for hours for a CITGM run is no big deal because we have so many Windows hosts. And we don't run CITGM on Raspberry Pi at all, so that's not an issue either.

So yeah, under ordinary conditions, Windows and Raspberry Pi tend to be the bottlenecks, but not too bad. But, I think, once a CITGM job or two get kicked off, AIX ends up taking literally hours.

[gdams]

@BridgeAR I am currently in the process of donating a much more powerful AIX machine to the community which will allow us to have many more machines and help with the CI backlog issues.

[Trott]

@gdams Is this worth putting on the Build WG agenda just so that in case this stalls out, it's there if obstacles or problems need to be discussed next week?

[Trott]

Mine always seem to be blocked on Arm and Windows. Did something change recently? Not that getting more AIX machines wouldn't be great if that's the new bottleneck. Anything that speeds up the CI is fantastic, its pretty painful ATM.

I think AIX becomes a bottleneck when someone runs CITGM because that ties up one of (I think) only two hosts for hours

I don't know what's changed in the last month or whatever, but AIX is definitely the big bottleneck, much more so than Windows and Raspberry Pi. In the past, IIRC, this would only be during CITGM runs. But it's a constant now. Part of it might be that AIX is so susceptible to whatever is causing the rash of failing tests lately that it needs to be re-run a lot, causing it to fall further behind in the queue than other platforms that don't need to be re-run during a Resume Build.

Nothing unusual is going on in terms of building right now--it's a typical quiet-ish Friday--but the CI queue is totally backed up and it's entirely due to waiting for AIX hosts to be available for work.

[Trott]

I'll also add that the tests used to run really fast on the AIX hosts. The build/compile took a long time, but once the tests were going, it was impressive. Not so much in CI anymore. Now everything is slow on AIX. I don't know if we swapped in hosts with less memory/CPU or something, but it sure seems like something significant changed.

[Trott]

All that complaining...er, I mean providing information....above aside, I do believe one or two additional hosts would resolve the issue entirely.

[mhdawson]

@gdams in parallel with getting the new machine to OSU can you also talk to David to see if anything with respect to the configuration changed? We might also want to double check that the ramdisk is still in place and working.

[gireeshpunathil]

While I have access to one of the CI, I see that it has 16 CPUs, so wondering why we are running gmake in single thread?

[rvagg]

@gireeshpunathil where are you seeing it run as a single?

[gireeshpunathil]

say for example if you take the current run -
https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/19870/consoleFull
and grep for 01:30:13 gmake -C out BUILDTYPE=Release V=1 I don't see a -j flag to it?

[rvagg]

it's supplied to the parent make call and it's supposed to coordinate across all child invocations as well to make sure the total isn't more than N processes for -j N

I was watching a job live and noted that git clean -fdx took an unexpectedly long time, could there be disk problems with these machines?

[gireeshpunathil]

ok, without knowing the full details on the CI script, I can state that gmake -C out BUILDTYPE=Release V=1 is slower than gmake -C out BUILDTYPE=Release V=1 -jN, but I see your point, if there are other tasks that need to run parallel to gmake the N goes for that (although I don't know what those tasks are).

slowness on git clean -fdx was noticed earlier, but then it was on a CIGTM run which installed a large number of modules and was taking lot of time.

I did some studies on CI runtimes of different platforms and my inference is that AIX runs are SLOW (average 50 minutes) , and the slowness is distributed across the run time, no throttling at any point.

system state does not show anything significant:

Topas Monitor for host:    power8-nodejs2       EVENTS/QUEUES    FILE/TTY                                
Wed Dec 19 01:40:08 2018   Interval:  2         Cswitch     906  Readch   277.5K                         
                                                Syscall   21286  Writech   19627                         
CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   Reads       218  Rawin         0                         
ALL   13.9   44.1    0.0   42.0   1.32  132.4   Writes       58  Ttyout     1067                         
                                                Forks         3  Igets         4                         
Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out  Execs         4  Namei      1360                         
Total    11.6     33.7    18.1     4.1     7.5  Runqueue    4.1  Dirblk        0                         
                                                Waitqueue   0.0                                          
Disk    Busy%     KBPS     TPS KB-Read KB-Writ                   MEMORY                                  
Total     1.3     26.1     6.0     0.0    26.1  PAGING           Real,MB   32768                         
                                                Faults    22513  % Comp     38                           
FileSystem        KBPS     TPS KB-Read KB-Writ  Steals        0  % Noncomp  38                           
Total            264.6   139.0  264.3    0.2    PgspIn        0  % Client   38                           
                                                PgspOut       0                                          
Name            PID  CPU%  PgSp Owner           PageIn        0  PAGING SPACE                            
python2.   45809776   2.3   0.3 iojs            PageOut       0  Size,MB     512                         
python2.   40042666   0.6   8.4 iojs            Sios          0  % Used      7                           
python2.   34996348   0.3   0.3 iojs                             % Free     93                           
sshd       39977210   0.1   0.5 root            NFS (calls/sec)                                          
java       38994050   0.1  51.6 iojs            SerV2         0  WPAR Activ     0                        
topas      12320868   0.0   2.1 iojs            CliV2         0  WPAR Total     0                        
sched        196614   0.0   0.4 root            SerV3         0  Press: "h"-help                         
syncd        720986   0.0   0.6 root            CliV3         0         "q"-quit                         
j2pg        2818138   0.0   8.1 root            SerV4         0                                          
getty       6029500   0.0   0.7 root            CliV4         0                       

iojs 30343316 58785818 0 00:55:52 - 0:00 gmake run-ci -j 5 JOBS=5

translating to:
iojs 40042666 21823510 2 01:16:39 - 0:09 /usr/bin/python tools/test.py -j 5 -p tap --logfil

looks good to me, but the same -j5 , if percolated to gmake as well, I think it will reduce the overall build time?