packet.net ARM8 resources (FYI / headsup)

[rvagg]

https://www.packet.net/ are an IaaS service that offer some pretty serious resources, including lots of bare-metal stuff, including server-class ARM8 machines. They are also a strategic investment by SoftBank, the owner of ARM Holdings, so they put us in touch and packet.net have been enthusiastic to support what we are doing and also to give their hardware a good workout.

Copying this next bit from nodejs/node#13603 so as not to repeat myself too much.

@nodejs/testing for context: the mininodes hosts are still active, they have just been relabelled to ubuntu1604-arm64_odroid_c2. The new packet.net hosts are proper server-class ARM machines, not these repurposed mobile chips that we've had access to until now.

The major difference you'll find on these new packet.net machines: they have 96 cores and 48G of RAM. We have not virtualized or containerized anything so they are running on bare metal. At the moment we are running at ~ JOBS=50 (doing some experimentation on that front). We also have CentOS 7 machines running at packet.net, our first non-Debian ARM machines.

They have access to very fast SSD so the bottlenecks appear pretty late in the parallelization. My assumption with this error when I first saw it was that they are too heavily parallelized so I was reducing JOBS, but it's only been appearing on the Ubuntu 16.04 machines so I guess this is some kind of system problem.

The hosts are all accessible to everyone that has nodejs_build_test ssh access, they configs are in the new ansible setup (ansible directory of the build repo, look in the inventory.yml if you want IPs but there is a way to dump everything to your .ssh/config if you want, I just can't tell you off the top of my head). There are 2 x Ubuntu 16.04 and 2 x CentOS 7.

The other important piece of information here, particularly for @nodejs/release, is that we've switched the ARM8/ARM64 release build machine from one of the original ARM8 machines ARM Holdings provided us (in my garage) to a new dedicated packet.net release machine. This change means we are moving from Ubuntu 14.04 to CentOS 7 for release builds. In our (mainly my) assessment this is not going to have a noticeable impact on users of these binaries as the libc version is actually older on CentOS 7 than it is on 14.04. We move from a libc of 2.19 to 2.18 with this, and what's more, it doesn't seem to change what the binaries are compiled against. Here's a comparison of 8.1.0 binaries compiled on each:

$ objdump -T ubuntu1404/node-v8.1.0-linux-arm64/bin/node | awk '{ print $5}' | grep GLIBC | sort | uniq
GLIBC_2.17
GLIBCXX_3.4
GLIBCXX_3.4.11
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.18
GLIBCXX_3.4.9
$ objdump -T centos7/node-v8.1.0-linux-arm64/bin/node | awk '{ print $5}' | grep GLIBC | sort | uniq
GLIBC_2.17
GLIBCXX_3.4
GLIBCXX_3.4.11
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.18
GLIBCXX_3.4.9

So, I'm pretty confident that we won't see any problems with these, but a heads-up if you see any issues about them not working on certain distros where they may have in the past, this could be one place to look.

[rvagg]

In addition to this FYI: I'm keen to retire the ARM8 resources in my garage, they were only early prototypes of the final AppliedMicro X-Gene machines so they are not optimally set up, they draw more power than they could be, the fans are noisy and they take up a bunch of space on my "rack". The new test & release resources essentially make them redundant already, if all goes smoothly from here, but they do add Ubuntu 14.04 coverage to our ARM8 resources. IMO this is not critical at all, with CentOS 7 we do even older now. The ARM8 machines in my garage do serve some auxiliary purposes at the moment, including acting as a jump-host for SSH access to the cluster and serving the NFS shares that the Raspberry PIs currently use. I'm in the process of repurposing an Intel NUC that I have laying around that's not doing anything useful to serve these jobs so it means I might be able to turn them off at some point in the near future.

packet.net have FreeBSD and Debian on their roadmap for ARM8 so we should get even more coverage soon. They have Core OS, so I guess we could even be testing on some fancy Docker setups, I'm not sure what that would be that would add value though.

Additionally, Scaleway, an ARMv7 provider that sponsors a couple of test machines and a release build machine, have branched out in to ARM8 and reached out to me recently, so we'll probably be expanding some redundancy and additional distro coverage with them too!