Use gcc 4.9.4 in CI

[gibfahn]

Now that nodejs/node#13466 has landed, we need to make sure all our boxes have the newer compiler levels.

According to @bnoordhuis in nodejs/node#13466 (comment), binaries built with the later version should run everywhere they did before, so there shouldn't be a problem just updating the level on all our boxes.

I'd assume most/all of our older machines are using gcc 4.8.4.

This might be a good time to test out our ansible scripts.

[seishun]

I would like to work on this, but I couldn't find documentation on how this whole thing works and how I can test it locally. I would be grateful if someone can point me in the right direction.

/cc @nodejs/build

[gibfahn]

@seishun old ansible scripts are in setup/, new ones are in ansible/, we're trying to move over to the new scripts, but they're not completely tested yet.

The README in the ansible directory should have the info you need to get started, and the scripts should be documented. If you run into any issues with it let us know.

[seishun]

@gibfahn I'm a complete noob here, would https://github.com/nodejs/build/blob/master/setup/TESTING_LOCALLY.md still work for testing new scripts?

[refack]

@seishun I tried to follow the guides once but got distracted... You doing it for the first time is a good chance to document missing pieces and hidden assumption, so it would be great if you open a doc PR or pass the info to me and I'll PR it.

[gibfahn]

@seishun in theory yes. Actually that document should really be ported over to ansible/.

[seishun]

There are 5 remaining platforms in CI with outdated gcc:

• centos7-64. This should be simple, just need to figure out if the ansible/ scripts work well for CentOS 7 or the old scripts in setup/ should be changed instead.
• rhel72-s390x. It's not listed in ansible/roles/baselayout/vars/main.yml, so presumably it was deployed using setup/rhel72-linuxonecc. I'm also not sure how one can test the changes. @refack mentioned that you can get trial access here - will that work?
• aix61-ppc64. Same as above - is setup/aix61 the right script? Is there any way I can test this?
• cc-armv6 and cc-armv7. These two run on a machine called node-msft-cross-compiler-1, which is absent in both ansible/inventory.yml and setup/ansible-inventory. Which OS does it run? Are there Ansible scripts for it?

[mhdawson]

For rhel72-s390x, @jBarz is working on this. The problem is that new ansible templates were created which did not included full support for all platforms. The old templates were, still are used to deploy to the existing machines. Please don't update without co-ordinating with @jBarz
For aix61-ppc64 - I think @gibfahn is looking into that, please don't update without co-ordinating with him.

[seishun]

For rhel72-s390x, @jBarz is working on this.
For aix61-ppc64 - I think @gibfahn is looking into that

Are there issues for tracking these?

[mhdawson]

AIX is coverd in #925

rhel72-s390x, possibly not, I more recently asked jBarz to start working on that. @jBarz can you open an issue for tracking the work on linuxOne.

[jBarz]

Done for linuxOne #1023

[joaocgreis]

@seishun about the cross compiler, there is a PR that never landed: #244

[seishun]

@joaocgreis Looks like it uses arm-linux-gnueabihf-gcc and arm-linux-gnueabihf-g++ from https://github.com/raspberrypi/tools. These files haven't been updated in 4 years. Any idea what could be used instead to have gcc 4.9.4 or newer?

[seishun]

Or is someone already working on upgrading gcc on that machine?

[mhdawson]

@rvagg I seem to remember you already mentioning doing some work for 4.9.4 on the raspberry PIs

[rvagg]

No, I was working on 4.9 for the armv7 wheezy release machines but couldn't come up with a solution for it that didn't rely on a newer libc that screwed up release builds, we suffer from the same problem with the release pi's too. This is all on Wheezy which we are using on release armv6 and armv7. We're using the same compiler for both, which is bad because we get armv6 binaries on armv7 but I don't have a solution yet.
If this is a must then I think we're going to have to consider building release builds with on Jessie, or Ubuntu 14.04 and dropping official support for Wheezy-era distros (Ubuntu 12.04 included) for the Node versions that require this level of compiler.

[seishun]

If this is a must then I think we're going to have to consider building release builds with on Jessie, or Ubuntu 14.04 and dropping official support for Wheezy-era distros (Ubuntu 12.04 included) for the Node versions that require this level of compiler.

Support for Wheezy ends on 31st May 2018, which is shortly after Node 10.x release, so dropping it at least on master is reasonable.

[rvagg]

Support for Wheezy ends on 31st May 2018, which is shortly after Node 10.x release, so dropping it at least on master is reasonable.

+1 but how should we document this? tbh I'm not at all happy with this approach of documenting kernel versions, nobody thinks in terms of kernel versions. libc versions are a tiny bit more relevant but perhaps we need to use distribution names or release timeframes, or at least make note of it in our supported systems documentation.

[seishun]

or at least make note of it in our supported systems documentation.

The documentation already mentions that EOL distros are not supported: https://github.com/nodejs/node/blob/master/BUILDING.md#supported-platforms-1

[seishun]

It looks like things have changed since 29 Nov 2017 (#762 (comment)).

• cc-armv6 and cc-armv7 are gone. @joaocgreis is this intentional?
• A test commit that requires gcc 4.9 fails on centos7-arm64: https://ci.nodejs.org/job/node-test-commit-arm/12930/nodes=centos7-arm64/ Was this machine introduced, re-enabled or modified recently? How is it deployed? Is someone working on upgrading gcc there?

[joaocgreis]

@seishun the arm-fanned job was not available because of cluster maintenance. It's back now.

About the compiler, I don't know what other compiler we can use. If someone wants to give it a try, we can probably give access or work out some way.

[seishun]

@joaocgreis
Access would be great. I also happen to have a Raspberry Pi 3 (I just need to buy a cable and a memory card, I think) so I can tinker with it locally, but I would need help setting things up. For instance, what's the environment on Raspberry Pi on CI? How do I make sure the cross-compiled binary works? Is it enough to just copy the binary to the Raspberry Pi and run it, or are there more sophisticated steps?

Also, is there a particular reason why it uses binaries under https://github.com/raspberrypi/tools/tree/master/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin? Is there something special about them? For example, why not use binaries under https://github.com/raspberrypi/tools/tree/master/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin instead? (4.9.3 < 4.9.4, but it should work as a stopgap measure)

[mhdawson]

Instructions for setup are here: https://github.com/nodejs/build/tree/master/setup/raspberry-pi (at least that is what I used when I setup the one for the release CI a little while ago)

[seishun]

Are these instructions meant for the cross-compiling setup? The installation of GCC makes me suspect it isn't. And it doesn't mention the OS.

[mhdawson]

No, for compilation on the machines themselves. Maybe @rvagg know about cross-compiling.

[joaocgreis]

About the folder with 4.9.3 binaries, it did not yet exist when I set up the cross-compile machine. I gave those a try locally and they seem to work, but they can't compile V8 6.5. The cross-compile machine should be very easy to update if that's the way forward, but the release machines compile natively and I don't know what compiler we can use there - it doesn't make sense to update the test CI if the releases would fail.

About the CI setup, cross-compiling is essential to reduce the time it takes to compile (up to 1 day in some conditions), but there's no other reason to use it. Thus, the Raspberry Pi workers connected to test CI don't compile node, but compile the addons - in node-test-binary-arm the addons row should be the only one that needs a compiler installed. The cross-compiler was deployed (once, more that 2 years ago) with setup/cross-compiler and the Raspberry Pis with setup/raspberry-pi. The releases are compiled natively to ensure that node can still be compiled on the devices and to have a last line of defense against any issue with cross-compilation.

Testing this locally is not very hard, at lease should not require Ansible. To compile just follow and adapt the script of node-cross-compile, commands are echoed to the console in any run. To run tests use node-test-binary-arm. This boils down to: clone the raspberrypi/tools repository and adapt our script to the correct location. Then source it on the shell where node will be compiled and run CONFIG_FLAGS='--dest-cpu=arm' make build-ci -j 10. Use the tar command at the end of the script to pack the binaries. Then, on a Raspberry Pi (can be emulated with qemu, there are instructions online), check out the same revision of node, unpack the binaries and run make test-ci-js (the --run argument used in CI is to divide the tests by several runners).

If there is no supported compiler that we can use for Raspberry Pi, shouldn't we be talking about dropping support for it in node v10 instead? Are there sill supported OSs for the Pi 1?

[seishun]

I gave those a try locally and they seem to work, but they can't compile V8 6.5.

What was the error?

but the release machines compile natively and I don't know what compiler we can use there

Support for Wheezy ends on 31st May 2018, I assume it will be dropped in Node 10. Jessie seems to have gcc 4.9.2 in the repos, can that compile V8 6.5?

Are there sill supported OSs for the Pi 1?

According to Wikipedia, Jessie works on Raspberry Pi 1.

[rvagg]

Support for Wheezy ends on 31st May 2018, I assume it will be dropped in Node 10. Jessie seems to have gcc 4.9.2 in the repos, can that compile V8 6.5?

But we are stuck with it for previous versions of Node, we need to be careful not to drop support for distros, compilers and libc that we are still going to be shipping in 4, 6, 8 even if we're moving to deprecating them for future versions. We are running Wheezy on the Pi1 and Pi2's and so far have no plan to migrate to Jessie for them. I'm not really happy with this push to move our compiler versions up without a solid plan to continue to test and support our older LTS branches that we have locked in minimum libc version commitments to.

[seishun]

But we are stuck with it for previous versions of Node, we need to be careful not to drop support for distros, compilers and libc that we are still going to be shipping in 4, 6, 8 even if we're moving to deprecating them for future versions. We are running Wheezy on the Pi1 and Pi2's and so far have no plan to migrate to Jessie for them.

Is there an issue with upgrading gcc on the Pi3s and leaving the Pi1s and Pi2s as-is?

I'm not really happy with this push to move our compiler versions up without a solid plan to continue to test and support our older LTS branches that we have locked in minimum libc version commitments to.

Can you suggest a better plan, then?

[joaocgreis]

I gave those a try locally and they seem to work, but they can't compile V8 6.5.

What was the error?

The error is exactly the same as the one in nodejs/node-v8#35. Strange thing there is that it happens while compiling native code, but I tried it in a machine with gcc 5.4.0 and it still happens.

[seishun]

Then it was a bug in V8 that has been fixed upstream: nodejs/node-v8#35 (comment).

[seishun]

Status update. cc-armv6 and cc-armv7 now use gcc 4.9 on master and node-test-commit-arm-fanned successfully builds a commit that requires gcc 4.9.

There are now 5 platforms in CI that require a compiler upgrade:

• centos7-arm64. I asked about it in Use gcc 4.9.4 in CI #762 (comment) but didn't get a response. cc @nodejs/build
• centos6-64. This is unexpected because it was upgraded in ansible: use gcc 4.9 on CentOS 6 #809. Did someone revert the changes?
• centos7-64. This is a simple change in the Ansible scripts if Duplicate 'centos7' in the init dictionary #1034 is resolved.
• rhel72-s390x. Waiting for linuxOne: gcc upgrade #1023.
• aix61-ppc64. Waiting for AIX: gcc upgrade requirement #925.

[seishun]

Per #1150 (comment), the CentOS 6 machines were redeployed using ansible/ scripts that still install an old gcc version. This is surprising because they previously didn't work (#801) and the offending lines haven't been fixed. Though it's alluded to in #1076, which does seem to fix #801.

Can someone confirm the above? I would be glad to update the ansible/ scripts to install a newer gcc version, but it seems I'll have to wait until #1076 lands. cc @rvagg

[rvagg]

I've converted one of the two release Raspberry Pis to the new Docker setup and have this in ci-release (covers most *nix's, not just the Pis):

exec_cmd="make -j $JOBS binary-upload \
  DESTCPU=\"$DESTCPU\" \
  ARCH=\"$ARCH\" \
  DISTTYPE=\"$disttype\" \
  DATESTRING=\"$datestring\" \
  COMMIT=\"$commit\" \
  CUSTOMTAG=\"$CUSTOMTAG\" \
  RELEASE_URLBASE=\"$RELEASE_URLBASE\" \
  CONFIG_FLAGS=\"$CONFIG_FLAGS\" \
"

if [[ "$NODE_LABELS" =~ pi1-docker ]]; then
  echo "$exec_cmd" > node-ci-exec
  if test $NODE_MAJOR_VERSION -ge 10; then
    sudo docker-node-exec.sh -v jessie
  else
    sudo docker-node-exec.sh -v wheezy
  fi
else
  sh -c "$exec_cmd"
fi

So v10+ will build under Jessie, which has 4.9.2 but 9 and prior will be with Wheezy which is 4.8. Trying it out now with the latest nightly but there's quite a cache to build up for Jessie now so it'll be slow for a couple of days.

[rvagg]

@seishun yeah, I used ansible/ to redeploy a whole new set of centos6 machines iirc. it should work with that PR but there's an outstanding issue with ARCH & DESTCPU that's still outstanding. I'll work to get that resolved but you should be able to assume that it'll get merged in almost the form that it's in now and you're welcome to build on top of it with updates. Remember though that we are building releases with centos6 so we still need to be able to support older compilers too. It'd be good if we could have two variations, one for old and one for new, and we can apply them as appropriate. I don't really like the idea of having to have double the number of CentOS6 machines but perhaps we can do compiler selection in the Jenkins job like the PPC jobs do now.

[seishun]

Remember though that we are building releases with centos6 so we still need to be able to support older compilers too.

Wasn't that discussed back and forth in #797 and #809? Binaries produced by devtoolset work just fine on vanilla OSs, so there is no need to build LTS with gcc 4.8 specifically. For instance, #809 (comment) and #797 (comment).

[mhdawson]

Just want to mention that building with 4.9.4 broke compatibility for RHEL 7.X for PPCLE, not sure if that is the case for X86 as well ....

[mhdawson]

In the end I went back to using 4.8 for 9.X and lower and set it to use 4.9.X for 10 and higher.

[seishun]

Looks like every platform now uses gcc 4.9 or newer on master: https://ci.nodejs.org/job/node-test-commit/17876/

At least one job (node-cross-compile) uses a gcc version older than 4.9.4 (4.9.3 specifically), but it seems it would take significant effort to use a newer version there, and it's probably not worth it.

I think this issue can be closed now.

[joaocgreis]

At least one job (node-cross-compile) uses a gcc version older than 4.9.4 (4.9.3 specifically), but it seems it would take significant effort to use a newer version there, and it's probably not worth it.

@rvagg did it: #1237

[refack]

I believe this has been resolved by installing GCC-6 and documenting in e6a71a0