CI appears broken on `main`

[bessbd]

All commits merged to main are failing since Feb 26.
They fail with

21:05:37  ==> erlfdb (compile)
21:05:38  Compiling c_src/atoms.c
21:05:38  Compiling c_src/main.c
21:05:38  In file included from c_src/main.c:18:
21:05:38  c_src/fdb.h:17:10: fatal error: 'foundationdb/fdb_c.h' file not found

which is weird, because https://github.com/apache/couchdb-ci/blob/main/bin/apt-dependencies.sh#L190 installs FoundationDB just like https://github.com/apache/couchdb-config/blob/main/.travis.yml .
(And branch / PR CI looks fine)

[bessbd]

I'd be happy to open a PR for this if someone helps me figure out why it's broken.

[wohali]

@bessbd I'll have a look at this later today and we can talk through together what's up.

If you try to follow the steps in https://github.com/apache/couchdb/blob/main/build-aux/Jenkinsfile.full (this is run after any merge to main) by hand, using the Docker images, can you reproduce it?

It may be as simple as re-building the Docker containers and re-pushing them. This is not currently done automatically.

[wohali]

This isn't something that can be fixed with a checkin. The macOS machine is failing to build: https://ci-couchdb.apache.org/blue/organizations/jenkins/jenkins-cm1%2FFullPlatformMatrix/detail/main/49/pipeline

@janl We need your help to fix this.

We also have FreeBSD machines that are presently offline. There was a stoarge failure AND backup failure (due to bugs in both ZFS and Ceph!) and the machines need to be rebuilt. This is presently masked by the macOS failures. I will rebuild them this evening,

We also need to rebuild the arm worker, this is Amazon's fault, they deleted our instance again - a year to the exact day it was recreated after the last time it got nuked unexpectedly. I've asked for a root cause analysis; nothing yet.

FYI, PRs only build on 3x Debian x86 instances, which is why they pass. Those images have fdb in them, as you pointed out.

[wohali]

I've gotten the ARM node back online. I've relaunched the couchdb-arm-1 node on AWS. For posterity here is the configuration:

While @davisp 's ansible setup could be used to manage this host as well, I did it by hand for expedience.

I ssh'ed into the machine and ran this to get prereqs installed, to create the Jenkins user, and to install the ASF Infra pubkey:

sudo apt update
sudo apt upgrade -y
sudo apt install -y openjdk-11-jre
sudo groupadd -g 910 jenkins
sudo useradd -u 910 -g 910 -m jenkins
sudo mkdir /home/jenkins/.ssh
echo "<PUBKEY FROM PRIVATE REPO>" | sudo tee /home/jenkins/.ssh/authorized_keys >/dev/null
sudo chown -R jenkins:jenkins /home/jenkins/.ssh
sudo chmod -R og-rwx /home/jenkins/.ssh

then, for Docker from https://docs.docker.com/engine/install/debian/:

sudo apt install -y apt-transport-https ca-certificates curl gnupg
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io
# check functionality
sudo docker run hello-world

I then went to Jenkins and updated the IP address to the new elastic IP address from AWS, and updated the hostkey with the contents of the node's /etc/ssh/ssh_host_ecdsa_key.pub . I pressed the Save button, then the Launch Agent button. The log showed it connected, downloaded, and ran the Jenkins agent.

The node then needed the nightly cronjobs from here, which I added by hand. The kill-old-docker.sh script is in the same repo.

---

Still TODO: the FreeBSD nodes, and when Jan can get to it, the Mac node.

[wohali]

The FreeBSD nodes are now back online. As far as I can tell, main has never successfully built on these nodes, so there is still work to be done.

See #3399 if you'd like to help, and paging @dch .

[janl]

the arm build seem to be failing again with the OP error message, so Jenkins doesn’t go all the way to the Mac build.

The fix in couchdb-erlfdb fix apache/couchdb-erlfdb@c25d524#diff-427d8ddce21ccc4fae24fb547c7a82c2a793ed57208d8343e519e0aec8119ff7 should also have fixed things on Mac.

[wohali]

Thanks @janl !

Looks like a minor naming mistake in couchdb-ci/build.sh prevented the arm and ppc images from getting pushed to docker hub correctly. I'm rebuilding the containers now, just to freshen them, pushing, and that should get us back in business.

[wohali]

OK, as it turns out, Apple don't provide packages for fdb on anything but x86_64 for Linux, so I had to add support for building that from scratch. That's building now (it's sloooowwwww). Once it's done I'll push a new arm64v8 container and the jobs should start passing. Note that as we don't have any ppc64le workers, I didn't bother rebuilding that image.

For those interested in how all this gubbins works, there is a job in Jenkins called "Update Docker Containers" that runs on all of the nodes and pulls the latest docker container every 24h. I've just updated it to also pull all the containers on arm and ppc (if workers ever come back.) If you have to push a new container and want it working now, just run the job by hand.

[wohali]

Update: fdbserver/fdbcli won't run inside of a cross-arch docker container, with the same error as https://forums.foundationdb.org/t/foundationdb-does-not-run-on-windows-subsystem-for-linux-wsl/809/10 . This causes a problem on install of the program, which I'm doing after building its package. It tries to run fdbcli and fails.

I've worked around this with a || true in the build script.

I tested the docker container on our arm64v8 worker, and it works, so I'm completing the automated build overnight and will push the image in the morning. At that point, we should be building main on all available secondary platforms again - Linux arm64v8, macOS, FreeBSD.

Does this mean it could run on a Raspberry Pi 4? Cuz that would be really cool :D — Bill Stephenson

[wohali]

CouchDB 3.x already does, and packages (that run under Raspbian) have been available for at least a year now.

This is specifically talking about main, aka future CouchDB 4.0, which is not yet released.

I was under the impression that CouchDB wouldn’t run on the new “Pi 4 8GB” because it uses an ARM chip (or something like that). It still runs on a P3 though and they’re still selling them so that’s good. — Bill

…

On Mar 7, 2021, at 12:19 AM, Joan Touzet ***@***.***> wrote: CouchDB 3.x already does, and packages (that run under Raspbian) have been available for at least a year now. This is specifically talking about main, aka future CouchDB 4.0, which is not yet released. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3TDEK6MCE22T4TIIRI5RDTCMLI3ANCNFSM4YPGWJPQ>.