Add docker build for s390x

[BbolroC]

Feature:
To make the docker images for s390x (Z-system) available

Rational
We are interested in Knative on the Z architecture (s390x). One of the core Knative components, Eventing, uses Zipkin for the cloud event tracing (See the use case). Knative already has the multi-arch nightly builds including the s390x support. So it makes sense for us to use the same Eventing s390x stack, as it has done for amd64.

Example Scenario
Clients who have Z hardware and want to use Knative on top of it can use Zipkin without any technical hurdles.

[codefromthecrypt]

Thanks for raising the issue @BbolroC I'll make comments and bold questions for you. Firstly, I noticed you raised some PRs so assume you are willing to help with this. However, until others mention it, we have to assume no-one else is familiar with this arch. Are you willing to be pinged in the future about s390x?

We have a "rule of three" policy here on things that add maintenance costs to the project. Sometimes vendors add their own hardware options before clients request them, but in volunteer OSS, we don't do the "if you build it, they will come" as sometimes.. they don't come and we are stuck with the tech debt, and no one to ask if it was ever used. You mentioned this is for clients who have Z hardware. Can you have a couple users comment directly on this issue that they will use this?

Some notes s390x:

• our base image will likely work https://pkgs.alpinelinux.org/packages?name=openjdk15&branch=edge
• automatically downloading NPM cross-platform won't work due to missing musl+390x here
  • musl isn't here either
  • installing NPM directly should work the same as arm64 though https://pkgs.alpinelinux.org/packages?name=nodejs-current
• integration tests will have similar problems to arm64 Support building Zipkin on ARM64 #3166 as we build docker images for tests. Notably Cassandra was a problem on arm64 and while isn't officially supported still, it happens to work at the moment.

travis recently cut services significantly and we can't even test arm64 anymore. We have to switch to GH actions which only has x86. In the past arm64 images failed and no one knew it and it cost me personally a lot of time. Do you know a free CI that can boot a s390x to ensure it actually works? (without use of an emulator)

cc @openzipkin/core just in case any of you have stake in s390x and can comment back

[codefromthecrypt]

cc @odidev not that you use s390x, but just in case as I know you have a lot of experience after the arm64 thing

[BbolroC]

Thanks for raising the issue @BbolroC I'll make comments and bold questions for you. Firstly, I noticed you raised some PRs so assume you are willing to help with this. However, until others mention it, we have to assume no-one else is familiar with this arch. Are you willing to be pinged in the future about s390x?

--> Of course, I am willing to help the s390x enablement and provide the architectural expertise in the future on demand. But I cannot guarantee to take the maintenance responsibility of the whole zipkin projects for s390x.

We have a "rule of three" policy here on things that add maintenance costs to the project. Sometimes vendors add their own hardware options before clients request them, but in volunteer OSS, we don't do the "if you build it, they will come" as sometimes.. they don't come and we are stuck with the tech debt, and no one to ask if it was ever used. You mentioned this is for clients who have Z hardware. Can you have a couple users comment directly on this issue that they will use this?

--> I cannot bring you any client's comments because the Knative enablement on Z-system is in progress. We try to start this s390x enablement for Zipkin because Zipkin is deployed as the tracing configuration with the docker image docker.io/openzipkin/zipkin in the upstream's e2e test. (but not really used in the tests, just configuration check so far) I can say we could put this discussion underneath again and would bring it back when the real clients using the Knative on Z-system would make a request for the Zipkin.

Some notes s390x:

• our base image will likely work https://pkgs.alpinelinux.org/packages?name=openjdk15&branch=edge

• automatically downloading NPM cross-platform won't work due to missing musl+390x here

  • musl isn't here either
  • installing NPM directly should work the same as arm64 though https://pkgs.alpinelinux.org/packages?name=nodejs-current

• integration tests will have similar problems to arm64 Support building Zipkin on ARM64 #3166 as we build docker images for tests. Notably Cassandra was a problem on arm64 and while isn't officially supported still, it happens to work at the moment.

travis recently cut services significantly and we can't even test arm64 anymore. We have to switch to GH actions which only has x86. In the past arm64 images failed and no one knew it and it cost me personally a lot of time. Do you know a free CI that can boot a s390x to ensure it actually works? (without use of an emulator)

--> Thanks for sharing the infos. I can image that you and the project members have been getting through so many problems to support the ARM. Regarding the CI support, as far as I know, Travis is the only available CI. Other than Travis, I do not know other available ones without emulation.

[odidev]

cc @odidev not that you use s390x, but just in case as I know you have a lot of experience after the arm64 thing

@adriancole
I don't have much idea on s390x architecture. I mostly work on arm64 architecture only. If I get any pointers in this I will surely try from my side and suggest here.

[codefromthecrypt]

thanks @BbolroC and @odidev.. we are in the process of rolling out an org-wide switch to GH actions. let's see if we get a few 👍 here... if so, it shouldn't be too hard to implement s390x based on what we did in arm64

(meanwhile s390x sounds like a cyberpunk film title)

[codefromthecrypt]

one problem we have is that s390x is behind on JDK 11 and 15

ex https://pkgs.alpinelinux.org/packages?name=openjdk11&branch=edge

amd64 and aarch64 is 11.0.9_p11-r0
s390x is 11.0.8_p10-r0

we keep a uniform version, that they are published mutually exclusive wrt patch this makes it impossible to use these packages

[codefromthecrypt]

@bratkartoffel are you the openjdk11 maintainer for alpine? was curious if so, if you have insight into the version skew on edge between s390x and amd64

[codefromthecrypt]

looks like there's already an issue about the latest build problem https://gitlab.alpinelinux.org/alpine/aports/-/issues/12104

[codefromthecrypt]

https://gitlab.alpinelinux.org/alpine/aports/-/issues/12128 also about crashes in various commands on s390x

[codefromthecrypt]

testcontainers/moby-ryuk#18 about testcontainers support

[BbolroC]

Thanks for the comment, @adriancole . ATM, I am sick and stay in the bed. From the the next week to the 1st week of December I will be busy in my private matter. Is it okay for you that I am not as that much responsive as I told you in this period of time? Sorry.

[codefromthecrypt]

@BbolroC we are async here, which was why I dumped all the tasks.. wasn't to suggest there was an urgency to this. most things like arch support get better as time passes anyway. get well soon regardless!

[codefromthecrypt]

for those following, this is still stuck as we need qemu to not crash in order to slide into our build process. I captured current status in the description of the pending PR and also a note on how to see the crash quickly openzipkin/docker-java#52 (comment)

[codefromthecrypt]

I think this has been done for a long time, but I just verified we're good now also.

$ docker buildx imagetools inspect openzipkin/zipkin
Name:      docker.io/openzipkin/zipkin:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:2839eaf9d3502172cc6465942c38d675da902d1f509d8d2447c7e50145dfdbc8
           
Manifests: 
  Name:      docker.io/openzipkin/zipkin:latest@sha256:a2c81b6b2159685f22df7e1101ee745d167f5e154465650bd0b32f4866573005
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      docker.io/openzipkin/zipkin:latest@sha256:2b3db60be51b29a567d542e6a1a82ef47659a8d23188529a72f1a1b191ec353a
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64
             
  Name:      docker.io/openzipkin/zipkin:latest@sha256:18ff14ecc46ef79505c0bd747db24bf5bd48c6d4a0b2498ca9d3dc55bef8cb31
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/s390x