Add reproducible comparison tests for windows and update tests for linux running out of docker image

[sophia-guo]

Linux support for reproducible comparision has supported without docker image requirement and windows support for building from SBOM script is available.

• linux ( not in docker image ) Update reproducible comparing tests on linux (not docker image) #3687

• windows Add reproducible comparison tests on windows #3711

Tests in aqa-tests can be updated to support more generic cases.

[sophia-guo]

Basically for running tests in jenkins anything needed to install in script https://github.com/adoptium/temurin-build/blob/master/tooling/reproducible/linux_repro_build_compare.sh and https://github.com/adoptium/temurin-build/blob/master/tooling/reproducible/windows_repro_build_compare.sh will have the permission issue.

[sxa]

What is the reason / use-case for wanting to do the Linux comparison outside docker? I'm unclear from this why that is required.

[smlambert]

What is the reason / use-case for wanting to do the Linux comparison outside docker? I'm unclear from this why that is required.

3-fold:

1. Running this test inside of a Docker container means we need to limit the test job to machines with the sw.tool.docker label, so a small set of machines, meaning longer queue times for dev.system and means the other test targets in dev.system get run in containers even though that is not a requirement for them.
2. More importantly, we would eventually like to demonstrate the ability to reproduce in various environments, including outside of containers (as we will do with the Windows test). To exercise the build 'recipe' / formulation we capture in the SBOM.
3. We will extend this test to also handle the case of 'comparability' with Temurin or Vanilla builds as the base, and other vendor builds as the test SDK to compare. For this scenario, the requirement is to demonstrate the rebuild outside of a container.

Later, we may also want to re-include a ContainerizedReproducibleTest test target that runs inside of a container that can be run manually, to verify that use-case is also still working.

[andrew-m-leonard]

The test nodes are setup using the same playbooks as the docker image, so to mostly extent dependencies should be the same.
However, the thing that may get us is the OS sysroot headers/glibc, but lets test it out and see... if that proves to be the case then we will need the devkit to resolve that.

[sxa]

I thought we already know that a different version of a Linux distribution was unable to give an identical build since we cannot guarantee that all of the system packages are identical, so this feels like a bit of a non-starter to me at the moment. We /may/ be able to get closer with a devkit but IMHO this won't work just now.

The test nodes are setup using the same playbooks as the docker image

This is incorrect. The static docker machines (those named test-docker which are a high proportion of the test systems, especially on aarch64 and x86-64) are set up with a minimal amount of packages to run the tests across a wide range of distributions to keep the image size down and to make sure we test in as minimal an environment as possible.

I would have thought that the thing we wanted to achieve for rebuilding from an SBoM was to have a clean image of a suitable type (which is why docker would be ideal) add the prerequisites from the SBoM (Currently they are defined in a list at

temurin-build/tooling/reproducible/linux_repro_build_compare.sh

Line 26 in e282be5

installPrereqs() {

using the CentOS/RHEL package names but that can hopefully be adjusted with the strace output) and run the build to allow the best chance of reproducibility.

While I appreciate that the choice to add this in the dev AQA suite means that requiring docker is suboptimal integrating it into that suite is a choice we have made which may not be appropriate for the goal, or have I misunderstood the goal and the aim is to make it work on any distribution (which is, IMHO, a signficantly larger undertaking of which I'm a bit unclear of the value)?

[andrew-m-leonard]

The test nodes are setup using the same playbooks as the docker image

This is incorrect. The static docker machines (those named test-docker which are a high proportion of the test systems, especially on aarch64 and x86-64) are set up with a minimal amount of packages to run the tests across a wide range of distributions to keep the image size down and to make sure we test in as minimal an environment as possible.

Ah, thanks for clarifying, I don't think we realised that, that could be an issue then

[sophia-guo]

Jenkins user doesn't have the permission to install whatever the prerequisites from the SBoM.

[sxa]

Jenkins user doesn't have the permission to install whatever the prerequisites from the SBoM.

Correct (and it should not have those permissions otherwise it would be able to affect the setup of the machine which would be a security concern).

How does this work for the existing reproducible build tests that are in the pipelines? Are they testing in a clean CentOS environment (such as a new docker container) or using the existing docker build container which has all of the appropriate things installed? I feel that one of those two options would be the preferred solution here. Ideally the first (and that's what I did to test the initial reproducibility script for Linux) but the second would be a valid comparison too.

[sophia-guo]

The existing reproducible build tests that are in the pipelines use the exact same environment as the temurin build itself. All linuxs are using the specified dockerImage -adoptopenjdk/****_build_image - using dockeimage at jenkins level.

Reproducible comparing tests in aqa-tests - running with dockerimage at TKG level ( docker command)

[sxa]

Reproducible comparing tests in aqa-tests - running with dockerimage at TKG level ( docker command)

I'm now confused - I thought from the title that this issue was about running outside of a docker image? Is the intention here to run the same docker image, but just from the TKG level? And is it the centos7_build_image or a based centos7? If we can do it from a base centos7 (which would allow you to install all of the packages from the SBoM as you could run in the container as root) from TKG that would be great!

In which case there is no concern with the ability to have a reproduce comparison, but it would still need to be tied to a machine that can run docker (which from Shelley's comment we can't currently do if it's in the dev suite) as we won't be able to do that in a test-docker- machine.

[sophia-guo]

Reproducible comparing tests in aqa-tests - running with dockerimage at TKG level ( docker command)

Using docker image one the initially one or stage one as originally according to the docs only docker one is available.

This issue is trying to run outside of a docker image for linuxs, which is more generic?

[sxa]

Using docker image one the initially one or stage one as originally according to the docs only docker one is available.

I had a chat with Sophia to clarify the above. "Stage one" here refers to adoptium/aqa-tests#4868

"the docs" referring to doing things in docker is the build wiki page at https://github.com/adoptium/temurin-build/wiki/How-to-create-a-%22reproducible-build%22-on-a-Linux-OS which uses the centos7_build_image

When I refer to a "clean" or "base" image I'm talking about the upstream docker images e.g. the output from docker pull centos:7 that has nothing on it. To use that - which would be preferred option - we would need to run in a docker container to be able to use yum. Any attempt to install packages onto a different distribution would likely fail because the names and versions would not be the same (especially for non-rpm distributions such as Debian/Ubuntu)

[sophia-guo]

According to the discussion with @sxa and @smlambert

Ideally

• ReproducibleTest for all platforms runs without need of Docker
• ReproducibleTest runs regularly in dev.system - update remove current extra docker label for dev.system

If can only run with Docker for linux

• ReproducibleTest for Linux runs in Docker - in special.system with additionalNodeLabel=sw.tool.docker
• ReproducibleTest for non-Linux (mac, windows) in dev.system no additionalNodeLabel.

[sophia-guo]

Failed adoptium/aqa-tests#5345. To help debug two updates will be helpful

• archive sbom file
• instant exit if jdk build fails.

[sophia-guo]

Tests are running in as part of aqa-tests builds, close this.