PR Builds: New Features #6525

babsingh · 2022-05-20T15:39:25Z

Ubuntu 16 (Ub16) is out of service. Thus, Ub16 x86_64 Dockerfile has
been removed.
We can write a script to generate Dockerfiles to support different
operating systems (OSs) and architectures. For the time being, only a
Dockerfile for Ubuntu 20 x86_64 has been added. This Dockerfile
can be used to derive a script to generate Dockerfiles for other
OSs and architectures since it has all the fundamental components.
OMR_RUNNING_IN_DOCKER=1 environment variable is set when the PR builds
are run inside Docker. This will be used in tests to verify the
behaviour port library sysinfo functions.
PortStrTest.str_test4 is temporarily disabled since it fails in a
container. Issue tracker: PortStrTest.str_test4 fails inside a Docker container #6566.
Changes to the default behaviour:
a) linux_x86 PR builds will be run on cgroup.v1 nodes inside a container.
b) linux_x86-64 PR builds will be run on cgroup.v2 nodes inside a container.
c) linux_ppc-64_le_gcc PR builds will be run on cgroup.v2 nodes.
PR build changes:
a) Existing PR build syntax should have no impact.
b) All new features are additive.
c) New features allow PR builds to be configurable.
New command example: jenkins build xlinux(OPTION1,OPTION2,...)
The following options are added:
a) cgroupv1 -> add label to run the build on a cgroup.v1 node.
E.g. -- jenkins build xlinux(cgroupv1)
b) !cgroupv1 -> remove label to run the build on a cgroup.v1 node.
E.g. -- jenkins build xlinux(!cgroupv1)
c) cgroupv2 -> add label to run the build on a cgroup.v2 node.
E.g. -- jenkins build xlinux(cgroupv2)
d) !cgroupv2 -> remove label to run the build on a cgroup.v2 node.
E.g. -- jenkins build xlinux(!cgroupv2)
e) docker -> run the build inside a Docker container.
E.g. -- jenkins build xlinux(cgroupv2,docker)
f) !docker -> do not run the build inside a Docker container.
E.g. -- jenkins build xlinux(cgroupv1,!docker)
g) cmake:'ARGS' -> ARGS will be appended during the configure phase.
E.g. -- jenkins build xlinux(cmake:'-DCMAKE_C_FLAGS=-DDUMP_DBG')
h) compile:'ARGS' -> ARGS will be appended during the compile phase.
E.g. -- jenkins build xlinux(compile:'-d')
i) test:'ARGS' -> ARGS will be appended while running the tests.
E.g. -- jenkins build xlinux(test:'-R porttest')
j) env:'VAR1=VAL1,VAR2=VAL2' -> environment variables will be added.
E.g. -- jenkins build xlinux(env:'GTEST_FILTER=PortDumpTest.*')
Example: jenkins build xlinux(<OPTIONS_A>),all
In this example, the xlinux PR build will use OPTIONS_A whereas
all other PR builds will use their default settings/options.
Example: jenkins build xlinux(<OPTIONS_A>),all(<OPTIONS_B>)
In this example, the xlinux PR build will use OPTIONS_A whereas
all other PR builds will use OPTIONS_B.
Example: jenkins build xlinux(<OPTIONS_A>),all,linux_x86-64(<OPTIONS_B>)
In this example, xlinux is an alias for linux_x86-64. Two different
sets of options are provided for the same PR build specfication. The set
of options of provided at the end will be enforced. In this example,
OPTIONS_B will be used whereas OPTIONS_A will be ignored. The same
analogy is also used if N-sets of options are specified for the a PR
build specification.
Related:

Signed-off-by: Babneet Singh sbabneet@ca.ibm.com

babsingh · 2022-05-20T15:43:05Z

jenkins build linux_x86_cgroupv1_docker

babsingh · 2022-05-20T15:45:19Z

jenkins build linux_x86

babsingh · 2022-05-20T15:46:25Z

jenkins build linux_x86_cgroupv1_docker

babsingh · 2022-05-20T15:46:43Z

jenkins build linux_x86

babsingh · 2022-05-20T15:50:46Z

jenkins build xlinux

babsingh · 2022-05-20T15:53:02Z

jenkins build xlinux

babsingh · 2022-05-20T15:58:19Z

jenkins build xlinux

babsingh · 2022-05-20T16:01:51Z

@jdekonin @AdamBrousseau Docker Pipeline plugin needs to be installed for OMR Jenkins.

Ref: https://stackoverflow.com/questions/41215997/jenkins-docker-pipeline-error.

https://ci.eclipse.org/omr/job/PullRequest-linux_x86-64/3312/console

groovy.lang.MissingPropertyException: No such property: docker for class: groovy.lang.Binding
	at groovy.lang.Binding.getVariable(Binding.java:63)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:251)
        ...

babsingh · 2022-05-20T16:17:58Z

@AdamBrousseau Currently, jenkins build {SPEC} does not accept _cgroupv1, _cgroupv2 and _docker as valid keywords. Can we introduce rules such that these keywords can be appended to existing specs?

Also, we can reuse existing specs in omrbuild.groovy without adding any new ones for supporting cgroupv1/v2 and docker.

babsingh · 2022-05-26T18:38:33Z

As per eclipse-cbi/jiro#214 (comment), the Eclipse foundation does not support the Docker plugin in Jenkins:

We don't support docker plugin in Jenkins as it would require DiD in our current, which is a major security issue. You can run pipeline in custom container though, as described in https://wiki.eclipse.org/Jenkins#Custom_container. Note that the container must be build and publish beforehand.

buildenv/docker/x86_64/ubuntu20/Dockerfile

buildenv/jenkins/containerbuild.groovy

babsingh · 2022-05-27T16:48:44Z

Will address error handling and container size concerns once the test environment is alive i.e. job to upload docker image to dockerhub. Then, will be able to test PR builds. Also, regex restrictions need to be removed.

AdamBrousseau · 2022-06-23T18:27:27Z

00:00:06.891  + docker ps -a
00:00:06.892  c:/jenkins/workspace/PullRequest-win_x86-64/Build@tmp/durable-d77eac37/script.sh: line 1: docker: command not found

I think we need to "protect" against unexpected behaviour a bit. In particular, if a Windows machine had docker installed, it would run the default ub20 x64 container which would be a different environment than you would expect. However, I wouldn't object to leaving that for a future improvement.

babsingh · 2022-06-23T18:29:32Z

TODOs from the OMR arch meeting #6528:

Rename: configure -> cmake. [Completed]
Error handling for docker on unsupported platforms: win, plinux, etc. Plan to address in a future PR while adding a generic script to generate Dockerfiles. Currently, the xLinux Dockerfile should work on win, pLinux and zLinux without any blockers. Also, the docker option can be expanded: docker:'container_os=<ub20|ub22|win*|*>,arch=<z|p|x|a|*>,host_os=<label>'.
Install Docker on all supported platforms.
Add labels to check if docker is installed on a machine.

babsingh · 2022-06-23T18:47:05Z

jenkins build xlinux(!cgroupv2,test:'-R porttest',configure:'-DCMAKE_CXX_FLAGS=-DDUMP_DBG -DCMAKE_C_FLAGS=-DDUMP_DBG',env:'GTEST_FILTER=PortDumpTest.*',compile:'-j8')

babsingh · 2022-06-24T06:02:27Z

jenkins build xlinux(!cgroupv2,test:'-R porttest',configure:'-DCMAKE_CXX_FLAGS=-DDUMP_DBG -DCMAKE_C_FLAGS=-DDUMP_DBG',env:'GTEST_FILTER=PortDumpTest.*',compile:'-j8')

babsingh · 2022-06-24T06:06:06Z

jenkins build xlinux(!cgroupv2,test:'-R porttest',configure:'-DCMAKE_CXX_FLAGS=-DDUMP_DBG -DCMAKE_C_FLAGS=-DDUMP_DBG',env:'GTEST_FILTER=PortDumpTest.*',compile:'-j8')

babsingh · 2022-06-24T06:06:23Z

jenkins build xlinux(!cgroupv2,test:'-R porttest',cmake:'-DCMAKE_CXX_FLAGS=-DDUMP_DBG -DCMAKE_C_FLAGS=-DDUMP_DBG',env:'GTEST_FILTER=PortDumpTest.*',compile:'-j8')

babsingh · 2022-06-24T06:09:30Z

jenkins build x32linux(!cgroupv1,docker,test:'-R porttest',cmake:'-DCMAKE_CXX_FLAGS=-DDUMP_DBG -DCMAKE_C_FLAGS=-DDUMP_DBG',env:'GTEST_FILTER=PortDumpTest.*',compile:'-j8')

babsingh · 2022-07-14T12:57:23Z

jenkins build all

babsingh · 2022-07-14T13:35:49Z

New changes are summarized in the below table. It highilghts the minimal infra coverage for cgroup support.

Label	Run build specs directly on the machine	Run build specs inside a container
cgroup.v1	Others	linux_x86
cgroup.v2	linux_ppc-64_le_gcc	linux_x86-64

"Others" refer to the remaining Linux build specs which have machine nodes with old operating systems and no cgroup labels. These machine nodes have the cgroup.v1 configuration by default.

jdekonin · 2022-07-19T13:00:42Z

jenkins build plinux(cgroupv1)

babsingh · 2022-07-19T13:02:05Z

jenkins build plinux(cgroupv1)

jdekonin · 2022-07-19T13:32:18Z

jenkins build plinux(cgroupv2)

0xdaryl · 2022-09-21T21:43:11Z

Can you add the documentation for the new Jenkins launch options to the Committer Guide? [1] It's really the only doc we have on the Jenkins options and it saves developers having to find this PR for usage info. Thanks.

[1] https://github.com/eclipse/omr/blob/master/doc/GuideForCommitters.md

1. Ubuntu 16 (Ub16) is out of service. Thus, Ub16 x86_64 Dockerfile has been removed. 2. We can write a script to generate Dockerfiles to support different operating systems (OSs) and architectures. For the time being, only a Dockerfile for Ubuntu 20 x86_64 has been added. This Dockerfile can be used to derive a script to generate Dockerfiles for other OSs and architectures since it has all the fundamental components. 3. OMR_RUNNING_IN_DOCKER=1 environment variable is set when the PR builds are run inside Docker. This will be used in tests to verify the behaviour port library sysinfo functions. 4. PortStrTest.str_test4 is temporarily disabled since it fails in a container. Issue tracker: eclipse-omr#6566. 5. Changes to the default behaviour: a) linux_x86 PR builds will be run on cgroup.v1 nodes inside a container. b) linux_x86-64 PR builds will be run on cgroup.v2 nodes inside a container. c) linux_ppc-64_le_gcc PR builds will be run on cgroup.v2 nodes. 6. PR build changes: a) Existing PR build syntax should have no impact. b) All new features are additive. c) New features allow PR builds to be configurable. 7. New command example: jenkins build xlinux(OPTION1,OPTION2,...) 8. The following options are added: a) cgroupv1 -> add label to run the build on a cgroup.v1 node. E.g. jenkins build xlinux(cgroupv1) b) !cgroupv1 -> remove label to run the build on a cgroup.v1 node. E.g. jenkins build xlinux(!cgroupv1) c) cgroupv2 -> add label to run the build on a cgroup.v2 node. E.g. jenkins build xlinux(cgroupv2) d) !cgroupv2 -> remove label to run the build on a cgroup.v2 node. E.g. jenkins build xlinux(!cgroupv2) e) docker -> run the build inside a Docker container. E.g. jenkins build xlinux(cgroupv2,docker) f) !docker -> do not run the build inside a Docker container. E.g. jenkins build xlinux(cgroupv1,!docker) g) cmake:'ARGS' -> ARGS will be appended during the configure phase. E.g. jenkins build xlinux(cmake:'-DCMAKE_C_FLAGS=-DDUMP_DBG') h) compile:'ARGS' -> ARGS will be appended during the compile phase. E.g. jenkins build xlinux(compile:'-d') i) test:'ARGS' -> ARGS will be appended while running the tests. E.g. jenkins build xlinux(test:'-R porttest') j) env:'VAR1=VAL1,VAR2=VAL2' -> environment variables will be added. E.g. jenkins build xlinux(env:'GTEST_FILTER=PortDumpTest.*') 9. Example: jenkins build xlinux(<OPTIONS_A>),all In this example, the xlinux PR build will use OPTIONS_A whereas all other PR builds will use their default settings/options. 10. Example: jenkins build xlinux(<OPTIONS_A>),all(<OPTIONS_B>) In this example, the xlinux PR build will use OPTIONS_A whereas all other PR builds will use OPTIONS_B. 11. Example: jenkins build xlinux(<OPTIONS_A>),all,linux_x86-64(<OPTIONS_B>) In this example, xlinux is an alias for linux_x86-64. Two different sets of options are provided for the same PR build specfication. The set of options of provided at the end will be enforced. In this example, OPTIONS_B will be used whereas OPTIONS_A will be ignored. The same analogy is also used if N-sets of options are specified for the a PR build specification. 12. Related: - eclipse-omr#1281 - eclipse-omr#6468 - eclipse-omr#6477 - eclipse-omr#6501 Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>

babsingh · 2022-09-21T22:58:24Z

Can you add the documentation for the new Jenkins launch options to the Committer Guide?

@0xdaryl Documentation added in a separate commit for the time being. If needed, I will squash it after the review.

buildenv/jenkins/omrbuild.groovy

babsingh requested review from charliegracie and mstoodle as code owners May 20, 2022 15:39

babsingh marked this pull request as draft May 20, 2022 15:41

babsingh marked this pull request as ready for review May 20, 2022 15:46

babsingh changed the title ~~Run PR builds in Docker~~ [WIP] Run PR builds in Docker May 20, 2022

github-actions bot added build / configure ci labels May 20, 2022

babsingh force-pushed the cgroupv2_infra branch from bc9c021 to 4d54396 Compare May 20, 2022 15:50

babsingh force-pushed the cgroupv2_infra branch from 4d54396 to 3072b49 Compare May 20, 2022 15:52

babsingh force-pushed the cgroupv2_infra branch from 3072b49 to 1dcbf0f Compare May 20, 2022 15:56

babsingh mentioned this pull request May 25, 2022

Need Docker plugin in OMR Jenkins eclipse-cbi/jiro#214

Closed

babsingh force-pushed the cgroupv2_infra branch 6 times, most recently from 8e3bdb7 to 1cd54c6 Compare May 26, 2022 18:00

AdamBrousseau suggested changes May 27, 2022

View reviewed changes

babsingh force-pushed the cgroupv2_infra branch from 1cd54c6 to c44371c Compare May 27, 2022 16:41

babsingh force-pushed the cgroupv2_infra branch from 4544ffb to 311712b Compare June 23, 2022 18:37

babsingh force-pushed the cgroupv2_infra branch from 311712b to 1318642 Compare June 24, 2022 06:00

babsingh force-pushed the cgroupv2_infra branch from 1318642 to f27ffa9 Compare July 14, 2022 12:53

babsingh mentioned this pull request Sep 21, 2022

[Don't Merge] Investigate porttest failures on ub20 390 #6719

Closed

charliegracie approved these changes Sep 21, 2022

View reviewed changes

babsingh force-pushed the cgroupv2_infra branch from f27ffa9 to fdf0c72 Compare September 21, 2022 22:38

babsingh requested a review from 0xdaryl as a code owner September 21, 2022 22:38

babsingh force-pushed the cgroupv2_infra branch 2 times, most recently from 30c44c4 to eed05cf Compare September 21, 2022 22:55

Documentation for the new PR build features

cdfc670

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>

babsingh force-pushed the cgroupv2_infra branch from eed05cf to cdfc670 Compare September 21, 2022 22:57

0xdaryl approved these changes Sep 22, 2022

View reviewed changes

0xdaryl merged commit 60618df into eclipse-omr:master Sep 22, 2022

keithc-ca reviewed Sep 22, 2022

View reviewed changes

buildenv/jenkins/omrbuild.groovy Show resolved Hide resolved

babsingh mentioned this pull request Sep 27, 2022

Fix discrepancy in GC Extensions Base functions #6715

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PR Builds: New Features #6525

PR Builds: New Features #6525

babsingh commented May 20, 2022 •

edited

Loading

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 26, 2022

babsingh commented May 27, 2022 •

edited

Loading

AdamBrousseau commented Jun 23, 2022

babsingh commented Jun 23, 2022 •

edited

Loading

babsingh commented Jun 23, 2022

babsingh commented Jun 24, 2022

babsingh commented Jun 24, 2022

babsingh commented Jun 24, 2022

babsingh commented Jun 24, 2022

babsingh commented Jul 14, 2022

babsingh commented Jul 14, 2022

jdekonin commented Jul 19, 2022

babsingh commented Jul 19, 2022

jdekonin commented Jul 19, 2022

0xdaryl commented Sep 21, 2022

babsingh commented Sep 21, 2022

PR Builds: New Features #6525

PR Builds: New Features #6525

Conversation

babsingh commented May 20, 2022 • edited Loading

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 20, 2022

babsingh commented May 26, 2022

babsingh commented May 27, 2022 • edited Loading

AdamBrousseau commented Jun 23, 2022

babsingh commented Jun 23, 2022 • edited Loading

babsingh commented Jun 23, 2022

babsingh commented Jun 24, 2022

babsingh commented Jun 24, 2022

babsingh commented Jun 24, 2022

babsingh commented Jun 24, 2022

babsingh commented Jul 14, 2022

babsingh commented Jul 14, 2022

jdekonin commented Jul 19, 2022

babsingh commented Jul 19, 2022

jdekonin commented Jul 19, 2022

0xdaryl commented Sep 21, 2022

babsingh commented Sep 21, 2022

babsingh commented May 20, 2022 •

edited

Loading

babsingh commented May 27, 2022 •

edited

Loading

babsingh commented Jun 23, 2022 •

edited

Loading