Skip to content

ci: use k8s pod template agents #2311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 18 commits into from
Closed

ci: use k8s pod template agents #2311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
725700f
ci: use k8s pod template agents
v1v Dec 5, 2021
4f79580
Debug
v1v Dec 5, 2021
e9e6ac8
More verbose
v1v Dec 5, 2021
e470af2
Otherwise it clashes with an existing OpenJdkPod in another repository
v1v Dec 5, 2021
8ca5d9a
use debian 11
v1v Dec 5, 2021
f5a2c5f
more debug
v1v Dec 5, 2021
e17e1d3
trying with mvn3 and jdk11
v1v Dec 5, 2021
49e04d8
Revert "trying with mvn3 and jdk11"
v1v Dec 5, 2021
4b4541c
Revert "use debian 11"
v1v Dec 5, 2021
1e16467
Revert "Otherwise it clashes with an existing OpenJdkPod in another r…
v1v Dec 5, 2021
c3f234f
Use the container step
v1v Dec 5, 2021
5fd959b
Remove debug
v1v Dec 5, 2021
c777625
Add next steps
v1v Dec 5, 2021
d4c425d
Prepare the context within the container closure
v1v Dec 5, 2021
1b59f0b
Enable matrix support
v1v Dec 5, 2021
8019abb
Trying to solve the concurrency issue with the top-level agent and t…
v1v Dec 5, 2021
700a358
use the right docker tags
v1v Dec 5, 2021
99f2581
Merge branch 'master' into feature/test-k8s-in-the-ci
mergify[bot] Dec 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .ci/k8s/OpenJdkPod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
apiVersion: v1
kind: Pod
spec:
containers:
- name: openjdk11
image: openjdk:11-jdk-buster
command:
- sleep
args:
- infinity
- name: openjdk12
image: openjdk:12
command:
- sleep
args:
- infinity
- name: openjdk13
image: openjdk:13
command:
- sleep
args:
- infinity
- name: openjdk14
image: openjdk:14
command:
- sleep
args:
- infinity
- name: openjdk15
image: openjdk:15
command:
- sleep
args:
- infinity
- name: openjdk16
image: openjdk:16
command:
- sleep
args:
- infinity
## TODO: Faster builds with some cached artifacts.
Copy link
Member Author

@v1v v1v Dec 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting enough:

image

vs

image

## mount a volume with the cached m2 folder in /var/lib/jenkins/.m2/repository
## for instance https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/maven-with-cache.groovy
## Maybe the infra folks already provide this caching!?
104 changes: 53 additions & 51 deletions Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
@Library('apm@current') _

pipeline {
agent { label 'linux && immutable' }
agent { kubernetes { yamlFile '.ci/k8s/OpenJdkPod.yml' } }
environment {
REPO = 'apm-agent-java'
BASE_DIR = "src/github.com/elastic/${env.REPO}"
Expand All @@ -16,6 +16,7 @@ pipeline {
ITS_PIPELINE = 'apm-integration-tests-selector-mbp/master'
MAVEN_CONFIG = '-Dmaven.repo.local=.m2'
OPBEANS_REPO = 'opbeans-java'
JDK_VERSION = 'openjdk11'
}
options {
timeout(time: 90, unit: 'MINUTES')
Expand Down Expand Up @@ -43,8 +44,6 @@ pipeline {
options { skipDefaultCheckout() }
environment {
HOME = "${env.WORKSPACE}"
JAVA_HOME = "${env.HUDSON_HOME}/.java/java11"
PATH = "${env.JAVA_HOME}/bin:${env.PATH}"
MAVEN_CONFIG = "${params.MAVEN_CONFIG} ${env.MAVEN_CONFIG}"
}
stages {
Expand Down Expand Up @@ -88,12 +87,14 @@ pipeline {
sh label: 'Prepare .m2 cached folder', returnStatus: true, script: 'cp -Rf /var/lib/jenkins/.m2/repository ${HOME}/.m2'
sh label: 'Size .m2', returnStatus: true, script: 'du -hs .m2'
}
dir("${BASE_DIR}"){
withOtelEnv() {
retryWithSleep(retries: 5, seconds: 10) {
sh label: 'mvn install', script: "./mvnw clean install -DskipTests=true -Dmaven.javadoc.skip=true"
container(env.JDK_VERSION) {
dir("${BASE_DIR}"){
withOtelEnv() {
retryWithSleep(retries: 5, seconds: 10) {
sh label: 'mvn install', script: "./mvnw clean install -DskipTests=true -Dmaven.javadoc.skip=true"
}
sh label: 'mvn license', script: "./mvnw org.codehaus.mojo:license-maven-plugin:aggregate-third-party-report -Dlicense.excludedGroups=^co\\.elastic\\."
}
sh label: 'mvn license', script: "./mvnw org.codehaus.mojo:license-maven-plugin:aggregate-third-party-report -Dlicense.excludedGroups=^co\\.elastic\\."
}
}
stash allowEmpty: true, name: 'build', useDefaultExcludes: false
Expand Down Expand Up @@ -121,11 +122,10 @@ pipeline {
Run only unit test.
*/
stage('Unit Tests') {
agent { kubernetes { yamlFile '.ci/k8s/OpenJdkPod.yml' } }
options { skipDefaultCheckout() }
environment {
HOME = "${env.WORKSPACE}"
JAVA_HOME = "${env.HUDSON_HOME}/.java/java11"
PATH = "${env.JAVA_HOME}/bin:${env.PATH}"
}
when {
beforeAgent true
Expand All @@ -135,12 +135,14 @@ pipeline {
withGithubNotify(context: 'Unit Tests', tab: 'tests') {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh """#!/bin/bash
set -euxo pipefail
./mvnw test
"""
container(env.JDK_VERSION) {
dir("${BASE_DIR}"){
withOtelEnv() {
sh """#!/bin/bash
set -euxo pipefail
./mvnw test
"""
}
}
}
}
Expand All @@ -155,24 +157,24 @@ pipeline {
Run smoke tests for different servers and databases.
*/
stage('Smoke Tests 01') {
agent { label 'linux && immutable' }
agent { kubernetes { yamlFile '.ci/k8s/OpenJdkPod.yml' } }
Copy link
Member Author

@v1v v1v Dec 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, this was the way to ensure the parallel stages don't clash with the workspace when using the top-level k8s definition.

I need to clarify this to be sure this is the right pattern to be used.

options { skipDefaultCheckout() }
environment {
HOME = "${env.WORKSPACE}"
JAVA_HOME = "${env.HUDSON_HOME}/.java/java11"
PATH = "${env.JAVA_HOME}/bin:${env.PATH}"
}
when {
beforeAgent true
expression { return params.smoketests_ci }
}
steps {
withGithubNotify(context: 'Smoke Tests 01', tab: 'tests') {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh './scripts/jenkins/smoketests-01.sh'
container(env.JDK_VERSION) {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh './scripts/jenkins/smoketests-01.sh'
}
}
}
}
Expand All @@ -187,24 +189,24 @@ pipeline {
Run smoke tests for different servers and databases.
*/
stage('Smoke Tests 02') {
agent { label 'linux && immutable' }
agent { kubernetes { yamlFile '.ci/k8s/OpenJdkPod.yml' } }
options { skipDefaultCheckout() }
environment {
HOME = "${env.WORKSPACE}"
JAVA_HOME = "${env.HUDSON_HOME}/.java/java11"
PATH = "${env.JAVA_HOME}/bin:${env.PATH}"
}
when {
beforeAgent true
expression { return params.smoketests_ci }
}
steps {
withGithubNotify(context: 'Smoke Tests 02', tab: 'tests') {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh './scripts/jenkins/smoketests-02.sh'
container(env.JDK_VERSION) {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh './scripts/jenkins/smoketests-02.sh'
}
}
}
}
Expand All @@ -224,8 +226,6 @@ pipeline {
options { skipDefaultCheckout() }
environment {
HOME = "${env.WORKSPACE}"
JAVA_HOME = "${env.HUDSON_HOME}/.java/java11"
PATH = "${env.JAVA_HOME}/bin:${env.PATH}"
NO_BUILD = "true"
}
when {
Expand Down Expand Up @@ -265,27 +265,27 @@ pipeline {
Build javadoc files.
*/
stage('Javadoc') {
agent { label 'linux && immutable' }
agent { kubernetes { yamlFile '.ci/k8s/OpenJdkPod.yml' } }
options { skipDefaultCheckout() }
environment {
HOME = "${env.WORKSPACE}"
JAVA_HOME = "${env.HUDSON_HOME}/.java/java11"
PATH = "${env.JAVA_HOME}/bin:${env.PATH}"
}
when {
beforeAgent true
expression { return env.ONLY_DOCS == "false" }
}
steps {
withGithubNotify(context: 'Javadoc') {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh """#!/bin/bash
set -euxo pipefail
./mvnw compile javadoc:javadoc
"""
container(env.JDK_VERSION) {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh """#!/bin/bash
set -euxo pipefail
./mvnw compile javadoc:javadoc
"""
}
}
}
}
Expand Down Expand Up @@ -328,7 +328,7 @@ pipeline {
}
}
matrix {
agent { label 'linux && immutable' }
agent { kubernetes { yamlFile '.ci/k8s/OpenJdkPod.yml' } }
axes {
axis {
// the list of support java versions can be found in the infra repo (ansible/roles/java/defaults/main.yml)
Expand All @@ -346,11 +346,13 @@ pipeline {
}
steps {
withGithubNotify(context: "Unit Tests ${JAVA_VERSION}", tab: 'tests') {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh(label: "./mvnw test for ${JAVA_VERSION}", script: './mvnw test')
container("${JAVA_VERSION}") {
deleteDir()
unstash 'build'
dir("${BASE_DIR}"){
withOtelEnv() {
sh(label: "./mvnw test for ${JAVA_VERSION}", script: './mvnw test')
}
}
}
}
Expand Down