Introduce ci workflow for kubernetes/openshift end to end testing #17674

iocanel · 2021-06-03T15:17:26Z

This pull request introduces a new workflow for running end to end tests using minikube and oc cluster up (this limits us to v3.11 but its better than nothing).

In detail:

The workflow is meant to be triggered only for pull requests labelled with the kubernetes label.
When the workflow is triggered it will run two new jobs: kubernetes and openshift.

The kubernetes job will spin up minikube and then trigger the maven invoker intgration tests for kubernetes.
The openshift job will spin up oc cluster up and then trigger the maven invoker intgration tests for openshift.

The integration tests have been also modified so that can be selected using profiles.

quarkus-bot · 2021-06-03T15:17:28Z

Thanks for your pull request!

The title of your pull request does not follow our editorial rules. Could you have a look?

title should preferably start with an uppercase character (if it makes sense!)

This message is automatically generated by a bot.

quarkus-bot · 2021-06-03T15:40:05Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `6ece4ee`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Download Maven Repo`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Download Maven Repo`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-03T16:59:40Z

This workflow status is outdated as a new workflow run has been triggered.

🚫 This workflow run has been cancelled.

Failing Jobs - Building `716c247`

⚠️ Artifacts of the workflow run were not available thus the report misses some details.

Status	Name	Step	Test failures	Logs	Raw logs
⌛	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-03T19:46:30Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `81cb75c`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-03T19:46:36Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `81cb75c`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-03T19:48:49Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `81cb75c`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-03T21:07:21Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `13a967b`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-03T21:20:49Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `87c4f91`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-04T06:59:34Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `562b23a`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-04T08:31:27Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `d886021`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

quarkus-bot · 2021-06-04T10:18:29Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `28d8ba5`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Kubernetes Integration Tests (v1.20.1)	`Run Kubernetes Invoker Tests`	⚠️ Check →	Logs	Raw logs
✖	Openshift Integration Tests (v3.11.0)	`Run Openshift Invoker Tests`	⚠️ Check →	Logs	Raw logs

.github/workflows/ci-kuberenetes-itests.yml

maxandersen · 2021-06-04T10:51:10Z

.github/workflows/ci-kuberenetes-itests.yml

+    branches:
+      - main
+  pull_request:
+    types: [ opened, edited, labeled, reopened, synchronize ]


a bit concerned abuot enabling this kinda run on all prs...wont we run out of resources quite quickly that way ? :)

Agree, this is why the workflow is limited only to prs labelled with area/kubernetes (see condition below)

maxandersen · 2021-06-04T10:53:55Z

.github/workflows/ci-kuberenetes-itests.yml

+      - name: Run Openshift Invoker Tests
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+          export QUARKUS_CONTAINER_IMAGE_GROUP=${{ github.actor }}


wont you need a way to make the published image unique - i.e. based on which pr/branch its running againts to avoid collisions?

That's a very good point. Maybe, we should set the version to something that includes the pr number, timestamp etc.

let's try with the sha

iocanel · 2021-06-10T08:11:34Z

@maxandersen @gsmet: I have nothing more to add! Can you please review?

iocanel · 2021-06-22T11:56:25Z

@maxandersen @gsmet: ping

maxandersen · 2021-06-22T21:47:45Z

Looks good to me. @gsmet any concerns we enable this now or wait after 2.0 build ?

iocanel · 2021-07-12T11:47:38Z

@gsmet: ping

quarkus-bot · 2021-07-12T15:54:57Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `e17f07c`

Status	Name	Step	Test failures	Logs	Raw logs
✖	JVM Tests - JDK 11	`Build`	⚠️ Check →	Logs	Raw logs
✖	JVM Tests - JDK 16	`Build`	⚠️ Check →	Logs	Raw logs
✖	Native Tests - Misc4	`Build`	⚠️ Check →	Logs	Raw logs

gsmet

Thanks, very promising! It needs a bit more work though.

.github/workflows/ci-kubernetes-itests.yml

...ts/kubernetes/maven-invoker-way/src/it/kubernetes-docker-build-and-deploy/invoker.properties

quarkus-bot · 2021-07-13T15:18:04Z

This workflow status is outdated as a new workflow run has been triggered.

🚫 This workflow run has been cancelled.

Failing Jobs - Building `5b41d8a`

⚠️ Artifacts of the workflow run were not available thus the report misses some details.

Status	Name	Test failures	Logs	Raw logs
⌛	Initial JDK 11 Build	⚠️ Check →	Logs	Raw logs
⌛	Attach pull request number	⚠️ Check →	Logs	Raw logs
⌛	CI Sanity Check	⚠️ Check →	Logs	Raw logs

iocanel · 2021-07-14T08:48:21Z

@gsmet: I think that all requested changes have been applied and related ci failures have been addressed. Still see some jobs failing, but they do seem irrelevant with the pr.

iocanel · 2021-09-03T13:21:52Z

@gsmet: howdy! is there anything else you need me to add?

gsmet · 2022-01-12T16:53:06Z

Sorry, my time was very limited until now and I never prioritized this one. I'm going to assign it to me and have a look in the next few days.

Thanks for the patience.

gsmet · 2022-07-06T11:21:42Z

Just a heads up that I'm actively working on this one and making significant changes (also setting up the appropriate Quay infrastructure for testing). I'll ping you when I have a state that requires your attention.

rsvoboda · 2022-07-18T16:05:05Z

oc cluster up (OpenShift 3.11) is used because CRC (OpenShift 4.x) has too high memory requirements to be a good fit for GH Actions runner?

gsmet · 2022-07-19T07:36:13Z

@iocanel I can see the version is fixed but I have a new failure in the Kubernetes tests:

Error: [ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:999-SNAPSHOT:build (default) on project kubernetes-docker-build-and-deploy-statefulset: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
Error: [error]: Build step io.quarkus.container.image.docker.deployment.DockerProcessor#dockerBuildFromJar threw an exception: java.lang.RuntimeException: Execution of 'docker build -f /home/runner/work/quarkus/quarkus/integration-tests/kubernetes/maven-invoker-way/target/it/kubernetes-docker-build-and-deploy-statefulset/src/main/docker/Dockerfile.jvm -t quay.io/quarkustesting/kubernetes-docker-build-and-deploy-statefulset:83ceab46cbdcff846dccb9e72bda431f75406baa /home/runner/work/quarkus/quarkus/integration-tests/kubernetes/maven-invoker-way/target/it/kubernetes-docker-build-and-deploy-statefulset' failed. See docker output for more details

Caused by:

[INFO] [INFO] [io.quarkus.container.image.docker.deployment.DockerProcessor] COPY failed: no source files were specified

And we still have the deployment issue I reported earlier in the OpenShift test:

Error: [error]: Build step io.quarkus.container.image.docker.deployment.DockerProcessor#dockerBuildFromJar threw an exception: java.lang.RuntimeException: Execution of 'docker push image-registry.openshift-image-registry.svc:5000/quarkustesting/openshift-docker-build-and-deploy:83ceab46cbdcff846dccb9e72bda431f75406baa' failed. See docker output for more details

gsmet

And now it's all good! Woot!

quarkus-bot · 2022-07-21T17:58:59Z

Failing Jobs - Building `432ae34`

Status	Name	Step	Failures	Logs	Raw logs
✖	Gradle Tests - JDK 11	`Build`	Failures	Logs	Raw logs
✔️	Gradle Tests - JDK 11 Windows
✖	JVM Tests - JDK 11	`Build`	Failures	Logs	Raw logs
✖	JVM Tests - JDK 17	`Build`	Failures	Logs	Raw logs
✖	JVM Tests - JDK 18	`Build`	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ Gradle Tests - JDK 11 #

- Failing: integration-tests/gradle

📦 integration-tests/gradle

✖ io.quarkus.gradle.devmode.CompositeBuildWithDependenciesDevModeTest.main line 24 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

✖ io.quarkus.gradle.devmode.MultiModuleNamedInjectionDevModeTest.main line 21 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

✖ io.quarkus.gradle.devmode.MultiSourceProjectDevModeTest.main line 22 - More details - Source on GitHub

org.awaitility.core.ConditionTimeoutException: Condition with lambda expression in io.quarkus.test.devmode.util.DevModeTestUtils that uses java.util.function.Supplier, java.util.function.Supplierjava.util.concurrent.atomic.AtomicReference, java.util.concurrent.atomic.AtomicReferencejava.lang.String, java.lang.Stringboolean was not fulfilled within 1 minutes.
	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
	at org.awaitility.core.CallableCondition.await(CallableCondition.java:78)

⚙️ JVM Tests - JDK 11 #

- Failing: extensions/resteasy-reactive/quarkus-resteasy-reactive/deployment 
! Skipped: extensions/avro/deployment extensions/grpc/deployment extensions/hibernate-reactive/deployment and 106 more

📦 extensions/resteasy-reactive/quarkus-resteasy-reactive/deployment

✖ io.quarkus.resteasy.reactive.server.test.multipart.TooLargeFormAttributeMultipartFormInputTest.test - More details - Source on GitHub

java.util.concurrent.RejectedExecutionException
	at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
	at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2076)

⚙️ JVM Tests - JDK 17 #

- Failing: integration-tests/elasticsearch-rest-client

📦 integration-tests/elasticsearch-rest-client

✖ Failed to execute goal io.fabric8:docker-maven-plugin:0.40.1:start (docker-start) on project quarkus-integration-test-elasticsearch-rest-client: I/O Error

⚙️ JVM Tests - JDK 18 #

- Failing: extensions/resteasy-reactive/quarkus-resteasy-reactive/deployment 
! Skipped: extensions/avro/deployment extensions/grpc/deployment extensions/hibernate-reactive/deployment and 106 more

📦 extensions/resteasy-reactive/quarkus-resteasy-reactive/deployment

✖ io.quarkus.resteasy.reactive.server.test.multipart.TooLargeFormAttributeMultipartFormInputTest.test - More details - Source on GitHub

java.util.concurrent.RejectedExecutionException
	at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
	at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2076)

iocanel added the area/kubernetes label Jun 3, 2021

quarkus-bot bot added the area/infra-automation anything related to CI, bots, etc. that are used to automated our infrastructure label Jun 3, 2021

iocanel force-pushed the k8s-itests branch from 6ece4ee to 716c247 Compare June 3, 2021 15:51

maxandersen reviewed Jun 4, 2021

View reviewed changes

.github/workflows/ci-kuberenetes-itests.yml Outdated Show resolved Hide resolved

maxandersen reviewed Jun 4, 2021

View reviewed changes

iocanel force-pushed the k8s-itests branch from 2109405 to 91af977 Compare June 9, 2021 09:21

iocanel changed the title ~~[wip] Introduce ci workflow for kubernetes/openshift end to end testing~~ Introduce ci workflow for kubernetes/openshift end to end testing Jun 9, 2021

iocanel force-pushed the k8s-itests branch from 91af977 to e17f07c Compare July 12, 2021 11:46

gsmet requested changes Jul 13, 2021

View reviewed changes

.github/workflows/ci-kubernetes-itests.yml Outdated Show resolved Hide resolved

.github/workflows/ci-kubernetes-itests.yml Outdated Show resolved Hide resolved

...ts/kubernetes/maven-invoker-way/src/it/kubernetes-docker-build-and-deploy/invoker.properties Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

gsmet self-assigned this Jan 12, 2022

rsvoboda mentioned this pull request May 16, 2022

Dockerfile.jvm build fails with kaniko: user: unknown user 1001 #25499

Closed

gsmet force-pushed the k8s-itests branch from 0d329fe to 9efff71 Compare July 5, 2022 13:32

This comment has been minimized.

Sign in to view

gsmet force-pushed the k8s-itests branch from 9efff71 to f1565c1 Compare July 13, 2022 13:15

This comment has been minimized.

Sign in to view

iocanel force-pushed the k8s-itests branch from c470877 to 83ceab4 Compare July 15, 2022 16:00

iocanel and others added 12 commits July 21, 2022 15:07

test: ci workflow for k8s related pull requests

b5b0e1d

fix: apply feedback

8b83312

fix: e2e tests are invoked explicitly using the e2e profiles

864aa6c

fix: k8s ci is triggered even if label is added on pr creation

53c3ab6

Fine tune Kubernetes CI workflow

cd7d456

Enable reporting for Kubernetes and OpenShift jobs

b947a25

Restore local testing for OpenShift and Kubernetes

c506f2e

Only run the scheduled workflow in the quarkus repository

749e5f6

fix: workaround failure when setting QUARKUS_CONTAINER_IMAGE

3610e3d

fix: outdated dockerfile for kubernetes-docker-b-and-d-statefulset

312b5c2

chore: remove refs to internal openshift registry

692ec20

Remove container-image.registry property: it's useless now

432ae34

gsmet force-pushed the k8s-itests branch from f08c403 to 432ae34 Compare July 21, 2022 13:07

gsmet approved these changes Jul 21, 2022

View reviewed changes

gsmet merged commit 70fd234 into quarkusio:main Jul 25, 2022

quarkus-bot bot added this to the 2.12 - main milestone Jul 25, 2022

Introduce ci workflow for kubernetes/openshift end to end testing #17674

Introduce ci workflow for kubernetes/openshift end to end testing #17674

Conversation

iocanel commented Jun 3, 2021

quarkus-bot bot commented Jun 3, 2021

quarkus-bot bot commented Jun 3, 2021 • edited

Failing Jobs - Building 6ece4ee

quarkus-bot bot commented Jun 3, 2021 • edited

Failing Jobs - Building 716c247

quarkus-bot bot commented Jun 3, 2021 • edited

Failing Jobs - Building 81cb75c

quarkus-bot bot commented Jun 3, 2021 • edited

Failing Jobs - Building 81cb75c

quarkus-bot bot commented Jun 3, 2021 • edited

Failing Jobs - Building 81cb75c

quarkus-bot bot commented Jun 3, 2021 • edited

Failing Jobs - Building 13a967b

quarkus-bot bot commented Jun 3, 2021 • edited

Failing Jobs - Building 87c4f91

quarkus-bot bot commented Jun 4, 2021 • edited

Failing Jobs - Building 562b23a

quarkus-bot bot commented Jun 4, 2021 • edited

Failing Jobs - Building d886021

quarkus-bot bot commented Jun 4, 2021 • edited

Failing Jobs - Building 28d8ba5

maxandersen Jun 4, 2021

Choose a reason for hiding this comment

iocanel Jun 4, 2021

Choose a reason for hiding this comment

maxandersen Jun 4, 2021

Choose a reason for hiding this comment

iocanel Jun 4, 2021

Choose a reason for hiding this comment

iocanel Jun 4, 2021

Choose a reason for hiding this comment

iocanel commented Jun 10, 2021

iocanel commented Jun 22, 2021

maxandersen commented Jun 22, 2021

iocanel commented Jul 12, 2021

quarkus-bot bot commented Jul 12, 2021 • edited

Failing Jobs - Building e17f07c

gsmet left a comment

Choose a reason for hiding this comment

quarkus-bot bot commented Jul 13, 2021 • edited

Failing Jobs - Building 5b41d8a

This comment has been minimized.

iocanel commented Jul 14, 2021

iocanel commented Sep 3, 2021

gsmet commented Jan 12, 2022

This comment has been minimized.

gsmet commented Jul 6, 2022

This comment has been minimized.

rsvoboda commented Jul 18, 2022

gsmet commented Jul 19, 2022

gsmet left a comment

Choose a reason for hiding this comment

quarkus-bot bot commented Jul 21, 2022

Failing Jobs - Building 432ae34

Failures

⚙️ Gradle Tests - JDK 11 #

📦 integration-tests/gradle

⚙️ JVM Tests - JDK 11 #

📦 extensions/resteasy-reactive/quarkus-resteasy-reactive/deployment

⚙️ JVM Tests - JDK 17 #

📦 integration-tests/elasticsearch-rest-client

⚙️ JVM Tests - JDK 18 #

📦 extensions/resteasy-reactive/quarkus-resteasy-reactive/deployment

quarkus-bot bot commented Jun 3, 2021 •

edited

Failing Jobs - Building `6ece4ee`

quarkus-bot bot commented Jun 3, 2021 •

edited

Failing Jobs - Building `716c247`

quarkus-bot bot commented Jun 3, 2021 •

edited

Failing Jobs - Building `81cb75c`

quarkus-bot bot commented Jun 3, 2021 •

edited

Failing Jobs - Building `81cb75c`

quarkus-bot bot commented Jun 3, 2021 •

edited

Failing Jobs - Building `81cb75c`

quarkus-bot bot commented Jun 3, 2021 •

edited

Failing Jobs - Building `13a967b`

quarkus-bot bot commented Jun 3, 2021 •

edited

Failing Jobs - Building `87c4f91`

quarkus-bot bot commented Jun 4, 2021 •

edited

Failing Jobs - Building `562b23a`

quarkus-bot bot commented Jun 4, 2021 •

edited

Failing Jobs - Building `d886021`

quarkus-bot bot commented Jun 4, 2021 •

edited

Failing Jobs - Building `28d8ba5`

quarkus-bot bot commented Jul 12, 2021 •

edited

Failing Jobs - Building `e17f07c`

quarkus-bot bot commented Jul 13, 2021 •

edited

Failing Jobs - Building `5b41d8a`

Failing Jobs - Building `432ae34`