Make setup-and-test configurable #38

geoand · 2021-01-28T15:43:30Z

The idea is to make it simple to allow an ecosystem CI run to test different Quarkus versions.

For example Kogito's GH Actions definition (at https://github.com/kiegroup/kogito-runtimes/blob/master/.github/workflows/quarkus-snapshot.yaml) could be changed to:

name: "Quarkus ecosystem CI"
on:
  watch:
    types: [started]

  # For this CI to work, ECOSYSTEM_CI_TOKEN needs to contain a GitHub with rights to close the Quarkus issue that the user/bot has opened,
  # while 'ECOSYSTEM_CI_REPO_PATH' needs to be set to the corresponding path in the 'quarkusio/quarkus-ecosystem-ci' repository

env:
  ECOSYSTEM_CI_REPO: quarkusio/quarkus-ecosystem-ci
  ECOSYSTEM_CI_REPO_FILE: context.yaml
  JAVA_VERSION: 11

  #########################
  # Repo specific setting #
  #########################

  ECOSYSTEM_CI_REPO_PATH:  kogito-quarkus

jobs:
  quarkus-master:
    name: "Build against latest Quarkus snapshot"
    runs-on: ubuntu-latest
    if: github.actor == 'quarkusbot'

    steps:
      - name: Install yq
        run: sudo add-apt-repository ppa:rmescandon/yq && sudo apt update && sudo apt install yq -y

      - name: Set up Java
        uses: actions/setup-java@v1
        with:
          java-version: ${{ env.JAVA_VERSION }}

      - name: Checkout repo
        uses: actions/checkout@v2
        with:
          path: current-repo
          ref: master

      - name: Checkout Ecosystem
        uses: actions/checkout@v2
        with:
          repository: ${{ env.ECOSYSTEM_CI_REPO }}
          ref: master
          path: ecosystem-ci

      - name: Test against Quarkus master
        run: ./ecosystem-ci/setup-and-test
        env:
          ECOSYSTEM_CI_TOKEN: ${{ secrets.ECOSYSTEM_CI_TOKEN }}

  quarkus-lts:
    name: "Build against latest Quarkus LTS"
    runs-on: ubuntu-latest
    if: github.actor == 'quarkusbot'

    steps:
      - name: Install yq
        run: sudo add-apt-repository ppa:rmescandon/yq && sudo apt update && sudo apt install yq -y

      - name: Set up Java
        uses: actions/setup-java@v1
        with:
          java-version: ${{ env.JAVA_VERSION }}

      - name: Checkout repo
        uses: actions/checkout@v2
        with:
          path: current-repo
          ref: master

      - name: Checkout Ecosystem
        uses: actions/checkout@v2
        with:
          repository: ${{ env.ECOSYSTEM_CI_REPO }}
          ref: master
          path: ecosystem-ci

      - name: Test against Quarkus master
        run: ./ecosystem-ci/setup-and-test
        env:
          ECOSYSTEM_CI_TOKEN: ${{ secrets.ECOSYSTEM_CI_TOKEN }}
          ISSUE_NUM=9999 #needs to be created
          QUARKUS_BRANCH=1.11

Basically this adds a second job to be run on the same trigger, but configured the setup-and-test script by overriding the issue number and quarkus branch name.
Setup could potentially be different, but I show the 2 jobs approach here since we want the test to run for both Quarkus branches whether or not either fails

geoand · 2021-01-28T16:10:08Z

cc @evacchi @gsmet

evacchi · 2021-01-28T16:11:20Z

cc @radtriste @danielezonca

evacchi · 2021-01-28T16:13:09Z

setup-and-test

-cd quarkus
-git checkout ${QUARKUS_SHA}
+# Checkout Quarkus
+if [[ -z "${QUARKUS_BRANCH}" ]]; then


@radtriste that looks like what you were suggesting

maxandersen · 2021-01-28T16:30:01Z

shouldn't the "overrides" come from the yaml driving the build ?

geoand · 2021-01-28T16:34:36Z

The problem is that we have one trigger for different builds. I suppose we could have second YAML file and simply set the env var to that file.
I just think it would be confusing what comes from where...

maxandersen · 2021-01-28T17:32:22Z

what i was thinking was to change context from:

timestamp: 2021-01-28_07-24-37
issues:
  latestCommit: 9142
  repo: quarkusio/quarkus
quarkus:
  version: 999-SNAPSHOT
  sha: 6cdd2078f1e99eddc4e739f28c7d7808ce8af12b

to something like:

timestamp: 2021-01-28_07-24-37
issues:
  latestCommit: 9142
  repo: quarkusio/quarkus
quarkus:
  version: 999-SNAPSHOT
  sha: 42323434234
alternatives:
	"latest": 
		timestamp: 2021-01-28_07-24-37
	  	issues:
	    	latestCommit: 9142
	    	repo: quarkusio/quarkus
	  	quarkus:
	    version: 999-SNAPSHOT
	    sha: 6cdd2078f1e99eddc4e739f28c7d7808ce8af12b
	"next.x":
	  timestamp: 2021-01-28_07-24-37
	  issues:
	    latestCommit: 9142
	    repo: quarkusio/quarkus
	  quarkus:
	    version: 1.11.1-SNAPSHOT
	    sha: 6cdd203435234094

then have that be what drives the build.

note, here i'm assuming we would also set version numbers on branches but if we didn't we would just need to ensure the builds does not share ~/.m2 repo to avoid overlapping 999-snapshot

geoand · 2021-01-28T17:42:02Z

That seems reasonable, but I really don't want to spend any more time than necessary on this (as of now) isolated case.
I would prefer getting something in for Kogito now and kicking the stone down the road as we haven't heard of this need from anyone else.

maxandersen · 2021-01-28T17:50:57Z

On last two synccups we had ask to be able to build ecosystem against other ébranches for ie. Vertx 4 - isn't this the same ?

maxandersen · 2021-01-28T17:54:13Z

And to be clear. Doing something now for kogito is fine. Just thought would be good to discuss how to fix the multi branch setup.

geoand · 2021-01-28T18:20:21Z

And to be clear. Doing something now for kogito is fine. Just thought would be good to discuss how to fix the multi branch setup.

Right 👍.

I'm definitely in favor of a proper solution when the need actually arises.
For now we have Kogito that really wants this feature, so I would prefer to get this in now and the work on the proper solution later or (or Guillaume could when he finds some time).

geoand · 2021-01-28T18:21:06Z

On last two synccups we had ask to be able to build ecosystem against other ébranches for ie. Vertx 4 - isn't this the same ?

I don't think the Vert.x branch thing is the same

geoand · 2021-01-29T07:11:53Z

I just pushed a second commit which is simplified version of what @maxandersen proposed.
With this version, Kogito could would now change its info.yaml in this repository to:

url: https://github.com/kiegroup/kogito-runtimes # Github repo of your extension
issues:
  repo: quarkusio/quarkus # this should be left as is when CI updates are going to be reported on the Quarkus repository
  latestCommit: 12486 # this is the number of the issue you created above
alternatives:
  lts:
    issue: 99999999 # some new issue number
    quakusBranch: 1.11

and update their pipeline definition (in its own repo) to something like:

name: "Quarkus ecosystem CI"
on:
  watch:
    types: [started]

  # For this CI to work, ECOSYSTEM_CI_TOKEN needs to contain a GitHub with rights to close the Quarkus issue that the user/bot has opened,
  # while 'ECOSYSTEM_CI_REPO_PATH' needs to be set to the corresponding path in the 'quarkusio/quarkus-ecosystem-ci' repository

env:
  ECOSYSTEM_CI_REPO: quarkusio/quarkus-ecosystem-ci
  ECOSYSTEM_CI_REPO_FILE: context.yaml
  JAVA_VERSION: 11

  #########################
  # Repo specific setting #
  #########################

  ECOSYSTEM_CI_REPO_PATH:  kogito-quarkus

jobs:
  quarkus-master:
    name: "Build against latest Quarkus snapshot"
    runs-on: ubuntu-latest
    if: github.actor == 'quarkusbot'

    steps:
      - name: Install yq
        run: sudo add-apt-repository ppa:rmescandon/yq && sudo apt update && sudo apt install yq -y

      - name: Set up Java
        uses: actions/setup-java@v1
        with:
          java-version: ${{ env.JAVA_VERSION }}

      - name: Checkout repo
        uses: actions/checkout@v2
        with:
          path: current-repo
          ref: master

      - name: Checkout Ecosystem
        uses: actions/checkout@v2
        with:
          repository: ${{ env.ECOSYSTEM_CI_REPO }}
          ref: master
          path: ecosystem-ci

      - name: Test against Quarkus master
        run: ./ecosystem-ci/setup-and-test
        env:
          ECOSYSTEM_CI_TOKEN: ${{ secrets.ECOSYSTEM_CI_TOKEN }}

  quarkus-lts:
    name: "Build against latest Quarkus LTS"
    runs-on: ubuntu-latest
    if: github.actor == 'quarkusbot'

    steps:
      - name: Install yq
        run: sudo add-apt-repository ppa:rmescandon/yq && sudo apt update && sudo apt install yq -y

      - name: Set up Java
        uses: actions/setup-java@v1
        with:
          java-version: ${{ env.JAVA_VERSION }}

      - name: Checkout repo
        uses: actions/checkout@v2
        with:
          path: current-repo
          ref: master

      - name: Checkout Ecosystem
        uses: actions/checkout@v2
        with:
          repository: ${{ env.ECOSYSTEM_CI_REPO }}
          ref: master
          path: ecosystem-ci

      - name: Test against Quarkus master
        run: ./ecosystem-ci/setup-and-test
        env:
          ECOSYSTEM_CI_TOKEN: ${{ secrets.ECOSYSTEM_CI_TOKEN }}
          ALTERNATIVE: lts

I think that this iteration is more than enough for the time being

radtriste · 2021-01-29T09:47:10Z

Hi @geoand,
Taking your latest example, should we update context.yaml instead of info.yaml
in the CI definition, we set ECOSYSTEM_CI_REPO_FILE: context.yaml

gsmet · 2021-01-29T09:54:50Z

Thanks @geoand . I think it's perfectly reasonable and I agree we shouldn't spend too much time on this.

geoand · 2021-01-29T09:55:23Z

@radtriste no, updating info.yaml is the way to go, as the context.yaml is automatically updated by ecosystem CI

radtriste · 2021-01-29T09:56:03Z

oh ok. good for me then :)

geoand · 2021-02-01T07:10:54Z

If there are not any objections, I'll go ahead and merge this

setup-and-test

geoand · 2021-02-02T10:33:49Z

Merging.

If all goes to hell tomorrow, we'll know what's to blame :)

Make setup-and-test configurable

804839b

geoand force-pushed the configurable-setup-and-test branch from 6e2a470 to 804839b Compare January 28, 2021 15:50

evacchi reviewed Jan 28, 2021

View reviewed changes

gsmet reviewed Feb 1, 2021

View reviewed changes

setup-and-test Outdated Show resolved Hide resolved

Allow specifying alternative configuration when running setup-and-test

bd9e8e6

geoand force-pushed the configurable-setup-and-test branch from 6c5a84b to bd9e8e6 Compare February 1, 2021 10:30

geoand merged commit 44620ca into master Feb 2, 2021

geoand deleted the configurable-setup-and-test branch February 2, 2021 10:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make setup-and-test configurable #38

Make setup-and-test configurable #38

geoand commented Jan 28, 2021 •

edited

geoand commented Jan 28, 2021

evacchi commented Jan 28, 2021

evacchi Jan 28, 2021

maxandersen commented Jan 28, 2021

geoand commented Jan 28, 2021

maxandersen commented Jan 28, 2021

geoand commented Jan 28, 2021

maxandersen commented Jan 28, 2021

maxandersen commented Jan 28, 2021

geoand commented Jan 28, 2021

geoand commented Jan 28, 2021

geoand commented Jan 29, 2021 •

edited

radtriste commented Jan 29, 2021

gsmet commented Jan 29, 2021

geoand commented Jan 29, 2021

radtriste commented Jan 29, 2021 •

edited

geoand commented Feb 1, 2021

geoand commented Feb 2, 2021

Make setup-and-test configurable #38

Make setup-and-test configurable #38

Conversation

geoand commented Jan 28, 2021 • edited

geoand commented Jan 28, 2021

evacchi commented Jan 28, 2021

evacchi Jan 28, 2021

Choose a reason for hiding this comment

maxandersen commented Jan 28, 2021

geoand commented Jan 28, 2021

maxandersen commented Jan 28, 2021

geoand commented Jan 28, 2021

maxandersen commented Jan 28, 2021

maxandersen commented Jan 28, 2021

geoand commented Jan 28, 2021

geoand commented Jan 28, 2021

geoand commented Jan 29, 2021 • edited

radtriste commented Jan 29, 2021

gsmet commented Jan 29, 2021

geoand commented Jan 29, 2021

radtriste commented Jan 29, 2021 • edited

geoand commented Feb 1, 2021

geoand commented Feb 2, 2021

geoand commented Jan 28, 2021 •

edited

geoand commented Jan 29, 2021 •

edited

radtriste commented Jan 29, 2021 •

edited