reusable smoke test for BYOC and self-hosted satellites

earthly · May 31, 2024 · b1f05e8 · b1f05e8
1 parent 335389e
commit b1f05e8
Show file tree

Hide file tree

Showing 120 changed files with 2,472 additions and 1,951 deletions.
diff --git a/.github/actions/stage2-setup/action.yml b/.github/actions/stage2-setup/action.yml
@@ -18,6 +18,10 @@ inputs:
     required: false
     type: boolean
     default: false
+  USE_NEXT:
+    required: false
+    type: boolean
+    default: false
   SATELLITE_NAME:
     required: false
     type: string
@@ -67,16 +71,26 @@ runs:
       shell: bash
     - run: |-
         echo "Extracting earthly binary from stage1 of build"
-        BUILDKITD_IMAGE=docker.io/earthly/buildkitd-staging TAG=${GITHUB_SHA}-ubuntu-latest-${{inputs.BINARY}} ${{inputs.SUDO}} ./earthly upgrade
+        export TAG=${GITHUB_SHA}-ubuntu-latest-${{inputs.BINARY}}
+        if [ "${{inputs.USE_NEXT}}" = "true" ]; then export TAG="$TAG-ticktock"; fi
+        BUILDKITD_IMAGE=docker.io/earthly/buildkitd-staging ${{inputs.SUDO}} ./earthly upgrade
         ${{inputs.SUDO}} chown -R $USER ~/.earthly # restore non-sudo user ownership
         test -n "${{inputs.BUILT_EARTHLY_PATH}}" || (echo "BUILT_EARTHLY_PATH is empty" && exit 1)
         mkdir -p $(dirname "${{inputs.BUILT_EARTHLY_PATH}}")
-        ${{inputs.SUDO}} mv ${HOME}/.earthly/earthly-${GITHUB_SHA}-ubuntu-latest-${{inputs.BINARY}} "${{inputs.BUILT_EARTHLY_PATH}}"
+        ${{inputs.SUDO}} ls "${HOME}/.earthly/"
+        ${{inputs.SUDO}} mv "${HOME}/.earthly/earthly-$TAG" "${{inputs.BUILT_EARTHLY_PATH}}"
         echo "extracted ${{inputs.BUILT_EARTHLY_PATH}}"
       shell: bash
+    - if: ${{ inputs.USE_NEXT == 'true' }}
+      run: |-
+        export expected_buildkit_client_sha="$(cat earthly-next | head -c 12)"
+        test -n "$expected_buildkit_client_sha" || ( echo "expected_buildkit_client_sha is empty" && exit 1)
+        (strings ${{inputs.BUILT_EARTHLY_PATH}} | grep "$expected_buildkit_client_sha" ) || ( echo "expected to find $expected_buildkit_client_sha in earthly binary" && exit 1)
+        echo "correctly found $expected_buildkit_client_sha in earthly binary; this confirms earthly-next was used"
+      shell: bash
     - run: |-
         echo "Setting up mirror credentials in .arg and .secret"
-        export earthly=${{inputs.BUILT_EARTHLY_PATH}}
+        export earthly="${{inputs.BUILT_EARTHLY_PATH}}"
         # setup secrets
         echo "DOCKERHUB_MIRROR_USER=$($earthly secret --org earthly-technologies --project core get -n dockerhub-mirror/user || kill $$)" > .secret
         echo "DOCKERHUB_MIRROR_PASS=$($earthly secret --org earthly-technologies --project core get -n dockerhub-mirror/pass || kill $$)" >> .secret

diff --git a/.github/workflows/README.md b/.github/workflows/README.md
@@ -1,6 +1,6 @@
-# Github Actions
+# GitHub Actions
 
-Documentation for this repo Github actions configuration
+Documentation for this repo GitHub actions configuration
 
 ## Skipping PR Workflows (DISABLED! SEE NOTE BELOW)
 
@@ -12,8 +12,8 @@ Some PR workflows (workflows triggered by PRs) in this repo might take substanti
 and it is not always necessary to run them all, especially in cases where the affected files are documents or any other file that don't affect our workflows or tests.  
 In addition, some of our tests use satellites and limiting the number of (concurrent) builds might help with their performance.
 
-### Github Builtin Support
-Github supports defining filters on workflows so that they won't get triggered, and there is a builtin filter that would do so according to affected files - [paths-ignore](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore).  
+### GitHub Builtin Support
+GitHub supports defining filters on workflows so that they won't get triggered, and there is a builtin filter that would do so according to affected files - [paths-ignore](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore).  
 The shortcoming of this filter is that when the workflow is marked as a `required check` and does not get triggered because of this filter,  
 the workflow will be stuck in `pending` and won't allow the PR to be merged.  
 Thus, using this filter is a good approach when handling workflows that are not required checks, but it's problematic otherwise.
@@ -22,18 +22,18 @@ Thus, using this filter is a good approach when handling workflows that are not
 An alternative to `paths-ignore` is examining the affected files at the `job` level instead.  
 Jobs allow you to set conditions on whether they should execute or not.  
 The advantage of using conditions on the job level is that if a job does not run due to a condition evaluation to false,
-Github will still mark the workflow as passing/successful instead of pending,  
+GitHub will still mark the workflow as passing/successful instead of pending,  
 and will not block the PR even if the workflow is marked as a required check ([reference](https://docs.github.com/en/actions/using-jobs/using-conditions-to-control-job-execution#overview)).  
 
 To examine the files, [dorny/paths-filter](https://github.com/dorny/paths-filter) is used.
-Similarly to Github's `paths-ignore`, `dorny/paths-filter` supports setting rules that match against the affected files, which evaluates to a boolean value.
+Similarly to GitHub's `paths-ignore`, `dorny/paths-filter` supports setting rules that match against the affected files, which evaluates to a boolean value.
 We can then use that boolean value to determine whether subsequent jobs in the workflow should execute or not. 
 
 ### Complexity due to reusable workflows
 
 #### Background
 A problem arises when the job we wish to conditionally run uses a reusable workflow.
-The problem is that Github treats such configuration as a parent-child relationship,
+The problem is that GitHub treats such configuration as a parent-child relationship,
 and any job that is skipped, by default will also cause its children jobs to skip as well.
 In addition, when we set `required checks`, we cannot properly select the parent job, only the child job.  
 In this scenario, if a child job is skipped as a result of skipping its parent, the child job will get stuck in `pending` and the PR will be blocked from merging (just like in the original problem).

diff --git a/.github/workflows/build-earthly.yml b/.github/workflows/build-earthly.yml
@@ -19,11 +19,15 @@ on:
         required: false
         type: boolean
         default: false
+      USE_NEXT:
+        required: false
+        type: boolean
+        default: false
 
 jobs:
 
   build-earthly:
-    name: build (and push) earthly using ${{inputs.BINARY}}
+    name: build (and push) earthly using ${{inputs.BINARY}} and earthly-next=${{inputs.USE_NEXT}}
     if: ${{!inputs.SKIP_JOB}}
     runs-on: ${{inputs.RUNS_ON}}
     env:
@@ -71,6 +75,9 @@ jobs:
       - name: Podman Login
         run: ${{inputs.SUDO}} ${{inputs.BINARY}} login docker.io --username "${{ secrets.DOCKERHUB_USERNAME }}" --password "${{ secrets.DOCKERHUB_TOKEN }}"
         if: inputs.BINARY == 'podman'
+      - name: Update Buildkit to earthly-next
+        if: inputs.USE_NEXT
+        run: ${{inputs.SUDO}} $(which earthly) +update-buildkit --BUILDKIT_GIT_SHA=$(cat earthly-next)
       - name: Build latest earthly using released earthly
         run: ${{inputs.SUDO}} $(which earthly) --use-inline-cache +for-linux
       - name: Earthly bootstrap using latest earthly build
@@ -81,7 +88,10 @@ jobs:
           EARTHLY_VERSION_FLAG_OVERRIDES="$(tr -d '\n' < .earthly_version_flag_overrides)"
           echo "EARTHLY_VERSION_FLAG_OVERRIDES=$EARTHLY_VERSION_FLAG_OVERRIDES" >> "$GITHUB_ENV"
       - name: Build and push +ci-release using latest earthly build
-        run: ${{inputs.SUDO}} ./build/linux/amd64/earthly --ci --push +ci-release --TAG_SUFFIX="${{inputs.RUNS_ON}}-${{inputs.BINARY}}"
+        run: |-
+          export TAG_SUFFIX="${{inputs.RUNS_ON}}-${{inputs.BINARY}}"
+          if [ "${{inputs.USE_NEXT}}" = "true" ]; then export TAG_SUFFIX="$TAG_SUFFIX-ticktock"; fi
+          ${{inputs.SUDO}} ./build/linux/amd64/earthly --ci --push --build-arg TAG_SUFFIX +ci-release
       - name: Buildkit logs (runs on failure)
         run: ${{inputs.SUDO}} ${{inputs.BINARY}} logs earthly-buildkitd
         if: ${{ failure() }}
diff --git a/.github/workflows/ci-docker-satellites.yml b/.github/workflows/ci-docker-satellites.yml
@@ -73,3 +73,29 @@ jobs:
       SATELLITE_NAME: "core-test"
       EARTHLY_ORG: "earthly-technologies"
     secrets: inherit
+
+  satellites-byoc-smoke-test:
+    needs: build-earthly
+    uses: ./.github/workflows/reusable-satellite-smoke-test.yml
+    with:
+      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
+      RUNS_ON: "ubuntu-latest"
+      BINARY: "docker"
+      SUDO: ""
+      SATELLITE_NAME: "core-byoc"
+      EARTHLY_ORG: "earthly-technologies"
+      USE_VPN: true
+    secrets: inherit
+
+  satellites-self-hosted-smoke-test:
+      needs: build-earthly
+      uses: ./.github/workflows/reusable-satellite-smoke-test.yml
+      with:
+        BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
+        RUNS_ON: "ubuntu-latest"
+        BINARY: "docker"
+        SUDO: ""
+        SATELLITE_NAME: "self-hosted"
+        EARTHLY_ORG: "earthly-technologies"
+        USE_VPN: true
+      secrets: inherit
diff --git a/.github/workflows/ci-docker-ubuntu.yml b/.github/workflows/ci-docker-ubuntu.yml
@@ -95,34 +95,6 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
-  docker-tests-no-qemu-group1-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group1"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
-  docker-tests-no-qemu-group2-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group2"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
   docker-tests-no-qemu-group3:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -137,20 +109,6 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
-  docker-tests-no-qemu-group3-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group3"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
   docker-tests-no-qemu-group4:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -165,20 +123,6 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
-  docker-tests-no-qemu-group4-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group4"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
   docker-tests-no-qemu-group5:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -193,20 +137,6 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
-  docker-tests-no-qemu-group5-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group5"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
   docker-tests-no-qemu-group6:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -221,20 +151,6 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
-  docker-tests-no-qemu-group6-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group6"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
   docker-tests-no-qemu-group7:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -249,20 +165,6 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
-  docker-tests-no-qemu-group7-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group7"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
   docker-tests-no-qemu-group8:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -277,20 +179,6 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
-  docker-tests-no-qemu-group8-no-logbus:
-    needs: build-earthly
-    if: ${{ !failure() }}
-    uses: ./.github/workflows/reusable-test.yml
-    with:
-      TEST_TARGET: "+test-no-qemu-group8"
-      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
-      RUNS_ON: "ubuntu-latest"
-      BINARY: "docker"
-      SUDO: ""
-      EXTRA_ARGS: "--logstream=false --logstream-upload=false --auto-skip"
-      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
-    secrets: inherit
-
   docker-tests-no-qemu-group9:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -473,6 +361,20 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
+  docker-test-oidc-command:
+    if: ${{ !failure() && (github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository) }}
+    needs: build-earthly
+    uses: ./.github/workflows/reusable-test.yml
+    with:
+      TEST_TARGET: "./tests/oidc+test"
+      BUILT_EARTHLY_PATH: "./build/linux/amd64/earthly"
+      RUNS_ON: "ubuntu-latest"
+      BINARY: "docker"
+      SUDO: ""
+      SKIP_JOB: ${{ needs.build-earthly.result != 'success' }}
+      EXTRA_ARGS: "--auto-skip"
+    secrets: inherit
+
   docker-tests-qemu:
     needs: build-earthly
     if: ${{ !failure() }}
@@ -860,6 +762,41 @@ jobs:
       EXTRA_ARGS: "--auto-skip"
     secrets: inherit
 
+  backwards-compatability-test:
+    needs: build-earthly
+    name: Backwards Compatability
+    runs-on: ubuntu-latest
+    env:
+      FORCE_COLOR: 1
+      EARTHLY_TOKEN: "${{ secrets.EARTHLY_TOKEN }}"
+      EARTHLY_INSTALL_ID: "earthly-githubactions"
+      # Used in our github action as the token - TODO: look to change it into an input
+      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+      - name: Docker mirror login (Earthly Only)
+        run: docker login registry-1.docker.io.mirror.corp.earthly.dev --username "${{ secrets.DOCKERHUB_MIRROR_USERNAME }}" --password "${{ secrets.DOCKERHUB_MIRROR_PASSWORD }}"
+        if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
+      - name: Retrieve earthly from build-earthly job
+        run: |-
+          BUILDKITD_IMAGE=docker.io/earthly/buildkitd-staging TAG=${GITHUB_SHA}-ubuntu-latest-docker ./earthly upgrade
+          mkdir -p $(dirname "./build/earthly")
+          mv ${HOME}/.earthly/earthly-${GITHUB_SHA}-ubuntu-latest-docker ./build/earthly
+      - name: Configure Earthly to use mirror (Earthly Only)
+        run: |-
+          ./build/earthly config global.buildkit_additional_config "'[registry.\"docker.io\"]
+          mirrors = [\"registry-1.docker.io.mirror.corp.earthly.dev\"]'"
+        if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
+      - name: Download older version of earthly (v0.8.0)
+        run: |-
+          sudo wget https://github.com/earthly/earthly/releases/download/v0.8.0/earthly-linux-amd64 -O /usr/bin/earthly-v0.8.0
+          sudo chmod +x /usr/bin/earthly-v0.8.0
+          earthly-v0.8.0 --version
+      - name: execute backwards compatability script
+        run: earthly=./build/earthly scripts/tests/backwards-compatability.sh
+
   tutorial:
     needs: build-earthly
     name: Tutorial