Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci-amd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -934,6 +934,8 @@ jobs:
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
kubernetes-combos: ${{ needs.build-info.outputs.kubernetes-combos }}
java-sdk-version: ${{ needs.build-info.outputs.java-sdk-version }}
lang-sdk-kubernetes-combo: >-
${{ needs.build-info.outputs.default-python-version }}-${{ needs.build-info.outputs.default-kubernetes-version }}
if: >
( needs.build-info.outputs.run-kubernetes-tests == 'true' ||
needs.build-info.outputs.run-helm-tests == 'true')
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/ci-arm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -927,6 +927,8 @@ jobs:
debug-resources: ${{ needs.build-info.outputs.debug-resources }}
kubernetes-combos: ${{ needs.build-info.outputs.kubernetes-combos }}
java-sdk-version: ${{ needs.build-info.outputs.java-sdk-version }}
lang-sdk-kubernetes-combo: >-
${{ needs.build-info.outputs.default-python-version }}-${{ needs.build-info.outputs.default-kubernetes-version }}
if: >
( needs.build-info.outputs.run-kubernetes-tests == 'true' ||
needs.build-info.outputs.run-helm-tests == 'true')
Expand Down
133 changes: 102 additions & 31 deletions .github/workflows/k8s-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ on: # yamllint disable-line rule:truthy
description: "JDK version used to build the lang-SDK Java artifacts natively in CI"
required: true
type: string
lang-sdk-kubernetes-combo:
description: "Single Python-Kubernetes combo (<python>-<k8s-version>) for the dedicated lang-SDK job"
required: true
type: string
permissions:
contents: read
jobs:
Expand Down Expand Up @@ -100,75 +104,142 @@ jobs:
make-mnt-writeable-and-cleanup: true
id: breeze
# preparing k8s environment with uv takes < 15 seconds with `uv` - there is no point in caching it.
# Provision the lang-SDK Go/Java toolchains on the host only for the single variant that runs the
# lang-SDK test (KubernetesExecutor, standard-naming off). LANG_SDK_NATIVE_TOOLCHAIN=true then makes
# breeze build the artifacts natively instead of in throwaway toolchain containers, skipping the
# image pulls and reusing the caches restored below. The module/build and Gradle caches are keyed
# The lang-SDK coordinator system test is intentionally NOT run in this matrix. It has its own
# dedicated single-combo job (tests-kubernetes-lang-sdk below), so it no longer adds the ~6 minutes
# of Go/Java build + coordinator run to each of these system-test jobs.
- name: "\
Run complete K8S tests ${{ matrix.executor }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}-\
${{env.KUBERNETES_VERSION}}-${{ matrix.use-standard-naming }}"
run: breeze k8s run-complete-tests --upgrade --no-copy-local-sources
env:
EXECUTOR: ${{ matrix.executor }}
USE_STANDARD_NAMING: ${{ matrix.use-standard-naming }}
VERBOSE: "false"
- name: "\
Print logs ${{ matrix.executor }}-${{ matrix.kubernetes-combo }}-\
${{ matrix.use-standard-naming }}"
run: |
for file in `find /tmp/kind_logs_*/ -type f` ; do
echo "::group::${file}"
cat $file
echo "::endgroup::"
done
if: failure() || cancelled() || inputs.include-success-outputs == 'true'
- name: "\
Upload KinD logs ${{ matrix.executor }}-${{ matrix.kubernetes-combo }}-\
${{ matrix.use-standard-naming }}"
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: "\
kind-logs-${{ matrix.kubernetes-combo }}-${{ matrix.executor }}-\
${{ matrix.use-standard-naming }}"
path: /tmp/kind_logs_*
retention-days: '7'
if: failure() || cancelled() || inputs.include-success-outputs == 'true'
- name: "Delete clusters just in case they are left"
run: breeze k8s delete-cluster --all
if: always()

# The Multi-Lang KubernetesExecutor coordinator system test builds a Go bundle and a Java jar and runs
# them as workers under KubernetesExecutor. It is executor/naming/version-independent, so instead of
# bolting it onto all six ``KubernetesExecutor``/``use-standard-naming == false`` system-test jobs above
# (which ran it redundantly and added ~6 minutes each), it runs once here on a single default combo and
# executes only the lang-SDK test (``-k test_lang_sdk_combined_dag_succeeds``) - not the full suite.
tests-kubernetes-lang-sdk:
timeout-minutes: 60
name: "K8S Lang-SDK:${{ inputs.lang-sdk-kubernetes-combo }}"
runs-on: ${{ fromJSON(inputs.runners) }}
env:
DEBUG_RESOURCES: ${{ inputs.debug-resources }}
INCLUDE_SUCCESS_OUTPUTS: ${{ inputs.include-success-outputs }}
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
VERBOSE: "true"
steps:
- name: "Cleanup repo"
shell: bash
run: sudo rm -rf ${GITHUB_WORKSPACE}/*
- name: "Prepare PYTHON_MAJOR_MINOR_VERSION and KUBERNETES_VERSION"
id: prepare-versions
env:
KUBERNETES_COMBO: ${{ inputs.lang-sdk-kubernetes-combo }}
run: |
echo "PYTHON_MAJOR_MINOR_VERSION=${KUBERNETES_COMBO}" | sed 's/-.*//' >> $GITHUB_ENV
echo "KUBERNETES_VERSION=${KUBERNETES_COMBO}" | sed 's/=[^-]*-/=/' >> $GITHUB_ENV
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
# env.PYTHON_MAJOR_MINOR_VERSION, env.KUBERNETES_VERSION are set in the previous
# step id: prepare-versions
- name: "Prepare breeze & PROD image: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}"
uses: ./.github/actions/prepare_breeze_and_image
with:
platform: ${{ inputs.platform }}
image-type: "prod"
python: ${{ env.PYTHON_MAJOR_MINOR_VERSION }}
use-uv: ${{ inputs.use-uv }}
make-mnt-writeable-and-cleanup: true
id: breeze
# Provision the lang-SDK Go/Java toolchains on the host so breeze builds the artifacts natively
# (LANG_SDK_NATIVE_TOOLCHAIN=true) instead of in throwaway toolchain containers, skipping the image
# pulls and reusing the caches restored below. The module/build and Gradle caches are keyed
# explicitly (rather than via the setup-* built-in caching) so the key carries an ``-vN-`` salt:
# bump it to force-invalidate a poisoned cache without waiting for a dependency change to rotate it.
# keep go-version in sync with go-sdk/go.mod and the sibling Go SDK job in ci-amd.yml.
- name: "Setup Go for lang-SDK build"
if: matrix.executor == 'KubernetesExecutor' && matrix.use-standard-naming == false
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
with:
go-version: 1.25
cache: false
- name: "Cache lang-SDK Go module + build cache"
if: matrix.executor == 'KubernetesExecutor' && matrix.use-standard-naming == false
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/.cache/go-build
~/go/pkg/mod
key: lang-sdk-go-v1-${{ runner.os }}-${{ hashFiles('kubernetes-tests/lang_sdk/go_example/go.sum') }}
key: lang-sdk-go-v1-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('kubernetes-tests/lang_sdk/go_example/go.sum') }} # yamllint disable-line rule:line-length
restore-keys: |
lang-sdk-go-v1-${{ runner.os }}-
lang-sdk-go-v1-${{ runner.os }}-${{ runner.arch }}-
- name: "Setup Java for lang-SDK build"
if: matrix.executor == 'KubernetesExecutor' && matrix.use-standard-naming == false
uses: actions/setup-java@1bcf9fb12cf4aa7d266a90ae39939e61372fe520 # v5.4.0
with:
distribution: 'temurin'
java-version: ${{ inputs.java-sdk-version }}
- name: "Cache lang-SDK Gradle distribution + dependencies"
if: matrix.executor == 'KubernetesExecutor' && matrix.use-standard-naming == false
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: lang-sdk-gradle-v1-${{ runner.os }}-${{ hashFiles('java-sdk/**/*.gradle*', 'java-sdk/**/gradle-wrapper.properties', 'kubernetes-tests/lang_sdk/java_example/**/*.gradle*') }} # yamllint disable-line rule:line-length
key: lang-sdk-gradle-v1-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('java-sdk/**/*.gradle*', 'java-sdk/**/gradle-wrapper.properties', 'kubernetes-tests/lang_sdk/java_example/**/*.gradle*') }} # yamllint disable-line rule:line-length
restore-keys: |
lang-sdk-gradle-v1-${{ runner.os }}-
- name: "\
Run complete K8S tests ${{ matrix.executor }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}-\
${{env.KUBERNETES_VERSION}}-${{ matrix.use-standard-naming }}"
run: breeze k8s run-complete-tests --upgrade --no-copy-local-sources
lang-sdk-gradle-v1-${{ runner.os }}-${{ runner.arch }}-
# No --upgrade here: the redeploy/upgrade path is already exercised across the main matrix above;
# this job only needs a single deploy to validate the Multi-Lang coordinator, saving ~1.5 minutes.
- name: "Run lang-SDK K8S test ${{ inputs.lang-sdk-kubernetes-combo }}"
run: >-
breeze k8s run-complete-tests --no-copy-local-sources
-- -k test_lang_sdk_combined_dag_succeeds
env:
EXECUTOR: ${{ matrix.executor }}
USE_STANDARD_NAMING: ${{ matrix.use-standard-naming }}
# Provision + run the lang-SDK coordinator system test in a single variant only
# (KubernetesExecutor, standard-naming off) to keep it off the other five k8s jobs.
RUN_LANG_SDK_K8S_TESTS: ${{ (matrix.executor == 'KubernetesExecutor' && matrix.use-standard-naming == false) && 'true' || 'false' }} # yamllint disable-line rule:line-length
LANG_SDK_NATIVE_TOOLCHAIN: ${{ (matrix.executor == 'KubernetesExecutor' && matrix.use-standard-naming == false) && 'true' || 'false' }} # yamllint disable-line rule:line-length
EXECUTOR: "KubernetesExecutor"
USE_STANDARD_NAMING: "false"
RUN_LANG_SDK_K8S_TESTS: "true"
LANG_SDK_NATIVE_TOOLCHAIN: "true"
VERBOSE: "false"
- name: "\
Print logs ${{ matrix.executor }}-${{ matrix.kubernetes-combo }}-\
${{ matrix.use-standard-naming }}"
- name: "Print logs lang-sdk-${{ inputs.lang-sdk-kubernetes-combo }}"
run: |
for file in `find /tmp/kind_logs_*/ -type f` ; do
echo "::group::${file}"
cat $file
echo "::endgroup::"
done
if: failure() || cancelled() || inputs.include-success-outputs == 'true'
- name: "\
Upload KinD logs ${{ matrix.executor }}-${{ matrix.kubernetes-combo }}-\
${{ matrix.use-standard-naming }}"
- name: "Upload KinD logs lang-sdk-${{ inputs.lang-sdk-kubernetes-combo }}"
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: "\
kind-logs-${{ matrix.kubernetes-combo }}-${{ matrix.executor }}-\
${{ matrix.use-standard-naming }}"
name: "kind-logs-lang-sdk-${{ inputs.lang-sdk-kubernetes-combo }}"
path: /tmp/kind_logs_*
retention-days: '7'
if: failure() || cancelled() || inputs.include-success-outputs == 'true'
Expand Down
34 changes: 20 additions & 14 deletions kubernetes-tests/lang_sdk/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,19 +95,25 @@ RUN_LANG_SDK_K8S_TESTS=true breeze k8s tests --executor KubernetesExecutor \
-- -k test_lang_sdk_combined_dag_succeeds
```

In CI (and for a one-shot local run) steps 2-3 are folded into the standard k8s job via
`breeze k8s run-complete-tests --lang-sdk-test`: it provisions the lang-SDK env after the base
deploy, then runs the test. The `k8s-tests.yml` workflow enables it (`RUN_LANG_SDK_K8S_TESTS=true`,
which `--lang-sdk-test` reads) for a single variant only -- KubernetesExecutor with standard-naming
off -- so the other five k8s jobs skip the test. The provisioning builds (Go bundle, Java jar, Java
worker image) and the localstack deploy run in parallel.
In CI (and for a one-shot local run) steps 2-3 are folded into a single `run-complete-tests` call via
`breeze k8s run-complete-tests --lang-sdk-test`: it provisions the lang-SDK env after the base deploy,
then runs the test. Rather than bolting this onto the regular k8s system-test matrix (which ran it
redundantly on all six `KubernetesExecutor` / standard-naming-off jobs and added ~6 minutes each), the
`k8s-tests.yml` workflow runs it in a **dedicated `tests-kubernetes-lang-sdk` job** on a single default
Python-Kubernetes combo (the `lang-sdk-kubernetes-combo` input, wired from the `default-python-version`
and `default-kubernetes-version` build-info outputs). That job sets `RUN_LANG_SDK_K8S_TESTS=true`
(which `--lang-sdk-test` reads) and runs only the lang-SDK test (`-k
test_lang_sdk_combined_dag_succeeds`), not the full suite; the regular system-test matrix no longer runs
it at all. The provisioning builds (Go bundle, Java jar, Java worker image) and the localstack deploy
run in parallel.

By default the Go bundle and Java jar are built inside ephemeral toolchain containers so a dev host
needs neither Go nor a JDK installed. In CI that variant sets `LANG_SDK_NATIVE_TOOLCHAIN=true`, which
makes breeze build both artifacts with the host `go` / `./gradlew` instead: the workflow installs the
toolchains via `actions/setup-go` and `actions/setup-java` and restores the Go module/build cache and
the Gradle distribution + dependency cache with `actions/cache`, so the build skips the per-run
toolchain-image pulls and cold dependency downloads. The cache keys carry a `-v1-` salt (see
`lang-sdk-go-v1-` / `lang-sdk-gradle-v1-` in `k8s-tests.yml`) — bump it to force-invalidate a poisoned
cache. The JDK version comes from the `java-sdk-version` build-info output (the `JAVA_SDK_VERSION`
breeze constant).
needs neither Go nor a JDK installed. In CI the dedicated job sets `LANG_SDK_NATIVE_TOOLCHAIN=true`,
which makes breeze build both artifacts with the host `go` / `./gradlew` instead: the workflow installs
the toolchains via `actions/setup-go` and `actions/setup-java` and restores the Go module/build cache
and the Gradle distribution + dependency cache with `actions/cache`, so the build skips the per-run
toolchain-image pulls and cold dependency downloads. The cache keys carry a `-v1-` salt and a
`runner.arch` segment (see `lang-sdk-go-v1-` / `lang-sdk-gradle-v1-` in `k8s-tests.yml`) — bump the salt
to force-invalidate a poisoned cache; the arch segment keeps the amd64 and arm64 caches separate. The
JDK version comes from the `java-sdk-version` build-info output (the `JAVA_SDK_VERSION` breeze
constant).
Loading