feat: Migrate TWA test-related workflows #627
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| name: TWA Frontend Tests | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - components/crud-web-apps/tensorboards/frontend/** | ||
| - releasing/version/VERSION | ||
| branches: | ||
| - main | ||
| - v*-branch | ||
| - notebooks-v1 | ||
|
|
||
| jobs: | ||
| frontend-format-linting-check: | ||
| name: Code format and lint | ||
| runs-on: ubuntu-22.04 | ||
| defaults: | ||
| run: | ||
| working-directory: components/crud-web-apps/tensorboards/frontend | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 16 | ||
| cache: 'npm' | ||
| cache-dependency-path: 'components/crud-web-apps/tensorboards/frontend/package-lock.json' | ||
|
|
||
| - name: Install dependencies | ||
| run: npm ci | ||
|
|
||
| - name: Check code format | ||
| run: npm run format:check | ||
|
|
||
| - name: Lint code | ||
| run: npm run lint-check | ||
|
|
||
| build-common-lib: | ||
| name: Build Kubeflow common library | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 16 | ||
| cache: 'npm' | ||
| cache-dependency-path: 'components/crud-web-apps/common/frontend/kubeflow-common-lib/package-lock.json' | ||
|
|
||
| - name: Install and build common library | ||
| run: | | ||
| cd components/crud-web-apps/common/frontend/kubeflow-common-lib | ||
| npm ci | ||
| npm run build | ||
|
|
||
| - name: Cache built common library | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: components/crud-web-apps/common/frontend/kubeflow-common-lib/dist | ||
| key: kubeflow-common-lib-${{ github.sha }} | ||
| restore-keys: | | ||
| kubeflow-common-lib- | ||
|
|
||
| frontend-unit-tests: | ||
| name: Unit tests | ||
| runs-on: ubuntu-22.04 | ||
| needs: build-common-lib | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 16 | ||
| cache: 'npm' | ||
| cache-dependency-path: 'components/crud-web-apps/tensorboards/frontend/package-lock.json' | ||
|
|
||
| - name: Restore built common library | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: components/crud-web-apps/common/frontend/kubeflow-common-lib/dist | ||
| key: kubeflow-common-lib-${{ github.sha }} | ||
| restore-keys: | | ||
| kubeflow-common-lib- | ||
|
|
||
| - name: Install common library dependencies | ||
| run: | | ||
| cd components/crud-web-apps/common/frontend/kubeflow-common-lib | ||
| npm ci | ||
| npm link ./dist/kubeflow | ||
|
|
||
| - name: Install TWA dependencies | ||
| run: | | ||
| cd components/crud-web-apps/tensorboards/frontend | ||
| npm ci | ||
| npm link kubeflow | ||
|
|
||
| - name: Run unit tests | ||
| run: | | ||
| cd components/crud-web-apps/tensorboards/frontend | ||
| npm run test:prod | ||
|
|
||
| frontend-ui-tests: | ||
| name: UI tests with Cypress | ||
| runs-on: ubuntu-22.04 | ||
| needs: build-common-lib | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: 16 | ||
| cache: 'npm' | ||
| cache-dependency-path: 'components/crud-web-apps/tensorboards/frontend/package-lock.json' | ||
|
|
||
| - name: Restore built common library | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: components/crud-web-apps/common/frontend/kubeflow-common-lib/dist | ||
| key: kubeflow-common-lib-${{ github.sha }} | ||
| restore-keys: | | ||
| kubeflow-common-lib- | ||
|
|
||
| - name: Install common library dependencies | ||
| run: | | ||
| cd components/crud-web-apps/common/frontend/kubeflow-common-lib | ||
| npm ci | ||
| npm link ./dist/kubeflow | ||
|
|
||
| - name: Install TWA dependencies | ||
| run: | | ||
| cd components/crud-web-apps/tensorboards/frontend | ||
| npm ci | ||
| npm link kubeflow | ||
|
|
||
| - name: Run Cypress tests | ||
| run: | | ||
| cd components/crud-web-apps/tensorboards/frontend | ||
| npm run serve > serve.log 2>&1 & npx wait-on http://localhost:4200 | ||
| npm run ui-test-ci-all | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| name: TWA Integration Test | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - components/crud-web-apps/tensorboards/** | ||
| - components/crud-web-apps/common/** | ||
| - releasing/version/VERSION | ||
| branches: | ||
| - main | ||
| - v*-branch | ||
| - notebooks-v1 | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.number }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| IMG: ghcr.io/kubeflow/notebooks/tensorboards-web-app | ||
| TAG: integration-test | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Build TWA Image | ||
| run: | | ||
| cd components/crud-web-apps/tensorboards | ||
| make docker-build | ||
|
|
||
| - name: Install KinD | ||
| run: ./components/testing/gh-actions/install_kind.sh | ||
|
|
||
| - name: Create KinD Cluster | ||
| run: kind create cluster --config components/testing/gh-actions/kind-1-25.yaml | ||
|
|
||
| - name: Load Image into KinD Cluster | ||
| run: | | ||
| kind load docker-image "${IMG}:${TAG}" | ||
|
|
||
| - name: Install kustomize | ||
| run: ./components/testing/gh-actions/install_kustomize.sh | ||
|
|
||
| - name: Install Istio | ||
| run: ./components/testing/gh-actions/install_istio.sh | ||
|
|
||
| - name: Build & Apply manifests | ||
| run: | | ||
| cd components/crud-web-apps/tensorboards/manifests | ||
| kubectl create ns kubeflow | ||
|
|
||
| export CURRENT_IMAGE="${IMG}" | ||
| export PR_IMAGE="${IMG}:${TAG}" | ||
|
|
||
| # escape "." in the image names, as it is a special characters in sed | ||
| export CURRENT_IMAGE=$(echo "$CURRENT_IMAGE" | sed 's|\.|\\.|g') | ||
| export PR_IMAGE=$(echo "$PR_IMAGE" | sed 's|\.|\\.|g') | ||
|
|
||
| kustomize build overlays/istio \ | ||
| | sed "s|${CURRENT_IMAGE}:[a-zA-Z0-9_.-]*|${PR_IMAGE}|g" \ | ||
| | kubectl apply -f - | ||
|
|
||
| kubectl wait pods -n kubeflow -l app=tensorboards-web-app --for=condition=Ready --timeout=300s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| name: TWA Multi-Arch Build Test | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - components/crud-web-apps/tensorboards/** | ||
| - components/crud-web-apps/common/** | ||
| - releasing/version/VERSION | ||
| branches: | ||
| - main | ||
| - v*-branch | ||
| - notebooks-v1 | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.number }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| IMG: ghcr.io/kubeflow/notebooks/tensorboards-web-app | ||
| PLATFORMS: linux/amd64,linux/ppc64le,linux/arm64/v8 | ||
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup QEMU | ||
| uses: docker/setup-qemu-action@v3 | ||
|
|
||
| - name: Setup Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - name: Build multi-arch images | ||
| uses: docker/build-push-action@v5 | ||
| with: | ||
| context: components/crud-web-apps | ||
| file: components/crud-web-apps/tensorboards/Dockerfile | ||
| platforms: ${{ env.PLATFORMS }} | ||
| push: false | ||
| load: false | ||
| tags: | | ||
| ${{ env.IMG }}:${{ github.sha }} | ||
| ${{ env.IMG }}:latest | ||
| cache-from: type=gha | ||
| cache-to: type=gha,mode=max | ||
|
Comment on lines
+34
to
+46
There was a problem hiding this comment. Please see my comments provided on a different PR that outlines how/why I'd like to simplify the configuration of
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While I greatly appreciate your effort and investigation into bringing caching into this workflow - I'd prefer for now we keep things more inline with how the
kubeflow/kubeflowworkflow is structured.I created a branch off your branch that strips the caching changes out of this workflow and verified its still running successfully if you want to use as reference:
Main motivations behind preferring to avoid caching for now:
K.I.S.Sprinciplesnotebooks-v1branch - I don't want to increase "complexity" by adding features/functionality not present inkubeflow/kubeflowthat aren't absolutely necessarybuild-push-actionas opposed to theMakefiletarget.. but I view that as "absolutely necessary" giving the "hanging build" observations you previously highlightednotebooks-v1andnotebooks-v2exist within the same repo - they are relegated to sharing the same cache. If we consistently rolled out caching changes in the manner outlined here - I am concerned we wouldn't have enough cache space to support all the various components (which means the cache would be full and constantly churning on cache misses). I saw a very simple NodeJS App consume ~100MB of cache (and we have a 10GB repo limit).If/as we see long run times become a problem for us testing/publishing packages - its good to keep these capabilities in mind.. but for now - I don't see the need to cache as the time taken to test and/or publish is not so egregious as to impact our work.
Happy to discuss more in the event you disagree!