Skip to content

Conversation

@emosbaugh
Copy link
Member

@emosbaugh emosbaugh commented Dec 3, 2025

What this PR does / why we need it:

This PR implements a new V3 E2E testing framework using Dagger for embedded-cluster headless CLI installations. The framework provides portable, reproducible end-to-end tests that run identically in both local development and CI environments.

Key Features:

  • Dagger-based orchestration: All E2E tests are defined as Dagger functions, enabling reproducible builds and tests
  • CMX VM provisioning: Automated VM lifecycle management using Replicated's CMX service for isolated test environments
  • Headless installation testing: Tests online installation scenarios via CLI
  • Comprehensive validation: Multi-stage validation checks covering cluster health, CRD status, app deployment, admin console, data directories, and pod/job health
  • 1Password integration: Centralized secret management for CMX tokens, SSH keys, and API credentials
  • Portable execution: Same test commands work locally and in CI without modification

Architecture:

  • dagger/e2e.go: E2E test orchestration with unified E2eRunHeadless function for both online scenarios
  • dagger/cmx.go: CMX VM provisioning, SSH command execution, installation logic, and airgap network policy management
  • dagger/validation.go: Comprehensive validation suite with 6 categories of checks
  • dagger/e2etypes.go: Type definitions for test results and validation outcomes
  • .github/workflows/v3-e2e.yaml: GitHub Actions workflow running tests in parallel

Changes:

  1. Added E2E test framework with support for online headless installations
  2. Implemented CMX VM provisioning and lifecycle management
  3. Created comprehensive validation suite checking cluster health, CRDs, app deployment, admin console, data directories, and pod/job status
  4. Added GitHub Actions workflow for automated CI testing
  5. Added airgap network policy support to block internet access during airgap tests
  6. Updated documentation with examples and usage instructions
  7. Cleaned up old/unused test files and scripts

Test Flow:

  1. Provision fresh Ubuntu 22.04 VM via CMX
  2. For airgap: Apply network policy to block internet access
  3. Download and install embedded-cluster via CLI
  4. Run comprehensive validation checks
  5. Clean up VM automatically (unless --skip-cleanup specified)

Example Usage:

# Online test
dagger call with-one-password --service-account=env:OP_SERVICE_ACCOUNT_TOKEN \
  e-2-e-run-headless \
  --scenario=online \
  --app-version=appver-dev-xpXCTO \
  --kube-version=1.33 \
  --license-file=./e2e/licenses/ci-v3.yaml

Which issue(s) this PR fixes:

Part of the V3 E2E testing initiative. Related to the embedded-cluster V3 testing strategy.

Does this PR require a test?

The PR itself adds comprehensive E2E tests. Tests are included and will run automatically in CI via the new .github/workflows/v3-e2e.yaml workflow.

Does this PR require a release note?

NONE

This is a CI/testing infrastructure change with no customer-facing impact.

Does this PR require documentation?

NONE

This is internal testing infrastructure. The dagger/README.md has been updated with comprehensive documentation for developers.

@github-actions
Copy link

github-actions bot commented Dec 3, 2025

This PR has been released (on staging) and is available for download with a embedded-cluster-smoke-test-staging-app license ID.

Online Installer:

curl "https://staging.replicated.app/embedded/embedded-cluster-smoke-test-staging-app/ci/appver-dev-1663529" -H "Authorization: $EC_SMOKE_TEST_LICENSE_ID" -o embedded-cluster-smoke-test-staging-app-ci.tgz

Airgap Installer (may take a few minutes before the airgap bundle is built):

curl "https://staging.replicated.app/embedded/embedded-cluster-smoke-test-staging-app/ci-airgap/appver-dev-1663529?airgap=true" -H "Authorization: $EC_SMOKE_TEST_LICENSE_ID" -o embedded-cluster-smoke-test-staging-app-ci.tgz

Happy debugging!

@emosbaugh emosbaugh force-pushed the emosbaugh/sc-131353/v3-e2e-tests-pr-3-headless-installation-e2e branch from 075f78e to 9116f9d Compare December 8, 2025 17:23
@emosbaugh emosbaugh changed the title chore(ci): v3 e2e test headless chore(ci): v3 e2e test headless online Dec 8, 2025
@emosbaugh emosbaugh marked this pull request as ready for review December 8, 2025 18:03
@emosbaugh
Copy link
Member Author

Test failure is unrelated. A follow up story has been created https://app.shortcut.com/replicated/story/131970/network-request-to-usage-projectcalico-org-in-airgap-mode

@emosbaugh emosbaugh merged commit 4a30e00 into main Dec 8, 2025
151 of 154 checks passed
@emosbaugh emosbaugh deleted the emosbaugh/sc-131353/v3-e2e-tests-pr-3-headless-installation-e2e branch December 8, 2025 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants