-
Notifications
You must be signed in to change notification settings - Fork 0
test: add comprehensive E2E test suite for CLI wrapper #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
P0 E2E Tests Added: - validate.e2e.test.ts: 8 tests for validation command - badge.e2e.test.ts: 9 tests for badge issue/verify commands - score.e2e.test.ts: 4 tests for scoring (via validation) - status.e2e.test.ts: 8 tests for CLI status/version checks Infrastructure: - vitest.config.e2e.ts: E2E test configuration - tests/e2e/setup.ts: Shared fixtures and CLI runner - tests/e2e/fixtures/: Test agent cards (valid, invalid, malformed) - .github/workflows/e2e.yml: E2E test workflow - package.json: Added test:e2e script Reorganized: - tests/unit/: Moved existing unit tests - tests/README.md: Test documentation - vitest.config.ts: Updated for unit tests only Total: 29 new E2E tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds a comprehensive E2E test suite for the CLI wrapper, reorganizing the test structure into separate unit and E2E directories. The changes include 29 new E2E tests validating P0 CLI commands (validate, badge, score, status) against a live server, along with supporting infrastructure and CI/CD workflows.
Key Changes
- Reorganized test structure separating unit tests from E2E tests
- Added 29 E2E tests across 4 test files with fixtures and shared test infrastructure
- Created separate vitest configurations for unit and E2E tests with appropriate timeouts and execution modes
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| vitest.config.ts | Updated coverage exclusions to exclude tests directory; missing include pattern for unit tests |
| vitest.config.e2e.ts | New E2E test configuration with serial execution and extended timeouts |
| tests/unit/cli.test.ts | Updated import paths from '../' to '../../src/' to reflect new directory structure |
| tests/e2e/setup.ts | Global E2E setup with server health check and configuration |
| tests/e2e/validate.e2e.test.ts | 8 tests for validate command with duplicated helper function |
| tests/e2e/status.e2e.test.ts | 8 tests for status commands using conditional test skipping |
| tests/e2e/score.e2e.test.ts | 8 tests for score command with temporary file creation concerns |
| tests/e2e/badge.e2e.test.ts | 9 tests for badge issuance and verification |
| tests/e2e/fixtures/*.json | Test data files including valid, invalid, and malformed agent cards |
| tests/README.md | Comprehensive test documentation with security issue in .env loading example |
| package.json | New test scripts using --dir flag which may not be a valid vitest option |
| .github/workflows/e2e.yml | CI/CD workflow with potential service networking and Node version concerns |
Comments suppressed due to low confidence (1)
vitest.config.ts:38
- The vitest.config.ts file does not specify an 'include' pattern to limit which test files are run. Without this, when running 'vitest run --config vitest.config.ts', vitest will use its default glob patterns which may pick up E2E tests as well. Add an 'include' pattern such as 'include: ["tests/unit//*.test.ts", "src//*.test.ts"]' to ensure only unit tests are executed with this configuration.
test: {
// Environment setup
environment: 'node',
// Run tests serially to avoid build race conditions
pool: 'forks',
poolOptions: {
forks: {
singleFork: true
}
},
// Set timeouts for integration tests
testTimeout: 30000,
hookTimeout: 30000,
// Coverage configuration
coverage: {
provider: 'v8',
include: ['src/**/*.ts'],
exclude: [
'src/**/*.test.ts',
'src/**/*.spec.ts',
'src/__tests__/**',
'tests/**',
'src/cli.ts' // CLI entry point is tested via integration
],
// Coverage thresholds - fail if below 70%
thresholds: {
lines: 70,
functions: 70,
branches: 70,
statements: 70
}
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add credentials block to service container using REPO_ACCESS_TOKEN secret to authenticate with ghcr.io for pulling the private capiscio-server image. Also updated: - Node versions: 16 -> 22 - Package manager: npm -> pnpm
Switch from pulling ghcr.io/capiscio/capiscio-server:latest to building from source. This allows E2E tests to run without requiring a release, enabling downstream testing during development. Pattern matches capiscio-e2e-tests approach: - Checkout capiscio-server with REPO_ACCESS_TOKEN - Build via Dockerfile or Go directly - Start server with test configuration - Run E2E tests against localhost:8080
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Remove score.e2e.test.ts (CLI has no 'score' command) - Remove status.e2e.test.ts (CLI has no 'status' command) - Remove setup.ts (tests no longer need server) - Fix validate tests to use --schema-only for offline validation - Fix badge tests to use actual CLI flags (--self-sign, --accept-self-signed) - Update valid-agent-card.json to match A2A v1.3.0 schema - Simplify e2e.yml workflow (no longer needs server/postgres) Tests now run entirely offline using CLI features: - badge issue --self-sign - badge verify --accept-self-signed --offline - validate --schema-only
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 12 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Extract runCapiscio/extractToken to shared helpers.ts - Rename malformed.json to malformed.txt to avoid linter issues - Update imports to use shared helpers - Remove unused artifact upload from e2e.yml - Update tests/README.md to match actual test coverage
Summary
Adds comprehensive E2E test suite for the CLI wrapper, validating all P0 CLI commands against the actual
capiscio-corebinary.P0 E2E Tests Added
validate.e2e.test.tsbadge.e2e.test.tsscore.e2e.test.tsstatus.e2e.test.tsTotal: 29 new E2E tests
Infrastructure
vitest.config.e2e.ts: E2E test configurationtests/e2e/setup.ts: Shared fixtures and CLI runnertests/e2e/fixtures/: Test agent cards (valid, invalid, malformed).github/workflows/e2e.yml: E2E test workflowpackage.json: Addedtest:e2escriptChanges
tests/unit/andtests/e2e/directoriestests/README.mdwith test documentationvitest.config.tsfor unit tests onlyTesting