Skip to content

Conversation

@beonde
Copy link
Member

@beonde beonde commented Dec 27, 2025

Summary

Adds comprehensive E2E test suite for the CLI wrapper, validating all P0 CLI commands against the actual capiscio-core binary.

P0 E2E Tests Added

Test File Tests Coverage
validate.e2e.test.ts 8 Validation command
badge.e2e.test.ts 9 Badge issue/verify commands
score.e2e.test.ts 4 Scoring (via validation)
status.e2e.test.ts 8 CLI status/version checks

Total: 29 new E2E tests

Infrastructure

  • vitest.config.e2e.ts: E2E test configuration
  • tests/e2e/setup.ts: Shared fixtures and CLI runner
  • tests/e2e/fixtures/: Test agent cards (valid, invalid, malformed)
  • .github/workflows/e2e.yml: E2E test workflow
  • package.json: Added test:e2e script

Changes

  • Reorganized tests into tests/unit/ and tests/e2e/ directories
  • Added tests/README.md with test documentation
  • Updated vitest.config.ts for unit tests only

Testing

# Run E2E tests
pnpm test:e2e

# Run unit tests
pnpm test

P0 E2E Tests Added:
- validate.e2e.test.ts: 8 tests for validation command
- badge.e2e.test.ts: 9 tests for badge issue/verify commands
- score.e2e.test.ts: 4 tests for scoring (via validation)
- status.e2e.test.ts: 8 tests for CLI status/version checks

Infrastructure:
- vitest.config.e2e.ts: E2E test configuration
- tests/e2e/setup.ts: Shared fixtures and CLI runner
- tests/e2e/fixtures/: Test agent cards (valid, invalid, malformed)
- .github/workflows/e2e.yml: E2E test workflow
- package.json: Added test:e2e script

Reorganized:
- tests/unit/: Moved existing unit tests
- tests/README.md: Test documentation
- vitest.config.ts: Updated for unit tests only

Total: 29 new E2E tests
Copilot AI review requested due to automatic review settings December 27, 2025 14:22
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a comprehensive E2E test suite for the CLI wrapper, reorganizing the test structure into separate unit and E2E directories. The changes include 29 new E2E tests validating P0 CLI commands (validate, badge, score, status) against a live server, along with supporting infrastructure and CI/CD workflows.

Key Changes

  • Reorganized test structure separating unit tests from E2E tests
  • Added 29 E2E tests across 4 test files with fixtures and shared test infrastructure
  • Created separate vitest configurations for unit and E2E tests with appropriate timeouts and execution modes

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
vitest.config.ts Updated coverage exclusions to exclude tests directory; missing include pattern for unit tests
vitest.config.e2e.ts New E2E test configuration with serial execution and extended timeouts
tests/unit/cli.test.ts Updated import paths from '../' to '../../src/' to reflect new directory structure
tests/e2e/setup.ts Global E2E setup with server health check and configuration
tests/e2e/validate.e2e.test.ts 8 tests for validate command with duplicated helper function
tests/e2e/status.e2e.test.ts 8 tests for status commands using conditional test skipping
tests/e2e/score.e2e.test.ts 8 tests for score command with temporary file creation concerns
tests/e2e/badge.e2e.test.ts 9 tests for badge issuance and verification
tests/e2e/fixtures/*.json Test data files including valid, invalid, and malformed agent cards
tests/README.md Comprehensive test documentation with security issue in .env loading example
package.json New test scripts using --dir flag which may not be a valid vitest option
.github/workflows/e2e.yml CI/CD workflow with potential service networking and Node version concerns
Comments suppressed due to low confidence (1)

vitest.config.ts:38

  • The vitest.config.ts file does not specify an 'include' pattern to limit which test files are run. Without this, when running 'vitest run --config vitest.config.ts', vitest will use its default glob patterns which may pick up E2E tests as well. Add an 'include' pattern such as 'include: ["tests/unit//*.test.ts", "src//*.test.ts"]' to ensure only unit tests are executed with this configuration.
  test: {
    // Environment setup
    environment: 'node',
    
    // Run tests serially to avoid build race conditions
    pool: 'forks',
    poolOptions: {
      forks: {
        singleFork: true
      }
    },
    
    // Set timeouts for integration tests
    testTimeout: 30000,
    hookTimeout: 30000,
    
    // Coverage configuration
    coverage: {
      provider: 'v8',
      include: ['src/**/*.ts'],
      exclude: [
        'src/**/*.test.ts',
        'src/**/*.spec.ts',
        'src/__tests__/**',
        'tests/**',
        'src/cli.ts' // CLI entry point is tested via integration
      ],
      // Coverage thresholds - fail if below 70%
      thresholds: {
        lines: 70,
        functions: 70,
        branches: 70,
        statements: 70
      }
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add credentials block to service container using REPO_ACCESS_TOKEN secret
to authenticate with ghcr.io for pulling the private capiscio-server image.

Also updated:
- Node versions: 16 -> 22
- Package manager: npm -> pnpm
Switch from pulling ghcr.io/capiscio/capiscio-server:latest to building
from source. This allows E2E tests to run without requiring a release,
enabling downstream testing during development.

Pattern matches capiscio-e2e-tests approach:
- Checkout capiscio-server with REPO_ACCESS_TOKEN
- Build via Dockerfile or Go directly
- Start server with test configuration
- Run E2E tests against localhost:8080
Copilot AI review requested due to automatic review settings December 27, 2025 14:33
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Remove score.e2e.test.ts (CLI has no 'score' command)
- Remove status.e2e.test.ts (CLI has no 'status' command)
- Remove setup.ts (tests no longer need server)
- Fix validate tests to use --schema-only for offline validation
- Fix badge tests to use actual CLI flags (--self-sign, --accept-self-signed)
- Update valid-agent-card.json to match A2A v1.3.0 schema
- Simplify e2e.yml workflow (no longer needs server/postgres)

Tests now run entirely offline using CLI features:
- badge issue --self-sign
- badge verify --accept-self-signed --offline
- validate --schema-only
Copilot AI review requested due to automatic review settings December 27, 2025 15:49
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 12 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Extract runCapiscio/extractToken to shared helpers.ts
- Rename malformed.json to malformed.txt to avoid linter issues
- Update imports to use shared helpers
- Remove unused artifact upload from e2e.yml
- Update tests/README.md to match actual test coverage
@beonde beonde merged commit ca9952a into main Dec 27, 2025
7 checks passed
@beonde beonde deleted the test/e2e-suite-2025-12-26 branch December 27, 2025 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants