Testing Infrastructure Improvements: Implement Recommendations from Testing Summary

[jeremyeder]

Overview

This issue tracks the analysis and implementation of testing recommendations documented in docs/testing/testing-summary.md.

A comprehensive testing audit has been completed, revealing several gaps in our current CI/CD testing strategy. While we have solid E2E testing and linting, critical unit/integration tests exist but are not enforced in CI.

Current State

Test Coverage Matrix

Component	Location	Framework	CI Enforcement	Status
E2E	e2e/cypress/e2e/	Cypress 13.x	✅ Blocking	Good
Backend	components/backend/tests/	Go test	⚠️ Not enforced	Tests exist but not run
Frontend	N/A	N/A	❌ Missing	0% coverage
Operator	components/operator/	Go test	⚠️ Not enforced	Tests exist but not run
Claude Runner	components/runners/claude-code-runner/tests/	pytest	✅ Blocking	Good

What's Working Well ✅

• Comprehensive E2E testing in Kind (Kubernetes in Docker)
• All linting enforced (Go: gofmt/vet/golangci-lint, Frontend: ESLint/TypeScript)
• Change detection optimizes CI performance
• Good artifact collection on failures (screenshots, videos, logs)
• Codecov integration for runner tests

Critical Gaps ⚠️

1. Backend/Operator Go tests exist but aren't run in CI - Breaking changes can merge
2. Zero frontend unit tests - UI bugs can slip through undetected
3. E2E tests skip session execution - No validation of actual Claude Code workflows
4. No security scanning - Vulnerabilities can be introduced unnoticed
5. No performance testing - Production scalability unknown

---

Recommendations

🔴 Critical Priority (High Value, Low Effort)

1. Enforce Backend/Operator Go Tests in CI

Problem: Comprehensive test suites exist in components/backend/tests/ and components/operator/internal/handlers/sessions_test.go but are NOT run in CI.

Impact: Breaking changes can merge if they pass linting but fail tests.

Solution: Add to .github/workflows/go-lint.yml:

- name: Test Backend
  working-directory: components/backend
  run: go test ./...

- name: Test Operator
  working-directory: components/operator
  run: go test ./...

Effort: 15 minutes | Value: High

---

2. Add Security Scanning

Problem: No automated vulnerability scanning for dependencies or container images.

Solution:

• Add Trivy image scanning to components-build-deploy.yml
• Add govulncheck for Go vulnerabilities
• Add npm audit to frontend-lint.yml

Effort: 30 minutes | Value: High

---

🟡 High Priority (High Value, Medium Effort)

3. Create Frontend Unit Tests

Problem: Zero test coverage for NextJS frontend (Jest configured but unused).

Solution:

1. Create example component tests (Button.test.tsx)
2. Add React Query hook tests (useProjects.test.ts)
3. Add page integration tests (projects/page.test.tsx)
4. Add npm test to frontend-lint.yml

Effort: 2-3 hours | Value: High

---

4. Expand E2E Tests to Include Session Execution

Problem: E2E tests validate deployment but skip actual Claude Code execution.

Solution:

• Add mock Claude API responses for testing
• Create test session that doesn't require Anthropic API key
• Verify session lifecycle (Pending → Running → Completed)

Effort: 3-4 hours | Value: Medium

---

🟢 Medium Priority (Medium Value, Variable Effort)

5. Add Performance/Load Testing

Problem: No tests for concurrent sessions, resource limits, timeout handling.

Solution:

• Add k6 or Locust load tests
• Test concurrent session creation (10, 50, 100 sessions)
• Measure operator reconciliation latency
• Test resource limits (CPU, memory, storage)

Effort: 1-2 days | Value: Medium

---

6. Isolate Operator Tests with envtest

Problem: Operator tests require manual cluster setup, not automated in CI.

Solution:

• Create dedicated operator-tests.yml workflow
• Use envtest for isolated controller testing
• Test watch loop reconnection, status updates, Job creation

Effort: 2-3 hours | Value: Medium

---

Quick Wins (Can be completed in < 1 hour each)

1. ✅ Enforce Backend Go Tests (15 min) - Add go test to go-lint.yml
2. ✅ Add Security Scanning (30 min) - Trivy + govulncheck workflows
3. ✅ Document Test Conventions (30 min) - Add to CLAUDE.md
4. ✅ Create Frontend Test Examples (1 hour) - 2-3 example component tests
5. ✅ Add Test Coverage Badges (15 min) - Codecov badges in README

---

Implementation Checklist

Phase 1: Critical Fixes (Week 1)

• Add backend Go tests to CI (go-lint.yml)
• Add operator Go tests to CI (go-lint.yml)
• Add Trivy image scanning (components-build-deploy.yml)
• Add govulncheck for Go vulnerabilities
• Add npm audit to frontend linting

Phase 2: Frontend Testing (Week 2)

• Create example component tests (Button, Card, etc.)
• Add React Query hook tests (useProjects, useSessions)
• Add page integration tests (projects page, session page)
• Add npm test to frontend-lint.yml
• Add frontend coverage reporting

Phase 3: E2E Enhancements (Week 3)

• Design mock Claude API response system
• Create test session without Anthropic API requirement
• Add session lifecycle tests (Pending → Running → Completed)
• Add session failure scenario tests
• Document E2E test expansion strategy

Phase 4: Advanced Testing (Week 4+)

• Add performance/load testing framework (k6 or Locust)
• Create operator envtest suite
• Add test coverage badges to README
• Document testing best practices in CLAUDE.md
• Create testing contribution guide

---

Success Criteria

• ✅ All existing backend/operator Go tests enforced in CI
• ✅ Security scanning catches vulnerabilities before merge
• ✅ Frontend has >50% test coverage for critical paths
• ✅ E2E tests validate complete session lifecycle
• ✅ Performance baselines established for concurrent operations
• ✅ All test results visible in PR checks

---

References

• Testing Summary: docs/testing/testing-summary.md
• E2E Testing Guide: docs/testing/e2e-guide.md
• E2E Tests: e2e/cypress/e2e/vteam.cy.ts
• Backend Tests: components/backend/tests/
• Operator Tests: components/operator/internal/handlers/sessions_test.go

---

Next Steps

1. Review and prioritize recommendations
2. Assign owners for each phase
3. Create sub-issues for complex items (frontend testing, performance testing)
4. Update this issue with progress as items are completed

Estimated Total Effort: 2-3 weeks for Phases 1-3, ongoing for Phase 4

cc: @jeremyeder

[jeremyeder]

This is waiting for someone to pick it up.

[jeremyeder]

create separate config files for backend and operator tests. dont use go-lint.yml