Skip to content

[plan] Create integration test suite for agent-runner environment parityย #11975

@github-actions

Description

@github-actions

Objective

Develop a comprehensive integration test suite that validates the agent container environment has parity with the GitHub Actions runner environment for essential tools, libraries, and environment variables.

Context

As we close the gap between the agent container and the action runner, we need automated tests to verify that:

  1. Mounted utilities are accessible and functional
  2. Runtime tools (Python, Node.js, Go, Ruby) can be found and executed
  3. Environment variables are correctly mirrored
  4. Shared libraries are properly linked

Without these tests, we risk regressions and incomplete implementations.

Approach

  1. Create a test workflow that runs inside the agent container and verifies:
    • Essential /usr/bin utilities are accessible (jq, curl, git, etc.)
    • All runtime binaries are in PATH and executable (node, python3, go, ruby)
    • Environment variables are set correctly (JAVA_HOME, ANDROID_HOME, etc.)
    • Shared libraries can be loaded (test with ldd on key binaries)
  2. Add Go integration tests in pkg/workflow/*_test.go files
  3. Create a smoke test workflow in .github/workflows/agent-runner-parity.md
  4. Document the test approach in specs/agent-container-testing.md

Files to Create/Modify

  • Create: pkg/workflow/agent_parity_test.go (integration tests)
  • Create: .github/workflows/agent-runner-parity.md (smoke test workflow)
  • Create: specs/agent-container-testing.md (test documentation)
  • Modify: Makefile (add make test-parity target)

Acceptance Criteria

  • Integration tests cover utility accessibility (at least 10 utilities)
  • Integration tests verify runtime availability (node, python3, go, ruby)
  • Integration tests validate environment variable presence (at least 5 vars)
  • Smoke test workflow runs successfully in CI
  • Test documentation explains testing strategy and coverage
  • All tests pass consistently across multiple runs
    Related to epic: build/test environment for agentic workflowย #11970

AI generated by Plan Command for #11970

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions