[docs] Update testing docs

[junaway]

---

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Mar 2, 2026 10:50am

[Copilot]

Pull request overview

This PR updates the monorepo testing taxonomy and structure by renaming “e2e” to “acceptance”, adding “integration” placeholders, updating docs, and introducing/relocating multiple Playwright and pytest acceptance suites across web, sdk, and api.

Changes:

• Renames web Playwright configuration and scripts from e2e to acceptance.
• Adds/organizes Playwright acceptance tests for OSS and EE (apps, playground, settings, testsets, etc.).
• Updates pytest markers/docs and adds API acceptance tests under api/oss/tests/pytest/acceptance/.

Reviewed changes

Copilot reviewed 22 out of 119 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
web/tests/playwright.config.ts	Point Playwright testDir to acceptance
web/tests/package.json	Rename scripts to test:acceptance*
web/oss/tests/playwright/integration/.gitkeep	Integration placeholder
web/oss/tests/playwright/acceptance/testsset/testset.spec.ts	Add OSS testset spec wrapper
web/oss/tests/playwright/acceptance/testsset/index.ts	Add OSS testset flow test
web/oss/tests/playwright/acceptance/smoke.spec.ts	Add OSS smoke navigation test
web/oss/tests/playwright/acceptance/settings/model-hub.ts	Add OSS model hub flow
web/oss/tests/playwright/acceptance/settings/model-hub.spec.ts	Add OSS model hub spec wrapper
web/oss/tests/playwright/acceptance/settings/api-keys.ts	Add OSS API keys flow
web/oss/tests/playwright/acceptance/settings/api-keys-management.spec.ts	Add (skipped) OSS API keys spec
web/oss/tests/playwright/acceptance/prompt-registry/prompt-registry-flow.spec.ts	Add OSS prompt registry spec
web/oss/tests/playwright/acceptance/prompt-registry/index.ts	Add OSS prompt registry flow
web/oss/tests/playwright/acceptance/playground/tests.ts	Add playground fixture extensions
web/oss/tests/playwright/acceptance/playground/run-variant.spec.ts	Add OSS playground spec wrapper
web/oss/tests/playwright/acceptance/playground/index.ts	Add OSS playground test suite
web/oss/tests/playwright/acceptance/playground/assets/types.ts	Add playground fixture types
web/oss/tests/playwright/acceptance/playground/assets/constants.ts	Add playground test data constants
web/oss/tests/playwright/acceptance/playground/assets/README.md	Document playground fixtures
web/oss/tests/playwright/acceptance/observability/observability.spec.ts	Add OSS observability spec wrapper
web/oss/tests/playwright/acceptance/observability/index.ts	Add OSS observability flow
web/oss/tests/playwright/acceptance/deployment/index.ts	Add OSS deployment flow
web/oss/tests/playwright/acceptance/deployment/deploy-variant.spec.ts	Add OSS deployment spec wrapper
web/oss/tests/playwright/acceptance/app/test.ts	Add app-management fixture extensions
web/oss/tests/playwright/acceptance/app/index.ts	Add OSS app creation tests
web/oss/tests/playwright/acceptance/app/create.spec.ts	Add OSS app creation spec wrapper
web/oss/tests/playwright/acceptance/app/assets/types.ts	Add app fixture types
web/oss/tests/playwright/acceptance/app/assets/README.md	Document app fixtures
web/ee/tests/playwright/integration/.gitkeep	Integration placeholder
web/ee/tests/playwright/acceptance/testsset/testset.spec.ts	Add EE testset wrapper (imports OSS)
web/ee/tests/playwright/acceptance/settings/model-hub.spec.ts	Add EE model-hub wrapper (imports OSS)
web/ee/tests/playwright/acceptance/settings/api-keys-management.spec.ts	Add EE API keys wrapper (imports OSS)
web/ee/tests/playwright/acceptance/prompt-registry/prompt-registry-flow.spec.ts	Add EE prompt registry wrapper (imports OSS)
web/ee/tests/playwright/acceptance/playground/run-variant.spec.ts	Add EE playground wrapper (imports OSS)
web/ee/tests/playwright/acceptance/observability/observability.spec.ts	Add EE observability wrapper (imports OSS)
web/ee/tests/playwright/acceptance/human-annotation/tests.ts	Add EE human-annotation fixture suite
web/ee/tests/playwright/acceptance/human-annotation/index.ts	Add EE human-annotation tests
web/ee/tests/playwright/acceptance/human-annotation/human-annotation.spec.ts	Add EE human-annotation spec wrapper
web/ee/tests/playwright/acceptance/human-annotation/assets/types.ts	Add human-annotation fixture types
web/ee/tests/playwright/acceptance/deployment/deploy-variant.spec.ts	Add EE deployment wrapper (imports OSS)
web/ee/tests/playwright/acceptance/auto-evaluation/tests.ts	Add EE auto-evaluation fixtures
web/ee/tests/playwright/acceptance/auto-evaluation/run-auto-evaluation.spec.ts	Add EE auto-eval spec wrapper
web/ee/tests/playwright/acceptance/auto-evaluation/index.ts	Add EE auto-evaluation tests
web/ee/tests/playwright/acceptance/auto-evaluation/assets/types.ts	Add auto-eval fixture types
web/ee/tests/playwright/acceptance/auto-evaluation/assets/README.md	Document auto-eval fixtures
web/ee/tests/playwright/acceptance/app/create.spec.ts	Add EE app creation wrapper (imports OSS)
services/oss/tests/pytest/integration/.gitkeep	Integration placeholder
services/oss/tests/pytest/acceptance/.gitkeep	Acceptance placeholder
services/ee/tests/pytest/integration/.gitkeep	Integration placeholder
services/ee/tests/pytest/acceptance/.gitkeep	Acceptance placeholder
sdk/pytest.ini	Rename marker to acceptance, add integration
sdk/oss/tests/pytest/integration/.gitkeep	Integration placeholder
sdk/oss/tests/pytest/acceptance/workflows/test_legacy_applications_manager.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/workflows/test_apps_shared_manager.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/observability/test_observability_traces.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/integrations/test_vault_secrets.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/integrations/test_testsets_manager.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/integrations/test_prompt_template_storage.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/integrations/test_evaluators_manager.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/healthchecks/test_healthchecks.py	Add SDK healthcheck acceptance tests
sdk/oss/tests/pytest/acceptance/healthchecks/conftest.py	Add SDK healthcheck fixtures
sdk/oss/tests/pytest/acceptance/evaluations/test_evaluations_flow.py	Rename marker usage to acceptance
sdk/oss/tests/pytest/acceptance/conftest.py	Add shared acceptance fixtures
sdk/ee/tests/pytest/integration/.gitkeep	Integration placeholder
sdk/ee/tests/pytest/acceptance/.gitkeep	Acceptance placeholder
docs/packs/testing.md	Update CLI examples to acceptance
docs/packs/hosting.md	Document per-worktree env vars
docs/designs/testing/testing.structure.specs.md	Rename e2e→acceptance; add integration
docs/designs/testing/testing.running.specs.md	Update commands and CI wording
docs/designs/testing/testing.interface.web.specs.md	Update web scripts naming
docs/designs/testing/testing.interface.sdk.specs.md	Update SDK split + paths
docs/designs/testing/testing.interface.api.specs.md	Update acceptance terminology
docs/designs/testing/testing.dimensions.specs.md	Update directory examples to acceptance
docs/designs/testing/testing.boundaries.specs.md	Update boundaries to acceptance/integration
docs/designs/testing/README.md	Update status matrix + commands
api/oss/tests/pytest/integration/.gitkeep	Integration placeholder
api/oss/tests/pytest/acceptance/workflows/test_workflows_retrieve.py	Add workflows acceptance tests
api/oss/tests/pytest/acceptance/workflows/test_workflows_queries.py	Add workflows acceptance tests
api/oss/tests/pytest/acceptance/workflows/test_workflows_basics.py	Add workflows acceptance tests
api/oss/tests/pytest/acceptance/workflows/test_workflow_variants_queries.py	Add workflow variants acceptance tests
api/oss/tests/pytest/acceptance/workflows/test_workflow_variants_basics.py	Add workflow variants acceptance tests
api/oss/tests/pytest/acceptance/tracing/test_traces_basics.py	Add tracing acceptance tests
api/oss/tests/pytest/acceptance/tracing/test_spans_basics.py	Add spans acceptance tests
api/oss/tests/pytest/acceptance/testsets/test_testsets_queries.py	Add testsets acceptance tests
api/oss/tests/pytest/acceptance/testsets/test_testsets_files.py	Add testset file acceptance tests
api/oss/tests/pytest/acceptance/testsets/test_testcases_basics.py	Add testcases acceptance tests
api/oss/tests/pytest/acceptance/healthchecks/test_healthchecks.py	Add API healthcheck acceptance tests
api/oss/tests/pytest/acceptance/evaluators/test_evaluators_queries.py	Add evaluators acceptance tests
api/oss/tests/pytest/acceptance/evaluations/test_evaluation_scenarios_queries.py	Add evaluation scenarios query tests
api/oss/tests/pytest/acceptance/evaluations/test_evaluation_scenarios_basics.py	Add evaluation scenarios CRUD tests
api/oss/tests/pytest/acceptance/evaluations/test_evaluation_runs_queries.py	Add evaluation runs query tests
api/oss/tests/pytest/acceptance/evaluations/test_evaluation_metrics_queries.py	Add evaluation metrics query tests
api/oss/tests/pytest/acceptance/annotations/test_annotations_basics.py	Add annotations acceptance tests
api/ee/tests/pytest/integration/.gitkeep	Integration placeholder

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[devin-ai-integration]

Devin Review found 1 potential issue.

🐛 1 issue in files not directly in the diff

🐛 Skip-guard fixture still checks for old e2e marker instead of new acceptance marker (sdk/oss/tests/pytest/acceptance/conftest.py:35-36)

The autouse fixture _skip_e2e_if_missing_env in sdk/oss/tests/pytest/acceptance/conftest.py:35 checks for request.node.get_closest_marker("e2e"), but all test files in this PR were updated to use pytest.mark.acceptance instead of pytest.mark.e2e. The marker name was never updated in the fixture.

Root Cause and Impact

The guard condition request.node.get_closest_marker("e2e") will never match any test because every test file now uses pytestmark = [pytest.mark.acceptance] (e.g., sdk/oss/tests/pytest/acceptance/workflows/test_apps_shared_manager.py:29, sdk/oss/tests/pytest/acceptance/evaluations/test_evaluations_flow.py:25, etc.).

Since get_closest_marker("e2e") returns None for all tests, the if block is never entered and pytest.skip() is never called. This means all acceptance tests will attempt to run even when AGENTA_API_URL and AGENTA_AUTH_KEY are not set, causing them to fail with connection errors instead of being gracefully skipped.

Previously, tests used pytest.mark.e2e and the fixture correctly matched that marker. The rename to acceptance was applied to all test files but not to this fixture.

View 5 additional findings in Devin Review.

[mmabrouk]

Thanks @jp-agenta !

[github-actions]

Railway Preview Environment

Status	Destroyed (PR closed)

Updated at 2026-03-02T11:14:25.783Z