refactor(tests): retire script enumeration; pytest discovers tests/integration/

.github/workflows/ci-integration.yml

@@ -148,12 +148,16 @@ jobs:
       - name: Run integration tests
         env:
           APM_E2E_TESTS: "1"
+          APM_RUN_INTEGRATION_TESTS: "1"
           GITHUB_APM_PAT: ${{ secrets.GH_CLI_PAT }}
           ADO_APM_PAT: ${{ secrets.ADO_APM_PAT }}
         run: |
           chmod +x scripts/test-integration.sh
           uv run ./scripts/test-integration.sh
-        timeout-minutes: 20
+        # Bumped from 20 to 30 minutes when test discovery widened from
+        # the 28 enumerated files to the full tests/integration/ suite
+        # (PR2 of #1166).
+        timeout-minutes: 30
 
   release-validation:
     name: Release Validation (Linux)

CHANGELOG.md

@@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Registry proxy now warns when `PROXY_REGISTRY_TOKEN` is set and `PROXY_REGISTRY_URL` uses `http://`, since the bearer token would be transmitted in plaintext; set `PROXY_REGISTRY_ALLOW_HTTP=1` to silence the warning for trusted internal proxies. (#1149)
 - Integration tests now use marker-driven discovery: 21 `pytestmark = pytest.mark.skipif(...)` chains across `tests/integration/` are replaced with declarative `requires_*` markers, with precondition logic centralized in `tests/integration/conftest.py` and auto-skipping at collection time. PR1 of #1166. (#1167)
 - Integration test apm-binary resolution now prefers the local build (`./dist/apm-<os>-<arch>/apm`) over a system-wide `apm` on `PATH`, so contributors validating the binary under test are not silently shadowed by a global install; the bearer-token marker (`requires_ado_bearer`) discards the captured JWT immediately and persists only the boolean outcome. (#1167)
+- `scripts/test-integration.sh` is now a thin orchestrator: it builds/locates the apm binary, sets up runtimes and tokens, then invokes `pytest tests/integration/` exactly once. The 28 per-file pytest enumerations were removed; the marker registry handles per-test gating, and new test files dropped into `tests/integration/` are picked up automatically. PR2 of #1166. (#1247)
 
 ### Fixed
 

docs/src/content/docs/contributing/integration-testing.md

@@ -56,6 +56,7 @@ what the test family you want actually requires.
 | `requires_runtime_codex` | The `codex` runtime installed under `~/.apm/runtimes/` | `apm runtime setup codex` |
 | `requires_runtime_copilot` | The GitHub Copilot CLI runtime installed under `~/.apm/runtimes/` | `apm runtime setup copilot` |
 | `requires_runtime_llm` | The `llm` runtime installed under `~/.apm/runtimes/` | `apm runtime setup llm` |
+| `live` | Tests that hit real GitHub repos via cloning; deselected by default | Override the deselect: `pytest -m live tests/integration -v` |
 
 Without any of those env vars or runtimes a `pytest tests/integration`
 invocation is silent rather than red: every test is collected and
@@ -98,14 +99,22 @@ system install:
    skip reason) and declare the marker in `pyproject.toml`. That is
    the only place the precondition needs to live.
 
-### Legacy: `scripts/test-integration.sh`
-
-`scripts/test-integration.sh` is the legacy wrapper that built a
-binary, set up runtimes, and shelled out to pytest. It is being
-retired (see `microsoft/apm#1166`); prefer the direct `pytest`
-invocations above. The script is still wired into CI for the moment
-and continues to work, but new test plumbing belongs in the marker
-registry, not in the bash script.
+### CI orchestrator: `scripts/test-integration.sh`
+
+`scripts/test-integration.sh` is the thin orchestrator the CI
+integration job invokes. Its sole responsibilities are: resolve
+GitHub / ADO tokens, detect platform, locate or build the apm
+PyInstaller binary, install runtimes (codex / copilot / llm),
+install python test dependencies, and run
+`pytest tests/integration/` once. All per-test gating lives in the
+marker registry described above. New integration tests dropped into
+`tests/integration/` are picked up automatically; add the right
+`requires_*` marker and the registry will skip the test when its
+precondition is missing.
+
+The orchestrator is mainly intended for reproducing the full CI
+environment end-to-end; for local iteration prefer the direct
+`pytest` invocations earlier on this page.
 
 ## CI/CD Integration
 

pyproject.toml

@@ -128,7 +128,7 @@ warn_return_any = true
 warn_unused_configs = true
 
 [tool.pytest.ini_options]
-addopts = "-m 'not benchmark'"
+addopts = "-m 'not benchmark and not live'"
 markers = [
     "integration: marks tests as integration tests that may require network access",
     "live: marks tests that hit real GitHub repos (requires network + optional GITHUB_TOKEN)",