Skip to content

feat: first-party file-source runtime config and provider-agnostic AI settings#65

Merged
joyzoursky merged 27 commits intomainfrom
feat/provider-agnostic-ai-config-clean
Apr 7, 2026
Merged

feat: first-party file-source runtime config and provider-agnostic AI settings#65
joyzoursky merged 27 commits intomainfrom
feat/provider-agnostic-ai-config-clean

Conversation

@IniZio
Copy link
Copy Markdown
Contributor

@IniZio IniZio commented Apr 6, 2026

What this PR introduces

  • add first-party CLI run commands:
    • skytest run test-case <display-id> --project-id <id> ...
    • skytest run project <project-id> [--display-id ...] ...
  • remove bespoke direct run script path and route execution through dispatch/status APIs
  • added support for file-source runtime config
  • add provider-agnostic team AI runtime config (OpenRouter / OpenAI-compatible)
  • harden auth proxy URL/header validation and add local Authgear bootstrap + seed defaults for self-host setups

Step-by-step reviewer guide (from zero)

  1. Clone skytest-agent and checkout this branch.
  2. Start SkyTest control plane:
    • make bootstrap
    • make dev
  3. Keep this running and verify health:
    • curl -fsS http://127.0.0.1:3000/api/health/live
  4. Create example project
  5. In SkyTest web (http://127.0.0.1:3000), create/get:
    • a project ID for Hanlun (SKYTEST_PROJECT_ID)
    • an API key (SKYTEST_API_KEY)
  6. In Hanlun repo, export envs and run:
    • export SKYTEST_BASE_URL=http://127.0.0.1:3000
    • export SKYTEST_PROJECT_ID=<your-project-id>
    • export SKYTEST_API_KEY=<your-api-key>
    • make -C .skytest setup
    • make -C .skytest sync
    • make -C .skytest run-project
  7. Expected final result: JSON summary with "status": "PASS" for all selected cases.

Verification in this PR

  • npx vitest --run apps/cli/src/cli-parser.test.ts apps/cli/src/commands/init.test.ts
  • npm run --workspace @skytest/web test -- src/app/api/test-cases/[id]/route.test.ts src/app/api/test-runs/dispatch/route.test.ts src/lib/runtime/__tests__/team-ai-config.test.ts
  • npm run verify (fails only at audit due existing Vite advisories: GHSA-4w7w-66w2-5vf9, GHSA-v2wj-q39q-566r, GHSA-p9ff-h696-f583)

@joyzoursky joyzoursky force-pushed the feat/provider-agnostic-ai-config-clean branch from 356fe22 to fda9cf8 Compare April 7, 2026 07:34
@joyzoursky joyzoursky force-pushed the feat/provider-agnostic-ai-config-clean branch from 933ad9b to 59e8aaa Compare April 7, 2026 10:15
@joyzoursky joyzoursky marked this pull request as ready for review April 7, 2026 11:01
@joyzoursky joyzoursky merged commit 902d943 into main Apr 7, 2026
5 checks passed
@joyzoursky joyzoursky deleted the feat/provider-agnostic-ai-config-clean branch April 7, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants