feat(test): add "test scaffold" to emit a schema-correct starter plan or backend skeleton#180
Merged
Merged
Conversation
2 tasks
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughAdds a new ChangesTest Scaffold Feature
Estimated code review effort: 2 (Simple) | ~15 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant CLI as test scaffold CLI
participant runScaffold
participant FileSystem
User->>CLI: testsprite test scaffold --type frontend|backend [--out path] [--force]
CLI->>runScaffold: call with ScaffoldOptions
runScaffold->>runScaffold: build CliPlanInput or CliBackendScaffold
alt --out provided
runScaffold->>FileSystem: check existsSync and directory guard
alt exists and no --force
runScaffold-->>CLI: refuse overwrite
else
runScaffold->>FileSystem: write scaffold file
runScaffold-->>CLI: return written output
end
else no --out
runScaffold-->>CLI: print scaffold JSON to stdout
end
CLI-->>User: output result
Related Issues: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
|
Approved — no changes requested on substance. This went CONFLICTING purely because today's merge wave landed a large batch into the same files first. Rebase onto current |
Andy00L
force-pushed
the
feat/test-scaffold
branch
from
6c806d0 to
6121944
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/commands/test.ts`:
- Around line 3877-3971: The `runScaffold` backend branch writes raw Python to
`--out` even when `opts.output` is `json`, so the file misses the documented
JSON envelope. Update the `runScaffold` path that builds `payload` and `body` so
the `openOutputFile`/`fileOut.writeChunk` branch writes `JSON.stringify(payload,
null, 2)` (or equivalent JSON output) whenever `opts.output === 'json'`, while
preserving the current raw code body for non-JSON output.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 8018b84e-48a3-45f5-9bb8-ffb25e03b193
⛔ Files ignored due to path filters (1)
test/__snapshots__/help.snapshot.test.ts.snapis excluded by!**/*.snap,!**/*.snap
📒 Files selected for processing (2)
src/commands/test.test.tssrc/commands/test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/commands/test.test.ts
Comment on lines
+3877
to
+3971
| /** JSON payload `test scaffold --type backend` prints under --output json. */ | ||
| export interface CliBackendScaffold { | ||
| type: 'backend'; | ||
| language: 'python'; | ||
| code: string; | ||
| } | ||
|
|
||
| /** | ||
| * `test scaffold` — emit a schema-correct starter test definition so a first | ||
| * test never starts from hand-copied JSON. Pure-local: no network, no | ||
| * credentials, no filesystem reads. The frontend template is a `CliPlanInput` | ||
| * (the exact shape `--plan-from` ingests; sourceRef: CliPlanInput / | ||
| * PLAN_STEP_TYPES above), so `scaffold | create --plan-from -`-style flows | ||
| * validate out of the box. The backend template is the minimal `requests` | ||
| * script the onboarding skill mandates: define a test function with a | ||
| * concrete status assertion, then CALL it (a defined-but-never-called test | ||
| * would pass without asserting anything). | ||
| */ | ||
| export async function runScaffold( | ||
| opts: ScaffoldOptions, | ||
| deps: TestDeps = {}, | ||
| ): Promise<CliPlanInput | CliBackendScaffold> { | ||
| const out = makeOutput(opts.output, deps); | ||
| const stderrFn = deps.stderr ?? ((line: string) => process.stderr.write(`${line}\n`)); | ||
| const env = deps.env ?? process.env; | ||
| // Pre-fill the project id from TESTSPRITE_PROJECT_ID when the caller's | ||
| // environment carries one; otherwise a clearly-marked placeholder the user | ||
| // swaps after running `testsprite project list`. | ||
| const projectId = | ||
| typeof env.TESTSPRITE_PROJECT_ID === 'string' && env.TESTSPRITE_PROJECT_ID.length > 0 | ||
| ? env.TESTSPRITE_PROJECT_ID | ||
| : '<run: testsprite project list>'; | ||
|
|
||
| let payload: CliPlanInput | CliBackendScaffold; | ||
| let body: string; | ||
| if (opts.scaffoldType === 'frontend') { | ||
| const plan: CliPlanInput = { | ||
| projectId, | ||
| type: 'frontend', | ||
| name: 'My first frontend test', | ||
| description: 'Replace with one sentence describing what this test verifies.', | ||
| priority: 'p2', | ||
| planSteps: [ | ||
| { | ||
| type: 'action', | ||
| description: 'Navigate to /login and sign in with a seeded test account', | ||
| }, | ||
| { type: 'action', description: 'Open the first product page and click "Add to cart"' }, | ||
| { type: 'assertion', description: 'Assert that the cart badge shows 1 item' }, | ||
| ], | ||
| }; | ||
| payload = plan; | ||
| body = `${JSON.stringify(plan, null, 2)}\n`; | ||
| } else { | ||
| const code = [ | ||
| 'import requests', | ||
| '', | ||
| '# Replace with your API base URL (must be reachable from the internet).', | ||
| 'BASE_URL = "https://staging.example.com"', | ||
| '', | ||
| '', | ||
| 'def test_health_endpoint() -> None:', | ||
| ' response = requests.get(f"{BASE_URL}/health", timeout=30)', | ||
| ' assert response.status_code == 200, f"expected 200, got {response.status_code}"', | ||
| '', | ||
| '', | ||
| '# The test function MUST be called: TestSprite executes this file top to', | ||
| '# bottom, so a defined-but-never-called function would pass vacuously.', | ||
| 'test_health_endpoint()', | ||
| '', | ||
| ].join('\n'); | ||
| payload = { type: 'backend', language: 'python', code }; | ||
| body = code; | ||
| } | ||
|
|
||
| if (opts.out !== undefined) { | ||
| const resolved = isAbsolute(opts.out) ? opts.out : resolve(process.cwd(), opts.out); | ||
| // Never clobber silently: scaffolds are starting points the user edits, so | ||
| // an accidental re-run must not erase their work. --force opts in. | ||
| if (!opts.force && existsSync(resolved)) { | ||
| throw localValidationError('out', `already exists: ${resolved}. Pass --force to overwrite`); | ||
| } | ||
| const sink = openOutputFile(opts.out); // reuses the directory/parent guards | ||
| const fileOut = makeFileOutput(opts.output, sink); | ||
| await fileOut.writeChunk(body); | ||
| await closeOutputFile(sink, true); | ||
| stderrFn(`Scaffold written to ${resolved}`); | ||
| return payload; | ||
| } | ||
|
|
||
| // No --out: the scaffold body IS the stdout payload (`> plan.json` works). | ||
| out.print(payload, () => body.trimEnd()); | ||
| return payload; | ||
| } | ||
|
|
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect makeFileOutput to see if it already serializes based on output mode
ast-grep run --pattern 'function makeFileOutput($_, $_) { $$$ }' --lang typescript src/lib/output.ts
rg -n -A 20 'function makeFileOutput' src/lib/output.tsRepository: TestSprite/testsprite-cli
Length of output: 163
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files 'src/**' | rg '(^|/)(output|test)\.(ts|js|tsx|jsx)$|src/commands/test\.ts|src/lib/dry-run/samples\.ts|src/lib/output\.ts'
echo
echo "== outlines =="
ast-grep outline src/lib/output.ts --view expanded || true
ast-grep outline src/commands/test.ts --view expanded || true
echo
echo "== locate file-output helpers =="
rg -n -A 40 -B 10 'function (makeFileOutput|openOutputFile|closeOutputFile)|const (makeFileOutput|openOutputFile|closeOutputFile)|class FileSink|interface FileSink' src/lib/output.ts src/commands/test.ts
echo
echo "== scaffold branch in test.ts =="
rg -n -A 80 -B 20 'runScaffold|scaffold written|makeFileOutput|openOutputFile|closeOutputFile|out.print\(' src/commands/test.tsRepository: TestSprite/testsprite-cli
Length of output: 50382
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== output.ts: Output class =="
sed -n '69,130p' src/lib/output.ts | cat -n
echo
echo "== test.ts: runScaffold =="
sed -n '3895,3970p' src/commands/test.ts | cat -n
echo
echo "== any tests for scaffold output =="
rg -n -A 6 -B 6 'runScaffold|CliBackendScaffold|test scaffold|Scaffold written|makeFileOutput' src --glob '*test*' --glob '*spec*'Repository: TestSprite/testsprite-cli
Length of output: 19925
--out should write the JSON envelope in --output json mode src/commands/test.ts:3959-3961
fileOut.writeChunk(body) always writes raw Python to the target, so testsprite test scaffold --type backend --output json --out foo skips the documented JSON envelope. Switch the file contents to payload/JSON.stringify(...) when opts.output === 'json'.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/commands/test.ts` around lines 3877 - 3971, The `runScaffold` backend
branch writes raw Python to `--out` even when `opts.output` is `json`, so the
file misses the documented JSON envelope. Update the `runScaffold` path that
builds `payload` and `body` so the `openOutputFile`/`fileOut.writeChunk` branch
writes `JSON.stringify(payload, null, 2)` (or equivalent JSON output) whenever
`opts.output === 'json'`, while preserving the current raw code body for
non-JSON output.
Source: Path instructions
Contributor
|
One more rebase, sorry — your own stack collided with itself: #132 and #176 merged first this round, moving the same |
… or backend skeleton
Andy00L
force-pushed
the
feat/test-scaffold
branch
from
6121944 to
925bab4
Compare
Contributor
Author
|
Rebased onto current main. Ready to merge. |
zeshi-du
approved these changes
Jul 6, 2026
zeshi-du
left a comment
zeshi-du
left a comment
Contributor
There was a problem hiding this comment.
Reviewed: scoped, tests included, CI green across the matrix.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
testsprite test scaffold— a first test today starts from hand-copiedJSON, and any shape mistake only surfaces as a paid create round-trip. This
emits a schema-correct starter: by default a frontend plan (the exact
CliPlanInputshape--plan-fromingests, derived from the source-of-truthtypes, not docs), or with
--type backendthe minimal Pythonrequestsskeleton the onboarding skill mandates (a status assertion, and the test
function is CALLED so it can never pass vacuously). Pure-local: no network,
no credentials.
--out <path>writes a file (refusing to clobber without--force); stdout otherwise, soscaffold > plan.jsonworks. The projectIdis pre-filled from TESTSPRITE_PROJECT_ID when present, else a placeholder
that names the command to run.
Related issue
Fixes #101
Type of change
Checklist
mainbranch.npm run lintandnpm run format:checkpass.npm run typecheckpasses.npm testpasses and coverage stays at or above the 80% gate.Notes for reviewers
The emitted step types come from the real
PLAN_STEP_TYPESenum via theCliPlanInputtype (a drift in the enum would fail compilation and theround-trip test). No overlap with
init/setup(configure + skill install)per the triage note. File writes reuse the existing
openOutputFileguards.Summary by CodeRabbit
New Features
test scaffoldCLI subcommand to generate starter test scaffolds.--output json.--type,--out, and--forceto control scaffold type and where the generatedplan.jsonis written.Bug Fixes
--force.--outwhen the target path is an existing directory unless--forceis provided.Tests
--out/overwrite behavior.