Skip to content

test(cli-e2e): add E2E testing package and CI workflow#870

Merged
binoy14 merged 1 commit into
mainfrom
binoy14/cli-testing-tools-investigation-integration-report
Apr 16, 2026
Merged

test(cli-e2e): add E2E testing package and CI workflow#870
binoy14 merged 1 commit into
mainfrom
binoy14/cli-testing-tools-investigation-integration-report

Conversation

@binoy14
Copy link
Copy Markdown
Contributor

@binoy14 binoy14 commented Apr 1, 2026

Summary

This PR builds the core E2E testing infrastructure for the Sanity CLI, with real PTY support for interactive tests.

E2E Package (@sanity/cli-e2e)

  • New private workspace package for E2E testing the Sanity CLI
  • Tests against the packed publishable artifact via pnpm pack in vitest global setup, ensuring the files field, lifecycle scripts, and bin entries are correct
  • Runs separately via pnpm test:e2e (not included in pnpm test)
  • CI workflow (.github/workflows/e2e.yml) with path-based filtering and Node 20/22/24 matrix

Dual-mode process spawning

  • runCli() helper with TypeScript overloads: interactive: false (default) returns {stdout, stderr, exitCode, error}, interactive: true returns InteractiveSession
  • Non-interactive mode: child_process.spawn with separate stdout/stderr streams
  • Interactive mode: node-pty for real PTY allocation (isTTY === true), enabling @inquirer/prompts to render
  • Clean abstraction — no node-pty types leak to consumers

InteractiveSession API

  • waitForText(pattern, opts?) — polls ANSI-stripped output for regex match
  • sendKey(key) — sends terminal escape sequences (Enter, ArrowUp, etc.)
  • sendControl(char) — sends Ctrl+key combos (e.g., Ctrl+C)
  • write(text) — raw stdin writes
  • waitForExit(timeout?) / kill() — process lifecycle
  • Auto-cleanup via vitest onTestFinished

Initial test coverage

  • sanity --help and sanity datasets list (authenticated + unauthenticated scenarios)

Closes SDK-1188
Closes SDK-1190
Closes SDK-1194
Closes SDK-1196

🤖 Generated with Claude Code

@binoy14 binoy14 requested a review from a team as a code owner April 1, 2026 18:28
@binoy14 binoy14 requested review from rexxars and removed request for a team April 1, 2026 18:28
@binoy14 binoy14 changed the title feat(cli-e2e): create @sanity/cli-e2e package scaffold feat(cli-e2e): create @sanity/cli-e2e package with renderCli() helper Apr 1, 2026
Copy link
Copy Markdown
Contributor Author

binoy14 commented Apr 1, 2026

@claude
Copy link
Copy Markdown

claude Bot commented Apr 1, 2026

Claude finished @binoy14's task in 2m 15s —— View job


Tasks

  • Gather context
  • Read changed files
  • Check for changeset
  • Provide review feedback

Review

Changeset: Not required — this is a test-only PR with no public API/behavior changes. Correctly omitted.


Issues

1. readEnv('SANITY_E2E_TOKEN') throws for all tests, including unauthenticated ones

packages/@sanity/cli-e2e/helpers/runCli.ts:43

readEnv is called unconditionally, which means even tests that pass env: {SANITY_AUTH_TOKEN: ''} to simulate no-auth scenarios will throw at construction time if SANITY_E2E_TOKEN is not set in the environment. This makes it impossible to run the unauthenticated test cases locally without the token.

Fix: use a fallback instead of throwing:

SANITY_AUTH_TOKEN: process.env.SANITY_E2E_TOKEN ?? '',

Then add a separate guard in getE2EProjectId() or in the authenticated test suite setup if the token is truly required there. Fix this →


2. pnpm pack may output just the filename, not a full path

packages/@sanity/cli-e2e/helpers/packCli.ts:17-31

The variable is named tarballName and the extracted value (tgzLine) is returned as if it's a full path. If pnpm pack --pack-destination <dir> outputs only the filename (e.g. sanity-cli-3.x.x.tgz) rather than the absolute path, the subsequent npm install ... "${tarballPath}" call will fail because npm won't find the file. Verify that pnpm pack with --pack-destination outputs the full path, or build it explicitly:

const tgzFilename = lines.findLast((line) => line.endsWith('.tgz'))
return join(packDest, tgzFilename)

3. sendControl does not validate its argument

packages/@sanity/cli-e2e/helpers/spawnPty.ts:85

const code = char.toLowerCase().codePointAt(0)! - 96

This silently produces nonsensical control codes for non-alphabetic characters (e.g. sendControl('1') → code point 49 - 96 = -47). Add a guard:

const code = char.toLowerCase().codePointAt(0)! - 96
if (code < 1 || code > 26) {
  throw new Error(`sendControl: expected a letter a-z, got "${char}"`)
}

4. .env.example is missing SANITY_E2E_DATASET

packages/@sanity/cli-e2e/.env.example

The CI workflow passes SANITY_E2E_DATASET as a secret and the fixture sanity.cli.ts reads it, but the .env.example only documents SANITY_E2E_TOKEN and SANITY_E2E_PROJECT_ID. Developers running locally won't know to set it.


5. actions/checkout@v6 — verify this tag exists

.github/workflows/e2e.yml:23,51

The latest stable actions/checkout tag at the time of writing is v4. Using @v6 will cause the workflow to fail with "Unable to resolve action". Confirm this is intentional or update to a known-good version.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 1, 2026

📦 Bundle Stats — @sanity/cli

Compared against main (b27d5327)

@sanity/cli

Metric Value vs main (b27d532)
Internal (raw) 2.1 KB -
Internal (gzip) 799 B -
Bundled (raw) 10.94 MB -
Bundled (gzip) 2.05 MB -
Import time 838ms -7ms, -0.8%

bin:sanity

Metric Value vs main (b27d532)
Internal (raw) 975 B -
Internal (gzip) 460 B -
Bundled (raw) 9.84 MB -
Bundled (gzip) 1.77 MB -
Import time 1.98s -7ms, -0.4%

🗺️ View treemap · Artifacts

Details
  • Import time regressions over 10% are flagged with ⚠️
  • Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.

📦 Bundle Stats — @sanity/cli-core

Compared against main (b27d5327)

Metric Value vs main (b27d532)
Internal (raw) 93.1 KB -
Internal (gzip) 21.9 KB -
Bundled (raw) 21.62 MB -
Bundled (gzip) 3.42 MB -
Import time 793ms +0ms, +0.0%

🗺️ View treemap · Artifacts

Details
  • Import time regressions over 10% are flagged with ⚠️
  • Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.

📦 Bundle Stats — create-sanity

Compared against main (b27d5327)

Metric Value vs main (b27d532)
Internal (raw) 976 B -
Internal (gzip) 507 B -
Bundled (raw) 50.7 KB -
Bundled (gzip) 12.6 KB -
Import time ❌ ChildProcess denied: node -
Details
  • Import time regressions over 10% are flagged with ⚠️
  • Sizes shown as raw / gzip 🗜️. Internal bytes = own code only. Total bytes = with all dependencies. Import time = Node.js cold-start median.

@binoy14 binoy14 marked this pull request as draft April 1, 2026 18:28
@socket-security
Copy link
Copy Markdown

socket-security Bot commented Apr 1, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addednpm/​node-pty@​1.1.09310010092100

View full report

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 1, 2026

Coverage Delta

No covered files changed in this PR.

Overall Coverage

Metric Coverage
Statements 83.0% (±0%)
Branches 73.0% (±0%)
Functions 82.9% (±0%)
Lines 83.4% (±0%)

@binoy14 binoy14 marked this pull request as ready for review April 1, 2026 18:43
@github-actions

This comment was marked as outdated.

@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch from b187a49 to c0b713b Compare April 1, 2026 19:10
@binoy14 binoy14 marked this pull request as draft April 1, 2026 19:11
@github-actions

This comment was marked as outdated.

@binoy14
Copy link
Copy Markdown
Contributor Author

binoy14 commented Apr 2, 2026

@claude add SANITY_E2E_PROJECT_ID to turbo.json global env vars

@squiggler-app
Copy link
Copy Markdown
Contributor

squiggler-app Bot commented Apr 2, 2026

Claude finished @binoy14's task in 36s —— View job


Adding SANITY_E2E_PROJECT_ID to turbo.json global env vars

  • Gather context
  • Find and update turbo.json
  • Commit and push changes

Added SANITY_E2E_PROJECT_ID to the globalEnv array in turbo.json (line 4). This ensures Turbo includes it in cache key calculations so CI cache misses correctly when the project ID changes.

@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch 3 times, most recently from 5091d84 to e640b79 Compare April 2, 2026 15:33
@github-actions

This comment was marked as outdated.

@binoy14 binoy14 changed the title feat(cli-e2e): create @sanity/cli-e2e package with renderCli() helper feat(cli-e2e): create @sanity/cli-e2e package with runCli() helper and CI workflow Apr 2, 2026
@binoy14 binoy14 changed the title feat(cli-e2e): create @sanity/cli-e2e package with runCli() helper and CI workflow feat(cli-e2e): add E2E testing package and CI workflow Apr 2, 2026
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch 4 times, most recently from db1dca6 to 7f876fd Compare April 2, 2026 17:22
@binoy14 binoy14 marked this pull request as ready for review April 2, 2026 23:37
@binoy14 binoy14 marked this pull request as draft April 2, 2026 23:38
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch from 18d29e1 to a7a20ce Compare April 3, 2026 15:10
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch from a7a20ce to a59e92b Compare April 6, 2026 17:38
@binoy14 binoy14 marked this pull request as ready for review April 7, 2026 15:07
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch from b1a21a6 to 8b9c074 Compare April 7, 2026 17:46
@binoy14 binoy14 marked this pull request as draft April 7, 2026 20:31
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch from 8b9c074 to a0ac2b5 Compare April 7, 2026 21:54
@socket-security
Copy link
Copy Markdown

socket-security Bot commented Apr 7, 2026

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action Severity Alert  (click "▶" to expand/collapse)
Warn Medium
Install-time scripts: npm node-pty during postinstall

Install script: postinstall

Source: node scripts/post-install.js

From: packages/@sanity/cli-e2e/package.jsonnpm/node-pty@1.1.0

ℹ Read more on: This package | This alert | What is an install script?

Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev.

Suggestion: Packages should not be running non-essential scripts during install and there are often solutions to problems people solve with install scripts that can be run at publish time instead.

Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/node-pty@1.1.0. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report

@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch 2 times, most recently from 1e64481 to 45fc3ca Compare April 8, 2026 20:40
squiggler-app Bot pushed a commit that referenced this pull request Apr 8, 2026
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch from 8a50948 to 6448c9f Compare April 10, 2026 14:29
@binoy14 binoy14 marked this pull request as ready for review April 10, 2026 15:14
binoy14 pushed a commit that referenced this pull request Apr 10, 2026
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch 2 times, most recently from b648ee4 to 314ecf4 Compare April 15, 2026 02:22
@binoy14 binoy14 requested a review from mttdnt April 15, 2026 02:24
@binoy14 binoy14 changed the title feat(cli-e2e): add E2E testing package and CI workflow test(cli-e2e): add E2E testing package and CI workflow Apr 15, 2026
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch 2 times, most recently from df7e11e to 47d5997 Compare April 16, 2026 17:09
Add a new @sanity/cli-e2e package for end-to-end testing of the Sanity CLI.

The package uses pnpm pack to create a tarball of @sanity/cli, installs it
into a temp directory via npm, then runs tests against the real binary. This
ensures E2E tests exercise the actual published distribution.

Includes:
- globalSetup that packs and installs the CLI tarball
- runCli() helper wrapping cli-testing-library's render()
- Two initial tests: datasets list and --help
- GitHub Actions E2E workflow
- Vite compatibility fixes, env loading, and test isolation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@binoy14 binoy14 force-pushed the binoy14/cli-testing-tools-investigation-integration-report branch from 47d5997 to 1b1aea7 Compare April 16, 2026 17:51
Copy link
Copy Markdown
Contributor

@mttdnt mttdnt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, tested it out locally and the testing api makes sense to me. Just leaving a couple questions,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one question if there was a pattern in mind for needing to find a specific value in a list and select. Is the idea to keep using sendKey until a certain value. IE finding the right project in a project select prompt.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not yet but maybe it might be needed at one point. Keeping the API surface minimal till it's actually needed is my thinking

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the dataset env to the example?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might not be needed, I might have to tweak this as I write more tests

@binoy14 binoy14 merged commit 9a7e204 into main Apr 16, 2026
56 checks passed
@binoy14 binoy14 deleted the binoy14/cli-testing-tools-investigation-integration-report branch April 16, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants