Skip to content

test(e2e): add E2E test suite for CLI v1.0#36699

Closed
gigglewang0417 wants to merge 2 commits into
mainfrom
feat/cli-e2e-test-suite
Closed

test(e2e): add E2E test suite for CLI v1.0#36699
gigglewang0417 wants to merge 2 commits into
mainfrom
feat/cli-e2e-test-suite

Conversation

@gigglewang0417
Copy link
Copy Markdown
Contributor

Covers auth, config, run, and CLI framework scenarios against a live staging server using vitest + real difyctl binary.

Suite layout:
test/e2e/
├── helpers/
│ ├── cli.ts — run(), withAuthFixture(), mintFreshToken()
│ ├── assert.ts — assertExitCode, assertJson, assertErrorEnvelope
│ ├── cleanup-registry.ts — staging data teardown
│ └── retry.ts — withRetry() for flaky network assertions
├── setup/
│ ├── global-setup.ts — health-check, disposable token mint
│ └── global-teardown.ts — conversation cleanup
└── suites/
├── auth/ — status, use, whoami, devices, logout
├── config/ — path, get/set/unset/view, env override
└── run/ — basic, streaming, conversation, file, HITL

Key design decisions:

  • Each test uses an isolated temp configDir via withAuthFixture()
  • Logout and devices-revoke tests run last to avoid invalidating the shared E.token used by all other suites
  • mintFreshToken() mints a disposable dfoa_ token on demand via the device flow API so revoke tests never touch the primary session
  • Global retry is 0; flaky network calls use withRetry() locally
  • test:e2e:smoke script filters to [P0] cases via testNamePattern

package.json: add test:e2e / test:e2e:smoke / test:e2e:local scripts
.gitignore: exclude .env.e2e, oclif.manifest.json, tmp/
.env.e2e.example: credential template for local setup

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Screenshots

Before After
... ...

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

Covers auth, config, run, and CLI framework scenarios against a live
staging server using vitest + real difyctl binary.

Suite layout:
  test/e2e/
  ├── helpers/
  │   ├── cli.ts          — run(), withAuthFixture(), mintFreshToken()
  │   ├── assert.ts       — assertExitCode, assertJson, assertErrorEnvelope
  │   ├── cleanup-registry.ts — staging data teardown
  │   └── retry.ts        — withRetry() for flaky network assertions
  ├── setup/
  │   ├── global-setup.ts — health-check, disposable token mint
  │   └── global-teardown.ts — conversation cleanup
  └── suites/
      ├── auth/           — status, use, whoami, devices, logout
      ├── config/         — path, get/set/unset/view, env override
      └── run/            — basic, streaming, conversation, file, HITL

Key design decisions:
- Each test uses an isolated temp configDir via withAuthFixture()
- Logout and devices-revoke tests run last to avoid invalidating
  the shared E.token used by all other suites
- mintFreshToken() mints a disposable dfoa_ token on demand via the
  device flow API so revoke tests never touch the primary session
- Global retry is 0; flaky network calls use withRetry() locally
- test:e2e:smoke script filters to [P0] cases via testNamePattern

package.json: add test:e2e / test:e2e:smoke / test:e2e:local scripts
.gitignore: exclude .env.e2e, oclif.manifest.json, tmp/
.env.e2e.example: credential template for local setup
@gigglewang0417 gigglewang0417 requested a review from a team as a code owner May 27, 2026 02:13
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. javascript Pull requests that update javascript code labels May 27, 2026
Comment thread .github/workflows/cli-e2e.yml Fixed
Triggers on pull_request when cli/** files change.
Spins up a full Dify stack via docker compose, provisions
an admin account + test apps, then runs the E2E smoke suite
(P0 cases only for CI speed).
@gigglewang0417 gigglewang0417 force-pushed the feat/cli-e2e-test-suite branch from 66294a2 to b734afd Compare May 27, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

javascript Pull requests that update javascript code size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants