Run typespec-ts unit tests in the repo-wide vitest run#4788
Open
xirzec wants to merge 6 commits into
Open
Conversation
Remove the typespec-ts exclusion from the root vitest configs so the repo-wide run includes its unit tests (test-next + unit-modular), while keeping the Spector end-to-end integration project out of that run via a top-level unit-only include. Scenario paths in the modular-unit suite now resolve against the package root so they no longer depend on cwd. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
xirzec
requested review from
JialinHuang803,
MaryGao,
bterlson,
maorleger,
markcowl,
qiaozha and
timotheeguerin
as code owners
microsoft-github-policy-service
Bot
added
the
emitter:typescript
Contributor
|
All changed packages have been documented.
Show changes
|
commit: |
Contributor
⚡ Benchmark Results✅ No performance regressions detected. Full details – comparing
|
| Metric | Baseline | Current | Change |
|---|---|---|---|
| total | 🔴 877.2ms | 🔴 743.4ms | -15.2% 🟢 |
| loader | 🟡 259.8ms | 🟡 221.0ms | -14.9% 🟢 |
| resolver | 🟢 33.0ms | 🟢 32.2ms | -2.4% |
| checker | 🟡 309.9ms | 🟡 266.9ms | -13.9% 🟢 |
| validation | 🟢 72.0ms | 🟢 65.9ms | -8.4% 🟢 |
| ↳ validation/@azure-tools/typespec-azure-core | 🟡 10.5ms | 🟢 9.2ms | -11.9% 🟢 |
| ↳ validation/@typespec/http | 🟡 12.2ms | 🟡 11.3ms | -7.8% |
| ↳ validation/@typespec/rest | 🟢 1.3ms | 🟢 1.1ms | -11.3% |
| ↳ validation/@typespec/versioning | 🔴 44.7ms | 🔴 41.2ms | -7.8% 🟢 |
| ↳ validation/compiler | 🟢 3.2ms | 🟢 3.1ms | -3.1% |
| linter | 🟢 187.4ms | 🟢 157.3ms | -16.1% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/auth-required | 🟢 0.1ms | 🟢 0.1ms | +0.2% |
| ↳ linter/@azure-tools/typespec-azure-core/bad-record-type | 🟢 0.5ms | 🟢 0.5ms | -2.0% |
| ↳ linter/@azure-tools/typespec-azure-core/byos | 🟢 7.4ms | 🟢 6.2ms | -17.1% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/casing-style | 🟢 1.2ms | 🟢 1.1ms | -4.2% |
| ↳ linter/@azure-tools/typespec-azure-core/composition-over-inheritance | 🟢 0.1ms | 🟢 0.1ms | -1.9% |
| ↳ linter/@azure-tools/typespec-azure-core/documentation-required | 🟢 1.6ms | 🟢 1.5ms | -5.7% |
| ↳ linter/@azure-tools/typespec-azure-core/friendly-name | 🟢 1.1ms | 🟢 1.0ms | -7.9% |
| ↳ linter/@azure-tools/typespec-azure-core/key-visibility-required | 🟢 0.3ms | 🟢 0.3ms | -11.4% |
| ↳ linter/@azure-tools/typespec-azure-core/known-encoding | 🟢 0.4ms | 🟢 0.4ms | -10.9% |
| ↳ linter/@azure-tools/typespec-azure-core/long-running-polling-operation-required | 🟢 0.6ms | 🟢 0.6ms | -6.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-case-mismatch | 🟢 0.6ms | 🟢 0.5ms | -3.5% |
| ↳ linter/@azure-tools/typespec-azure-core/no-closed-literal-union | 🟢 0.8ms | 🟢 0.7ms | -3.6% |
| ↳ linter/@azure-tools/typespec-azure-core/no-enum | 🟢 0.2ms | 🟢 0.2ms | +9.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-error-status-codes | 🟢 0.2ms | 🟢 0.2ms | -5.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-explicit-routes-resource-ops | 🟢 0.1ms | 🟢 0.1ms | -14.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-format | 🟢 0.7ms | 🟢 0.6ms | -14.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-generic-numeric | 🟢 0.7ms | 🟢 0.7ms | -6.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-header-explode | 🔴 23.6ms | 🟡 18.6ms | -20.9% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/no-legacy-usage | 🟢 1.7ms | 🟢 1.6ms | -7.1% |
| ↳ linter/@azure-tools/typespec-azure-core/no-multiple-discriminator | 🟢 0.2ms | 🟢 0.2ms | -8.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-nullable | 🟢 0.4ms | 🟢 0.3ms | -10.3% |
| ↳ linter/@azure-tools/typespec-azure-core/no-offsetdatetime | 🟢 1.7ms | 🟢 1.5ms | -12.6% |
| ↳ linter/@azure-tools/typespec-azure-core/no-openapi | 🟢 2.2ms | 🟢 1.8ms | -17.6% |
| ↳ linter/@azure-tools/typespec-azure-core/no-private-usage | 🟢 2.7ms | 🟢 2.4ms | -9.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-query-explode | 🔴 24.7ms | 🟡 19.6ms | -20.7% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/no-response-body | 🔴 29.6ms | 🔴 24.0ms | -18.9% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces | 🟢 0.1ms | 🟢 0.1ms | +2.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch | 🟢 6.7ms | 🟢 5.3ms | -20.4% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/no-rpc-path-params | 🟢 0.3ms | 🟢 0.3ms | -10.8% |
| ↳ linter/@azure-tools/typespec-azure-core/no-string-discriminator | 🟢 0.1ms | 🟢 0.1ms | -5.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unknown | 🟢 0.3ms | 🟢 0.3ms | -11.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unnamed-union | 🟢 0.6ms | 🟢 0.6ms | -6.6% |
| ↳ linter/@azure-tools/typespec-azure-core/operation-missing-api-version | 🟢 0.3ms | 🟢 0.3ms | -5.6% |
| ↳ linter/@azure-tools/typespec-azure-core/request-body-problem | 🟢 0.4ms | 🟢 0.4ms | -9.2% |
| ↳ linter/@azure-tools/typespec-azure-core/require-versioned | 🟢 0.0ms | 🟢 0.0ms | +1.8% |
| ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem | 🔴 28.6ms | 🔴 23.7ms | -17.2% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/rpc-operation-request-body | 🟢 0.6ms | 🟢 0.5ms | -13.5% |
| ↳ linter/@azure-tools/typespec-azure-core/spread-discriminated-model | 🟢 0.4ms | 🟢 0.3ms | -10.9% |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-names | 🟢 6.7ms | 🟢 5.4ms | -19.5% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-operations | 🟢 0.2ms | 🟢 0.2ms | -10.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-child-resources | 🟡 15.3ms | 🟢 9.5ms | -38.1% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-agent-base-type-lifecycle-operations | 🟢 0.2ms | 🟢 0.2ms | -12.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version | 🟢 9.5ms | 🟢 8.6ms | -9.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-no-key | 🟢 0.2ms | 🟢 0.2ms | -0.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-usage-discourage | 🟢 0.1ms | 🟢 0.1ms | -4.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-delete-operation-response-codes | 🟢 4.4ms | 🟢 4.3ms | -3.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts | 🟡 13.9ms | 🟡 12.3ms | -11.5% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-record | 🟢 0.5ms | 🟢 0.5ms | -1.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes | 🟢 1.2ms | 🟢 1.4ms | +21.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-put-operation-response-codes | 🟢 0.2ms | 🟢 0.2ms | -0.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-action-no-segment | 🟢 0.4ms | 🟢 0.3ms | -7.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-duplicate-property | 🟢 0.3ms | 🟢 0.3ms | -3.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-interface-requires-decorator | 🟢 0.1ms | 🟢 0.1ms | +6.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-action-verb | 🟢 0.1ms | 🟢 0.1ms | -7.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-envelope-property | 🟢 0.2ms | 🟢 0.2ms | -3.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-version-format | 🟢 0.2ms | 🟢 0.2ms | +7.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-key-invalid-chars | 🟢 0.4ms | 🟢 0.4ms | -10.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-name-pattern | 🟢 0.1ms | 🟢 0.1ms | +9.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation | 🟢 0.4ms | 🟢 0.4ms | -0.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response | 🟢 7.5ms | 🟢 6.9ms | -7.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-patch | 🟢 0.7ms | 🟢 0.6ms | -7.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-path-segment-invalid-chars | 🟢 0.3ms | 🟢 0.3ms | -7.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-provisioning-state | 🟢 0.3ms | 🟢 0.4ms | +8.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/beyond-nesting-levels | 🟢 0.2ms | 🟢 0.2ms | -0.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/empty-updateable-properties | 🟢 0.3ms | 🟢 0.3ms | -4.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/improper-subscription-list-operation | 🟢 0.1ms | 🟢 0.1ms | +0.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header | 🟡 17.2ms | 🟡 13.5ms | -21.3% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-operations-endpoint | 🟢 0.1ms | 🟢 0.1ms | +4.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-x-ms-identifiers | 🟢 0.8ms | 🟢 0.8ms | +6.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-empty-model | 🟢 0.2ms | 🟢 0.2ms | -0.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-override-props | 🟢 0.2ms | 🟢 0.2ms | +3.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-resource-delete-operation | 🟢 0.4ms | 🟢 0.3ms | -8.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body | 🔴 25.9ms | 🔴 20.9ms | -19.3% 🟢 |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/patch-envelope | 🟢 0.3ms | 🟢 0.3ms | -6.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/resource-name | 🟢 0.3ms | 🟢 0.3ms | -5.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/secret-prop | 🟢 4.6ms | 🟢 4.2ms | -9.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/unsupported-type | 🟢 0.6ms | 🟢 0.5ms | -4.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/version-progression | 🟢 0.2ms | 🟢 0.2ms | +14.2% |
| ↳ linter/@azure-tools/typespec-client-generator-core/property-name-conflict | 🟢 1.9ms | 🟢 1.7ms | -9.0% |
| ↳ linter/@azure-tools/typespec-client-generator-core/require-client-suffix | 🟢 1.2ms | 🟢 1.3ms | +7.7% |
| emit | 🔴 5.71s | 🔴 4.45s | -22.1% 🟢 |
| ↳ emit/@Azure-Tools | 🟢 0.0ms | 🟢 0.0ms | +0.0% |
| ↳ emit/@azure-tools/typespec-autorest | 🟢 108.7ms | 🟢 95.2ms | -12.4% 🟢 |
| ↳ emit/@azure-tools/typespec-python | 🔴 2.15s | 🔴 1.65s | -23.5% 🟢 |
| ↳ emit/@typespec | 🟢 0.0ms | 🟢 0.0ms | +0.0% |
| ↳ emit/@typespec/http-client-js | 🔴 521.8ms | 🔴 430.1ms | -17.6% 🟢 |
| ↳ emit/@typespec/openapi3 | 🟢 94.8ms | 🟢 79.9ms | -15.7% 🟢 |
| ↳ emit/@typespec/openapi3/compute | 🟢 82.5ms | 🟢 69.5ms | -15.7% 🟢 |
| ↳ emit/@typespec/openapi3/write | 🟢 12.2ms | 🟢 10.2ms | -16.2% 🟢 |
Averaged across 3 specs (azure-arm-resource-manager, azure-core-dataplane, azure-full).
Threshold: changes > ±5% are highlighted.
🟢 Fast · 🟡 Moderate (stages >200ms, rules >10ms) · 🔴 Slow (stages >400ms, rules >20ms)
Contributor
|
You can try these changes here
|
Member
There was a problem hiding this comment.
Hhm it does seem like this unfortunately adds quite a bit of time to the CI no?
Before
This pr
which tests are those?
I do 100% want to find a shared solution across all our emitters here so we can run everything with common commands and its not a scavenger hunt when you touch multiple packages
Member
Author
There was a problem hiding this comment.
Ah yeah, looks like this is still pretty heavy. As an alternative, I'm going to add only our "test-next" tests (which take 14s locally to run) and as a compromise to avoid breaks from TCGC expand the TS-emitter workflow to run on tcgc changes specifically
Member
Author
There was a problem hiding this comment.
of course, a different PR (#4494) managed to break our tests... addressing that in a separate PR
In Vitest 4 poolOptions was removed; its sub-options are now top-level test.* fields. The nested poolOptions.forks.execArgv was silently ignored, so the --max-old-space-size cap was never applied. Hoist execArgv to the top level so it takes effect. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… tcgc changes The repo-wide vitest run exists as a cross-package regression net. Only the fast test-next suite (~14s) is included there; the heavy unit-modular suite (~150s of TypeSpec compiles) is kept out to keep the run fast. To preserve cross-package emit coverage, ci-typescript.yml now also triggers on typespec-client-generator-core (tcgc) changes, so a tcgc change runs the full typespec-ts unit + e2e suites. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
4 tasks
typespec-ts emits from the ARM library (workspace dep), so ARM lib changes can alter generated output (as #4494 did, silently breaking modular-unit snapshots). Add packages/typespec-azure-resource-manager to ci-typescript.yml triggers alongside tcgc so those cross-package emit regressions run the full typespec-ts unit + e2e suites. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
timotheeguerin
pushed a commit
to timotheeguerin/typespec-azure
that referenced
this pull request
Jul 2, 2026
…ure#4795) (Azure#4796) ## Why PR Azure#4494 ("Improve documentation for ARM library decorators, interfaces, and models") added `@doc("")` to the ARM `ArmResourceActionAsync` operation template, so those operations no longer emit a public doc comment. Because Azure#4494 did not touch `packages/typespec-ts/**`, `ci-typescript.yml` never ran on it, and the stale `modular-unit` scenario snapshot landed on `main` unnoticed. This made the **typespec-ts / CI -> Run unit tests** job red on `main` and on every typespec-ts PR (including Azure#4788). ## What changed - Updated the `bodyOptionalParameterName.md` modular-unit scenario to drop the now-empty ARM operation doc comment (`/** A long-running resource action. */`) and the corresponding sample summary text, matching what the current ARM library emits. - Added an `internal` chronus changeset for `@azure-tools/typespec-ts`. ## Notes for reviewers - The drift was narrower than the issue implied. Only `bodyOptionalParameterName.md` actually failed: its `backup` op uses `ArmResourceActionAsync` (which got `@doc("")`). `lroPaging.md` still passes because its `suspend` op uses `ArmResourceActionAsyncBase`, which Azure#4494 left with a public doc, so it correctly still emits `A long-running resource action.` - I did not commit the full `SCENARIOS_UPDATE=true` output. Write mode reformats every scenario through prettier and produced 46 changed files (+1039/-763), but 44 of those were pure formatting noise: the runner's assertion is whitespace-normalized (`ignoreWeirdLine`), so those never actually failed. I discarded the noise and hand-applied only the 3 genuine doc removals to keep the diff minimal and reviewable. ## Verification - `pnpm unit-test` -> 657 passed - `pnpm test-next` -> 252 passed - `pnpm chronus verify` -> clean - prettier `--check` on changed files -> clean ## Follow-up (out of scope) ARM library (`typespec-azure-resource-manager`) changes can alter typespec-ts emitted output but do not trigger `ci-typescript.yml`, so this drift class can recur. Consider widening the workflow `paths` triggers to include the emitter's TypeSpec library dependencies. Tracked separately. Fixes: Azure#4795 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: JialinHuang803 <139532647+JialinHuang803@users.noreply.github.com>
The test-next ts-morph integration tests (e.g. binder.test.ts) invoke the TypeScript type checker and sit near Vitest's 5s default even unloaded. In the repo-wide run every package's tests run concurrently, and under that pressure one binder test exceeded 5s and failed on the Windows CI runner. The repo-wide run uses this file's top-level test config (nested projects are ignored there), which had no testTimeout. Set a 10s timeout (double the default) at the top level and on the test-next project so it holds whether run repo-wide or standalone, without masking a genuine hang. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Removes the
typespec-tsexclusion from the rootvitest.config.ts/vitest.config.fast.tsso the repo-wide vitest run now includes the typespec-ts unit tests (test-next+unit-modular), while keeping the end-to-end Spector integration project (integration-azure-modular) out of that run.Why it wasn't this simple
Two things had to be solved:
Vitest flattens but ignores nested projects. When the root config references
packages/typespec-ts/vitest.config.tsas a project, vitest 4 ignores that file's nestedtest.projectsand instead uses its top-leveltest.include. So I added a top-levelincludelisting only the unit globs, plus theunit-modularpool/timeout settings (testTimeout: 0, forked pool, 1 GB heap) at the top level. The three named projects remain for local--projectselection (pnpm test-next,pnpm unit-test,pnpm integration-test:alone).The real reason for the original exclusion: the generated
modular-unitscenario suites resolved scenario directories relative toprocess.cwd(), so they only worked when cwd waspackages/typespec-ts. They now resolve against the package root (viaimport.meta.url) inscenario-runner.tsandscenario-suites.guard.test.ts, making them cwd-independent.Validation
vitest.config.fast.tsruns the typespec-ts modular-unit suite (657 tests) — all pass.pnpm test-next(252) andpnpm unit-test(657) still pass; the integration project still resolves forpnpm integration-test:alone.gen:scenario-suitesoutput unchanged; lint,tsc, prettier, and chronus verify all pass.The dedicated
ci-typescript.ymlworkflow is intentionally left as-is (fast path-filtered unit feedback + the e2e Spector job).