chore(tests-only): migrate to auto-sharding/balancing Vitest CI workflow

[mmaietta]

📣 Context/Reasoning for this (large) change

• It turns out that not all the test suite was being run in the CI as the test matrix is manually assigned via TEST_FILES env var.
• This PR resolves that through its auto-scaling + auto-sharding implementation.

Notes/Disclaimer:

• I wrote the code, but the PR description heavily leveraged ClaudeAI to make it informative + comprehensive.
• Test-only changes. Zero production code is modified.

🎯 Summary

This PR fundamentally transforms the test infrastructure from a static, manually-maintained system to a dynamic, self-optimizing system using dynamic shard planning based on historical test execution data, replacing static test matrices with an intelligent test file distribution across platforms.

🔍 Key Objectives:

1. Eliminated Manual Test Distribution

   • Removed hardcoded testFiles arrays from GitHub workflows
   • Tests now automatically distributed based on historical performance

2. Intelligent Sharding

   • Per-platform shard calculation (Linux, Windows, macOS)
   • Bin-packing algorithm/approach ensures balanced shard execution times
   • Adapts to test suite changes and file additions automatically

3. Performance Optimization

   • Smart cache tracks test durations and failures
   • Fast tests run first (fail-fast optimization)
   • Heavy tests use mutex to prevent resource contention

4. CI/CD Simplification

   • Reduced from 5 test jobs to 3 unified jobs
   • Temporarily increased timeouts while the CI balances out the sharding pattern
   • Consolidated setup leveraging reusable GH actions

5. Cache Management

   • Hierarchical cache restoration (PR-specific → branch → base)
   • Automatic cache merging across all test shards
   • Persistent GH cache storage for continuous optimization

6. Infrastructure Cleanup

   • Removed obsolete certificate files
   • Reorganized test directory structure
   • Updated test files for new system (including adding describe.ifMac ifWindows and ifLinux operators so that sharding works correctly across each OS)

---

🔧 GitHub Actions & Workflows Changes

.github/actions/pretest/action.yml (Major refactor)

Added (new approach):

• Single consolidated step: uses: ./.github/actions/pnpm to reduce duplicated GH Action logic
• Vitest Smart Cache restoration with hierarchical fallback:

  key: vitest-smart-cache-${{ github.ref_name }}-${{ github.sha }}
  restore-keys: vitest-smart-cache-${{ github.ref_name }}-

• Cache status logging to show whether cache was restored, repo-committed file used, or new cache will be created

.github/workflows/test.yaml (Complete restructure)

New Workflow Structure:

1. Shard Planning Job (shard-plan):

runs-on: ubuntu-latest
outputs:
  linux: ${{ steps.plan.outputs.linux }}
  windows: ${{ steps.plan.outputs.windows }}
  macos: ${{ steps.plan.outputs.macos }}

• Runs pnpm --silent ci:test:count to get JSON output with shard counts per platform
• Extracts platform-specific arrays using jq:
  • LINUX_SHARDS=$(echo "$SHARD_JSON" | jq -c '.linux')
  • WINDOWS_SHARDS=$(echo "$SHARD_JSON" | jq -c '.win32')
  • MACOS_SHARDS=$(echo "$SHARD_JSON" | jq -c '.darwin')
• Outputs JSON arrays that GitHub Actions can iterate over

2. Unified Test Jobs:

Key Changes:

• Removed TEST_FILES environment variable entirely
• Added VITEST_SHARD_INDEX for dynamic shard assignment
• Each job uploads smart cache artifact after completion
• macOS job removed brew install powershell/tap/powershell and wine-stable (only keeps rpm)

3. Special Jobs (Modified):

test-updater:

• Still runs separately (needs API tokens)
• Now uploads smart cache artifact
• Uses ADDITIONAL_DOCKER_ARGS to pass tokens

test-e2e:

• Unchanged functionality
• Now uploads smart cache artifact

test-linux-native:

• NEW job (currently disabled with if: false due to unstable tests [upcoming fix PR])
• For native Linux testing without Docker

4. Cache Merging Job (merge-smart-cache):

needs: [test-linux, test-windows, test-macos, test-e2e, test-updater, test-linux-native]
if: always()

• Downloads all vitest-smart-cache-* artifacts
• Merges them using Node.js script that:
  • Starts with repo-committed cache as base
  • Merges all test run caches (keeping most recent timestamp)
  • Deletes individual artifacts
  • Saves merged cache for future runs with hierarchical keys

---

📁 vitest-scripts Folder (New CI infrastructure)

test/vitest-scripts/_vitest-smart-cache.json storing test execution metadata

• Tracks historical test performance across all platforms

test/vitest-scripts/cache.ts

• Handles read/write operations for smart cache
• Implements cache merging logic (timestamp-based conflict resolution)
• Validates cache integrity

test/vitest-scripts/file-discovery.ts (63 lines)

• Scans filesystem for test files
• Filters by platform (Linux/Windows/macOS specific tests)
• Categorizes by test type (heavy/platform/etc.) via Vitest meta task metadata property

test/vitest-scripts/shard-builder.ts

• Distributes tests across shards based on historical duration
• Implements bin-packing algorithm for balanced shards
• Accounts for platform-specific test requirements

test/vitest-scripts/smart-config.ts

• Defines shard strategy parameters
• Platform-specific configuration (different weights for Linux/Win/Mac)
• Tunable duration thresholds for shard optimization

test/vitest-scripts/smart-shard-count.ts

• Analyzes test suite characteristics
• Outputs JSON corresponding to shard count and shard index: {"linux": [1,2,3], "win32": [1,2], "darwin": [1,2,3,4]}
• Used by GitHub Actions to create matrix by using shard index w/ VITEST_SHARD_INDEX env var

test/vitest-scripts/smart-shard.ts

• Reads VITEST_SHARD_INDEX env var
• Loads appropriate shard from cache
• Executes Vitest with filtered test files

test/vitest-scripts/vitest-heavy-mutex.ts

• Prevents resource-intensive tests from running concurrently
• Implements file-based locking mechanism
• Ensures system stability during heavy tests (e.g., namely CPU-heavy tests: auto-update flows, fpm, and dmg to name a few)

test/vitest-scripts/vitest-setup.ts

• Configures Vitest environment
• Sets up global test utilities
• Initializes smart cache system and custom chainable

test/vitest-scripts/vitest-smart-reporter.ts

• Custom Vitest reporter
• Captures test execution metrics (duration, failures, retries)
• Feeds data back to smart cache
• Generates performance reports

test/vitest-scripts/vitest-smart-sequencer.ts

• Controls test execution order based on historical data
• Runs heavy tests first (frontload to prevent overloading CPU usage)
• Implements intelligent sharding logic

[changeset-bot]

⚠️ No Changeset found

Latest commit: 69f3764

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[Copilot]

Pull request overview

This PR transforms the test infrastructure from a static, manually-maintained test matrix to a dynamic auto-sharding system with intelligent test distribution based on historical performance data.

Changes:

• Introduced smart cache system that tracks test execution metrics (duration, failures) per platform
• Implemented dynamic shard planning with bin-packing algorithm for balanced test distribution across CI runners
• Added custom Vitest infrastructure including sequencer, reporter, and heavy test mutex for resource management

Reviewed changes

Copilot reviewed 74 out of 78 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
vite.config.ts	Updated Vitest configuration to use custom sequencer and reporters, removed HTTPS cert setup
test/vitest-scripts/*	New smart test infrastructure for sharding, caching, and platform-specific test discovery
.github/workflows/test.yaml	Restructured workflow with unified shard planning job and platform-specific test matrices
.github/actions/pretest/action.yml	Consolidated setup using reusable pnpm action, added smart cache restoration
test/src/**/*Test.ts	Updated tests to use new conditional APIs (ifMac, ifWindows, ifLinux, heavy)
package.json	Updated CI test commands to use new shard scripts

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	typedoc-plugin-markdown@​4.2.8
	typescript-json-schema@​0.64.0
	typescript@​5.8.2
	typedoc@​0.26.11

View full report

[socket-security]

Warning

Review the following alerts detected in dependencies.

According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.

Action	Severity	Alert  (click "▶" to expand/collapse)
Warn		Obfuscated code: npm typedoc is 98.0% likely obfuscated Confidence: 0.98 Location: Package overview From: package.json → npm/typedoc@0.26.11 ℹ Read more on: This package | This alert | What is obfuscated code? Next steps: Take a moment to review the security alert above. Review the linked package source code to understand the potential risk. Ensure the package is not malicious before proceeding. If you're unsure how to proceed, reach out to your security team or ask the Socket team for help at support@socket.dev. Suggestion: Packages should not obfuscate their code. Consider not using packages with obfuscated code. Mark the package as acceptable risk. To ignore this alert only in this pull request, reply with the comment @SocketSecurity ignore npm/typedoc@0.26.11. You can also ignore all packages with @SocketSecurity ignore-all. To ignore an alert for all future pull requests, use Socket's Dashboard to change the triage state of this alert.

View full report