Skip to content

Switch GitHub CI to C++ test runner (#1963)#1963

Open
lavenzg wants to merge 13 commits intofacebook:static_hfrom
lavenzg:export-D98531546
Open

Switch GitHub CI to C++ test runner (#1963)#1963
lavenzg wants to merge 13 commits intofacebook:static_hfrom
lavenzg:export-D98531546

Conversation

@lavenzg
Copy link
Copy Markdown
Contributor

@lavenzg lavenzg commented Mar 27, 2026 •
edited by meta-codesync bot
Loading

Summary:

Replace the Python test_runner.py invocation with the C++ test-runner
binary in the GitHub Actions CI workflow for macOS, Linux, and Linux
ARMv7 jobs. The C++ runner is ~8x faster and runs tests in-process.

Key changes:

  • macOS/Linux: use build/bin/test-runner with --test-intl and --skiplist
  • JIT matrix variant: replace --vm-args='-Xjit=force' with --jit=force
  • ARMv7: same as Linux, no JIT variant
  • Windows: kept on Python runner (C++ runner uses POSIX signal APIs
    for crash isolation that aren't available on Windows)

Differential Revision: D98531546

@meta-cla meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Mar 27, 2026
@meta-codesync
Copy link
Copy Markdown

meta-codesync bot commented Mar 27, 2026

@lavenzg has exported this pull request. If you are a Meta employee, you can view the originating Diff in D98531546.

lavenzg added a commit to lavenzg/hermes that referenced this pull request Mar 27, 2026
@lavenzg @meta-codesync
Switch GitHub CI to C++ test runner (facebook#1963)
41e1b4c 
Summary:
Pull Request resolved: facebook#1963

Replace the Python test_runner.py invocation with the C++ test-runner
binary in the GitHub Actions CI workflow for macOS, Linux, and Linux
ARMv7 jobs. The C++ runner is ~8x faster and runs tests in-process.

Key changes:
- macOS/Linux: use build/bin/test-runner with --test-intl and --skiplist
- JIT matrix variant: replace --vm-args='-Xjit=force' with --jit=force
- ARMv7: same as Linux, no JIT variant
- Windows: kept on Python runner (C++ runner uses POSIX signal APIs
  for crash isolation that aren't available on Windows)

Differential Revision: D98531546
@meta-codesync meta-codesync bot changed the title Switch GitHub CI to C++ test runner Switch GitHub CI to C++ test runner (#1963) Mar 27, 2026
lavenzg added a commit to lavenzg/hermes that referenced this pull request Mar 27, 2026
@lavenzg @meta-codesync
Switch GitHub CI to C++ test runner (facebook#1963)
96fe60d 
Summary:
Pull Request resolved: facebook#1963

Replace the Python test_runner.py invocation with the C++ test-runner
binary in the GitHub Actions CI workflow for macOS, Linux, and Linux
ARMv7 jobs. The C++ runner is ~8x faster and runs tests in-process.

Key changes:
- macOS/Linux: use build/bin/test-runner with --test-intl and --skiplist
- JIT matrix variant: replace --vm-args='-Xjit=force' with --jit=force
- ARMv7: same as Linux, no JIT variant
- Windows: kept on Python runner (C++ runner uses POSIX signal APIs
  for crash isolation that aren't available on Windows)

Differential Revision: D98531546
lavenzg added a commit to lavenzg/hermes that referenced this pull request Mar 27, 2026
@lavenzg @meta-codesync
Switch GitHub CI to C++ test runner (facebook#1963)
01f8062 
Summary:
Pull Request resolved: facebook#1963

Replace the Python test_runner.py invocation with the C++ test-runner
binary in the GitHub Actions CI workflow for macOS, Linux, and Linux
ARMv7 jobs. The C++ runner is ~8x faster and runs tests in-process.

Key changes:
- macOS/Linux: use build/bin/test-runner with --test-intl and --skiplist
- JIT matrix variant: replace --vm-args='-Xjit=force' with --jit=force
- ARMv7: same as Linux, no JIT variant
- Windows: kept on Python runner (C++ runner uses POSIX signal APIs
  for crash isolation that aren't available on Windows)

Differential Revision: D98531546
zhaogang and others added 13 commits March 30, 2026 12:37
@meta-codesync
Phase 1: C++ project skeleton & CLI
415926d 
Summary:
Add C++ test runner project that links hermesvm_a for in-process execution.
CLI supports -j, --timeout, --show-slowest-tests, --dump-source flags.
Build: ninja test-runner in cmake-build-release.

Differential Revision: D98522855
@meta-codesync
Phase 2: Test discovery & skiplist
54d832c 
Summary:
Add test file discovery and skiplist filtering:

TestDiscovery.h:
- Recursive directory traversal finding .js files
- Suite detection (test262, mjsunit, CVEs, esprima, flow) via rfind markers
- Excludes test262 _FIXTURE.js files
- Sorted output for deterministic ordering

Skiplist.h:
- Parses utils/testsuite/skiplist.json using Hermes JSONParser
- Supports all categories: manual_skip_list, skip_list, lazy_skip_list,
  permanent_skip_list, handlesan_skip_list, intl_tests, platform_skip_list
- Supports unsupported_features and permanent_unsupported_features
- Handles both bare string and {paths, comment} entry formats
- Substring matching for path-based skipping (matches Python/Rust behavior)

main.cpp:
- Wires discovery and skiplist into CLI pipeline
- Auto-detects skiplist.json by walking up from test paths
- Prints Python-compatible output: "-- Testing: N tests, max J concurrent tasks --"

Differential Revision: D98522850
@meta-codesync
Phase 3: Frontmatter parsing & source preprocessing
16fae15 
Summary:
Add YAML frontmatter parser (Frontmatter.h) that extracts test262
metadata (flags, features, includes, negative expectations) and strips
frontmatter + license headers from test source.

Add harness file cache (HarnessCache.h) that loads all .js files from
test262/harness/ into memory once, then assembles complete test source
by prepending harness includes with optional 'use strict;' prefix.

Implement --dump-source mode in main.cpp: parses frontmatter, loads
harness, assembles source with correct strict/non-strict variants
(onlyStrict, noStrict, raw, module flags), and prints to stdout.

Auto-detects test262 suite root from test paths for harness loading.
Feature-based skiplist filtering applied during dump-source.

Differential Revision: D98522852
@lavenzg @meta-codesync
Phase 4: In-process execution engine
2439837 
Summary:
Add the core test execution engine (Executor.h/cpp) and integrate it
into main.cpp with result tallying and failure reporting.

Executor.h: Public API declarations:
- ResultCode enum with granular failure categories: CompileFailed,
  CompileTimeout, ExecuteFailed, ExecuteTimeout, PermanentlySkipped
- resultCodeName() for display strings
- TestResult struct with test name, result code, message, and duration
- ExecConfig struct for thread count and timeout configuration
- compileSource(): compile JS to bytecode via BCProviderFromSrc
- executeTestVariant(): compile + run a single test variant
- WorkQueue: thread-safe FIFO work queue (std::deque) with
  mutex+condvar for distributing tests to a configurable thread pool
- runAllTests(): orchestrate parallel test execution

Executor.cpp: Implementation with extracted static helpers:
- captureException(): capture and clear thrown exception from runtime
- createTestRuntime(): create configured HermesRuntime with ES6Proxy,
  MicrotaskQueue, Test262, HermesInternal, AsyncGenerators, Eval, and
  TimeLimitMonitor for timeout enforcement
- drainTaskQueue(): process queued setTimeout callbacks with microtask
  draining after each one
- executeCompiledTest(): run bytecode in fresh runtime, install console
  bindings ($262, setTimeout), handle negative expectations (parse,
  runtime, resolution phases), drain microtasks and task queue
- processTestEntry(): read test file, parse frontmatter, determine
  strict/non-strict/raw variants, skip by feature and testIntl.js,
  short-circuit on first variant failure, thread-safe result collection
- reportProgress(): periodic progress display (every 100 tests)

main.cpp: Integrates executor with:
- shouldSkipByFeature() and filterBySkiplist() for path-based skipping
- Result tallying by category with failure detail reporting
- Slowest test display (--show-slowest-tests)
- Summary statistics with pass/fail/skip/timeout counts

CMakeLists.txt: Added Executor.cpp and hermesConsoleHost_obj link
dependency.

Differential Revision: D98522849
@lavenzg @meta-codesync
Phase 5: Result reporting & progress display
856ab29 
Summary:
Refactor result reporting into structured helpers and add permanent
skip tracking, wall-clock timing, and formatted summary table.

Executor changes:
- Add permanentFeatureSkipped atomic counter to processTestEntry()
  and runAllTests(), tracking PermanentUnsupportedFeature skips
  separately from regular feature skips.

main.cpp reporting helpers (extracted from inline Phase 4 code):
- ResultTally struct: per-category counters (compileFail,
  compileTimeout, executeFail, executeTimeout) with failures vector
  and totalFailed() accessor.
- tallyResults(): categorize results by ResultCode, leveraging the
  executor's one-result-per-file short-circuit behavior.
- printSlowestTests(): display N slowest tests sorted by duration,
  with seconds format (e.g., "0.10s") and dashed borders.
- printResults(): formatted summary table matching Python/Rust runner
  output with Total, Passes, Failures, Skipped, Permanently Skipped,
  Pass Rate, and failure breakdown rows. Pass Rate computed over
  executed tests only (excludes skipped). Assert guards unsigned
  subtraction invariant for skip accounting.

main.cpp integration:
- FilterResult gains permanentlySkippedCount, tracked via
  PermanentSkipList SkipReason in filterBySkiplist().
- Wall-clock timing via steady_clock around runAllTests().
- Removed inline result tallying and "Skipped N tests via skiplist"
  message, replaced by printResults() call.

Differential Revision: D98522847
@meta-codesync
Add enableTDZ flag to CompileFlags
54395dd 
Summary:
Add enableTDZ (Temporal Dead Zone) flag to hbc::CompileFlags
and wire it through BCProviderFromSrc to CodeGenerationSettings.
This allows callers like the test runner to enable TDZ checking
for let/const when compiling test262 tests.

Differential Revision: D98821643
@meta-codesync
Phase 6: Full verification & benchmarking
418b067 
Summary:
Harden the test runner for full test262 suite verification.

Executor changes:
- Enable ES6 block scoping and TDZ in both RuntimeConfig and
  CompileFlags for full let/const semantics.
- Suppress print/alert output by installing no-op overrides before
  running test bytecode, preventing stdout noise from test output.
- Move expectRuntimeError computation into executeCompiledTest(),
  removing it from executeTestVariant() parameter list.
- Remove negative.errorType matching for expected compile errors
  (too fragile against Hermes-specific error messages).
- Skip module tests early (Hermes doesn't support ES module
  execution) instead of forcing them to non-strict mode.
- Extract buildTestIncludes() as public API (moved from main.cpp
  buildIncludes), now skips harness includes for raw tests.
- Swap drainTaskQueue parameter order (ctx before scope) for
  consistency.
- Rename hermesRuntime -> jsiRuntime in TestRuntimeEnv.
- Use std::move for timeLimitMonitor in createTestRuntime return.

Progress display:
- Replace "N/total tests complete" with percentage-based milestones
  "Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.."
- Thread-safe via atomic compare_exchange_strong, each milestone
  printed exactly once. Move reportProgress() call to lambda in
  runAllTests() instead of inside processTestEntry().

Frontmatter changes:
- Defer license header stripping and blank line trimming until after
  YAML parsing, so raw tests preserve their source exactly.

main.cpp changes:
- Remove buildIncludes() (moved to Executor as buildTestIncludes).
- Skip module tests in dumpTestSource() instead of forcing non-strict.
- Minor comment and doc string cleanups.

Differential Revision: D98823646
@lavenzg @meta-codesync
Phase 7: Skiplist integration, handle sanitizer & intl support
7419527 
Summary:
Integrate the skip list system with the C++ test runner. Adds handle
sanitizer disable support (matching the Python runner's handlesan_skip_list),
supported feature detection (compile-time via ifdefs), non-intl path
skipping, and the --test-intl CLI flag.

Key changes:
- Skiplist: add handlesanPaths_, supportedFeatures_, shouldDisableHandleSan(),
  shouldSkipPathNonIntl(), and compile-time feature detection
- Executor: thread disableHandleSan through createTestRuntime/executeCompiledTest/
  executeTestVariant/processTestEntry
- main: add --test-intl flag, extend filterBySkiplist() with testIntl and lazy params

Differential Revision: D98824412
@meta-codesync
Add DESIGN.md documenting architecture and design decisions
716577e 
Differential Revision: D98522853
@meta-codesync
Add -O flag for optimization passes
120c930 
Summary:
Add support for enabling compiler optimization passes via the -O CLI flag,
matching the Python runner's behavior. When -O is passed, the full
optimization pipeline (runFullOptimizationPasses) is invoked during
compilation via BCProviderFromSrc::create().

Default is no optimization (-O0), matching the Python runner's default
behavior (--opt flag with store_true, defaulting to False).

Verified: 38,418 passes, 0 failures with -O on full test262 suite.

Differential Revision: D98522851
@meta-codesync
Add --lazy flag for lazy compilation mode
972cc34 
Summary:
Add support for lazy compilation in the C++ test runner, matching the
Python runner's --lazy flag behavior.

When --lazy is passed:
- CompileFlags.lazy = true is set, enabling lazy compilation in
  BCProviderFromSrc
- RuntimeModuleFlags.persistent is set to false (incompatible with lazy)
- Tests in lazy_skip_list are skipped (matching Python runner's
  conditional: if lazy: skip_categories.append(LAZY_SKIP_LIST))

When --lazy is not passed (default):
- Tests in lazy_skip_list run normally and are not skipped, matching the
  Python runner's behavior where LAZY_SKIP_LIST is only appended when
  the lazy flag is set

Differential Revision: D98522848
@meta-codesync
Add --jit flag for JIT compilation mode
550aebc 
Summary:
Adds a `--jit` flag with three modes: off (default), on, and force.
This maps directly to RuntimeConfig's EnableJIT and ForceJIT settings,
matching the Python runner's `--vm-args='-Xjit=force'` behavior.

The flag is threaded through ExecConfig → processTestEntry →
executeTestVariant → executeCompiledTest → createTestRuntime, where
it configures the RuntimeConfig builder with .withEnableJIT() and
.withForceJIT().

Differential Revision: D98858012
@lavenzg @meta-codesync
Switch GitHub CI to C++ test runner (facebook#1963)
b874810 
Summary:
Pull Request resolved: facebook#1963

Replace the Python test_runner.py invocation with the C++ test-runner
binary in the GitHub Actions CI workflow for macOS, Linux, and Linux
ARMv7 jobs. The C++ runner is ~8x faster and runs tests in-process.

Key changes:
- macOS/Linux: use build/bin/test-runner with --test-intl and --skiplist
- JIT matrix variant: replace --vm-args='-Xjit=force' with --jit=force
- ARMv7: same as Linux, no JIT variant
- Windows: kept on Python runner (C++ runner uses POSIX signal APIs
  for crash isolation that aren't available on Windows)

Differential Revision: D98531546
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lavenzg