Skip to content

Cut CI wall clock: rebalance the Spark test jobs and surface the slow tail #19528

Description

@yihua

The critical path for a PR is test-spark-java17-java-tests-part1 at roughly 70 minutes, and the part1/part2/part3 splits follow module boundaries rather than measured duration.

  • Rebalance the splits by measured job duration, and re-measure whenever what those jobs run changes.
  • Report the slowest tests per job so the long tail is visible and can be fixed or moved off the PR path.
  • Give contributors early signal: run the cheap validations and a fast compile before the long test jobs, so an obviously broken PR fails in minutes rather than after an hour.

Part of #19524.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ciCI/CD relatedpriority:highSignificant impact; potential bugs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions