More Aggressively Skip Duplicate Workflows & Cancel Workflow Runs if One Test in The Group Fails To Speed Up CI

[kbendick]

Feature Request / Improvement

Right now, we depend on the workflow itself to cancel duplicates, using the built-in features of the github actions via the concurrency_group.

iceberg/.github/workflows/flink-ci.yml

Lines 52 to 54 in c965af5

	concurrency:
	group: ${{ github.workflow }}-${{ github.ref }}
	cancel-in-progress: ${{ github.event_name == 'pull_request' }}

While this is good enough, it requires each individual run to start and then notice that it is cancelled. As the number of actions has increased, we are spending more time waiting on CI.

One case where we can get time back is cancelling duplicates in bulk. For example, it's not uncommon for somebody to open a PR, then push to it, and then push to it again not too long after (if they notice a small issue etc).

In the case above, we'd have to wait for each action run to be matched to a runner, start, then notice that it's a duplicate and get cancelled.

If we use something like https://github.com/marketplace/actions/skip-duplicate-actions, we can have it so that when a duplicate workflow is picked up, it will cancel all but the latest workflow runs. Thus saving a lot of time in churning workflows that would be cancelled.

This is not a high priority, but something to consider.

I'd also still like to get it so that we have one starting point and then all of that spreads out via github's workflow_dispatch, so that if one test fails, all test groups fail simultaneously. This way, if the Java CI fails, we won't have 7 or 9 concurrent Spark test workloads that still run for that patch (as that patch isn't going to be mergeable in that state anyway).

Query engine

No response

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'