CI: Select PR matrices incrementally

[ajantha-bhat]

Mailing list discussion: https://lists.apache.org/thread/36vxlql61gojbg639c86mnz78n57kvgm

Summary

• add a shared PR CI planner that uses changed files, full-ci labels, and global build/workflow changes to select PR matrices
• keep full Java/Spark/Flink/Hive/Kafka/Delta/CVE coverage for main, release branches, tags, full-ci, and global Gradle/workflow changes
• use Java 17 as the primary JVM for ordinary PRs, while full CI paths keep Java 17 and Java 21
• preserve the existing Gradle cache policy from Build: Designate a single Gradle cache writer across CI workflows #16356: only java-ci.yml build-checks (17) writes cache on main; all other Gradle jobs are read-only

Selective PR behavior

• Spark-only changes run Spark jobs, not Flink/Hive/Kafka jobs
• spark/v4.1/** selects Spark 4.1 only; other versioned Spark paths behave similarly
• flink/v2.0/** selects Flink 2.0 only; other versioned Flink paths behave similarly
• API/Core/Data/file-format changes run Java checks plus latest Spark and latest Flink canaries
• runtime/bundle CVE scans are limited to affected runtime artifacts, while dependency/global changes run the full CVE matrix

Gradle cache behavior

This PR does not add new Gradle cache writers. PR jobs restore cache read-only, release/tag jobs remain read-only, and main keeps the single canonical writer introduced in #16356.

Validation

• YAML parse for edited workflows
• git diff --check origin/main...HEAD
• bash -n .github/scripts/plan-pr-ci.sh
• synthetic planner assertions for Spark 4.1, Flink 2.0, core canaries, global build changes, and Kafka/CVE selection
• ./gradlew -h

[kevinjqliu]

thanks for the PR @ajantha-bhat

i've done some work with gradle cache recently (#16356) and made it so that there's only 1 canonical writer. Before this change, I saw that we were constantly getting thrashed by multiple cache writers and cache utilization was really low.

Theres also a security component to this, we should only write to cache on push to main branch. We should not allow PRs to write to the shared cache since that's a cache poisoning vulnerability.

I'm curious how this change effects when to save to cache and how its reused

[ajantha-bhat]

Thanks @kevinjqliu, good catch.

I updated the PR to preserve #16356's cache model. The selective PR matrix planner now stands on its own, and this PR no longer adds custom cache artifact merge/store actions or additional Gradle cache writers.

Behavior after the update:

• PRs: Gradle cache restore only, no writes
• main: only java-ci.yml build-checks (17) writes, same as Build: Designate a single Gradle cache writer across CI workflows #16356
• release branches/tags: read-only cache, full coverage
• ordinary PRs: selected matrices with Java 17
• full-ci/global changes: full matrices with Java 17 and 21

So the runner reduction now comes from selective matrices, while the shared Gradle cache writer remains unchanged.