[GSoC 2026] Kafka Streams runner: put the runner behind an opt-in build flag

[junaiddshaukat]

Summary

Part of #18479. Puts the Kafka Streams runner behind an opt-in build flag, so that it can live in master without being built for, or released to, anyone who has not asked for it.

./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:build

Without the flag the runner's subprojects are not included in the build at all.

Why

The runner is worth having somewhere people can build it, use it, and work on it — the point being to give it somewhere to be developed and maintained rather than to ship it. It is not ready to be released, and it has known bugs rather than only missing features: bundles are not yet closed after a bounded time (#39633), and other cases are still being looked at. Merging that as part of the normal build would be hard to defend.

An opt-in flag separates the two questions. Nothing reaches a user who did not ask for it, and no release contains it, so if the runner becomes stable enough the flag comes off and it is built like any other; and if it does not, it can be dropped again without affecting anyone.

What changed

settings.gradle.kts only includes :runners:kafka-streams and its subprojects when the property is set. Three places assumed they were always there:

• javaPreCommit had a hard dependsOn on the runner's build.
• The Python portable suite registered kafkaStreamsValidatesRunner unconditionally, and its job server is only a project when the flag is set.
• Both Kafka Streams CI workflows now pass the flag, since otherwise there is nothing for them to build.

Documentation says what the flag is and why it exists, and every command on the runner's page now carries it.

The options table on that page had also fallen behind and is corrected here: maxBundleSize no longer doubles as the source's per-poll limit, which #39748 separated, and readMaxElementsPerPoll, readMaxPollTimeMs and sessionTimeoutMs were missing.

Testing

Checked in both directions, since the point of the change is what happens when the flag is absent:

./gradlew projects                                   # no Kafka Streams projects
./gradlew projects -Pwith-kafka-streams-runner       # all of them
./gradlew javaPreCommit --dry-run                    # configures, no Kafka Streams tasks
./gradlew -Pwith-kafka-streams-runner :runners:kafka-streams:build

kafkaStreamsValidatesRunner resolves with the flag and does not exist without it.

[github-actions]

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

[je-ik]

LGTM, can we please rebase this on feat/18479-kafka-streams-runner-skeleton so that it can be merge independently on the previous PR #39761 (review)?

[junaiddshaukat]

LGTM, can we please rebase this on feat/18479-kafka-streams-runner-skeleton so that it can be merge independently on the previous PR #39761 (review)?

Rebased onto feat/18479-kafka-streams-runner-skeleton, so it is one commit and can merge independently of #39761.

I checked the gate still holds after the rebase: ./gradlew projects shows no Kafka Streams projects, and with -Pwith-kafka-streams-runner it shows all three.

One check on this PR fails and I do not think it is the change. beam_PreCommit_Yaml_Xlang_Direct dies in "Setup environment" with the action gradle/actions/setup-gradle is not allowed in apache/beam. It passes on master and fails only on my fork branch, and this PR triggers it because it touches the root build files. Looks like the actions allowlist for fork PRs, but you can see more than I can there.