KAFKA-9020: Streams sub-topologies should be sorted by sink -> source relationship #7495
Conversation
ableegoldman
force-pushed
the
subtopology-order
branch
from
ca46990
to
b4c7b3e
Compare
|
Java 11/2.12 failed with Java 11/2.13 failed with Java 8 failed with retest this please |
|
Java 8 and Java 11.12 passed |
|
ping either of @guozhangwang @vvcephei for a second review |
|
Java 11/2.13 failed - test results already cleaned up. retest this please |
|
on this PR builder run Java 11/2.12 and Java 11/2.13 passed |
|
Merged #7495 into trunk. |
… relationship (#7495) Subtopologies are currently ordered alphabetically by source node, which prior to KIP-307 happened to always result in the "correct" (ie topological) order. Now that users may name their nodes anything they want, we must explicitly order them so that upstream node groups/subtopologies come first and the downstream ones come after. Reviewers: Guozhang Wang <wangguoz@gmail.com>, Bill Bejeck <bbejeck@gmail.com>
|
cherry-picked to 2.4 |
|
|
||
| for (final String nodeName : Utils.sorted(allSourceNodes)) { | ||
| // Traverse in topological order | ||
| for (final String nodeName : nodeFactories.keySet()) { |
There was a problem hiding this comment.
q: how could we guarantee a topological traversal on a map without order?
There was a problem hiding this comment.
nodeFactories is in topological order, for example in InternalStreamsBuilder#buildAndOptimizeTopology we only add a node once all its parents have been written to the topology already
Subtopologies are currently ordered alphabetically by source node, which prior to KIP-307 happened to always result in the "correct" (ie topological) order. Now that users may name their nodes anything they want, we must explicitly order them so that upstream node groups/subtopologies come first and the downstream ones come after.