[STORM-3685] Detect and prevent cycles when Topology is submitted. #3322

bipinprasad · 2020-08-07T17:24:26Z

What is the purpose of the change

Topology is expected to be a Directed Acyclic Graph. Cycles in component flow can cause unexpected behavior, for example a deadlock when one of the loop components signals a back-pressure. Or properly detecting proximity when scheduling.

How was the change tested

New test to create topology with and without cycles and test detection.

bipinprasad · 2020-08-10T14:33:10Z

Close/Reopen for the rebuild - failures due to license file failures for JDK 11?

bipinprasad · 2020-08-11T20:24:21Z

close/reopen to rebuild.

storm-client/src/jvm/org/apache/storm/utils/Utils.java

storm-client/test/jvm/org/apache/storm/utils/UtilsTest.java

storm-client/src/jvm/org/apache/storm/utils/Utils.java

bipinprasad · 2020-08-13T19:58:30Z

close/reopen to rebuild.

storm-client/test/jvm/org/apache/storm/utils/UtilsTest.java

storm-client/src/jvm/org/apache/storm/utils/Utils.java

Ethanlm

I have some minor comments (including previous comments above). Otherwise looks good to me.

storm-client/src/jvm/org/apache/storm/utils/Utils.java

Ethanlm · 2020-08-31T21:12:52Z

storm-client/src/jvm/org/apache/storm/StormSubmitter.java

@@ -233,7 +235,7 @@ public static void submitTopologyAs(String name, Map<String, Object> topoConf, S
        conf.putAll(topoConf);
        topoConf.putAll(prepareZookeeperAuthentication(conf));

-        validateConfs(conf, topology);
+        validateConfs(conf, topology, name);


I feel like this is not the right place to detect cycles. We should probably put it in another function, like
StormCommon.validateBasic

https://git.vzbuilders.com/storm/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/StormCommon.java#L156

or create a new method.

Created new method StormCommon.validateCycleFree

Moving it to Utils is also okay, like
Utils.validateTopologyBlobStoreMap
Utils.validateTopologyName
etc.

Probably makes more sense since findComponentCycles is implemented inside Utils.

And we could make Utils.findComponentCycles "@VisibleForTesting". So the public method will be only Utils. validateCycleFree and the internal implementation can be changed if we'd like in the future.

@bipinprasad
Sorry for not being clear at the first comment. What do you think about moving it to Utils?^ Trying to not expose too many public methods. Also StormCommon is org.apache.storm.daemon.StormCommon. StormSubmitter is completely on client side. I feel like Utils is a better place for this.

Ok. Will switch.

Moved from StormCommons to Utils.

…topologies for cycle detection.

…lt connections counts. Fix topo comments.

…n StormSubmitter.submitTopologyAs().

Ethanlm · 2020-09-02T14:31:35Z

storm-client/src/jvm/org/apache/storm/StormSubmitter.java

@@ -51,6 +52,7 @@
 import org.apache.storm.validation.ConfigValidation;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
+import sun.net.ProgressListener;


Is this used anywhere?

No. Will remove this.

Ethanlm · 2020-09-02T14:31:46Z

storm-client/src/jvm/org/apache/storm/StormSubmitter.java

+            StormCommon.validateCycleFree(topology, name);
+        } catch (InvalidTopologyException ex) {
+            LOG.warn(ex.get_msg(), ex);
+            System.out.println(ex.get_msg());


We should remove this

Did we not want to warn the user when submitting?

LOG.warn should be sufficient. We don't need System.out.println()

ok. removed sysout.

Ethanlm · 2020-09-02T14:32:12Z

storm-client/src/jvm/org/apache/storm/StormSubmitter.java

+        try {
+            StormCommon.validateCycleFree(topology, name);
+        } catch (InvalidTopologyException ex) {
+            LOG.warn(ex.get_msg(), ex);


Will ex already contains ex.get_msg()? If so, we will see repeating messages, which should be avoid.

The constructor seems to imply that the message will not be in the Exception, since the proper super method is not called.

Do you have an output example that I can take a look?

InvalidTopologyException constructor does not call Exception.super(msg): https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/generated/InvalidTopologyException.java#L111

hence, its msg variable is not available in the Exception.getMessage()

Right. So the log will look like

19:22:23.981 [main] WARN o.a.s.StormSubmitter - Topology wc contains cycles in components "count,split" org.apache.storm.generated.InvalidTopologyException: null at org.apache.storm.daemon.StormCommon.validateCycleFree(StormCommon.java:596) ~[storm-client-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:244) [storm-client-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:214) [storm-client-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:177) [storm-client-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] at org.apache.storm.topology.ConfigurableTopology.submit(ConfigurableTopology.java:119) [storm-client-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] at org.apache.storm.starter.WordCountTopology.run(WordCountTopology.java:58) [storm-starter-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] at org.apache.storm.topology.ConfigurableTopology.start(ConfigurableTopology.java:68) [storm-client-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] at org.apache.storm.starter.WordCountTopology.main(WordCountTopology.java:36) [storm-starter-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]

We can probably use WrappedInvalidTopologyException inside validateCycleFree method so the stacktrace will make more sense. What do you think?
It will be like

org.apache.storm.utils.WrappedInvalidTopologyException: Topology wc2 contains cycles in components "count,split" at org.apache.storm.daemon.StormCommon.validateCycleFree(StormCommon.java:596) ~[storm-client-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] .....

That is better than a scary "null" in the stack trace.
Changed.

@throws

…fix @throws tag; remove superfluous sysout..

…idTopologyException.

Ethanlm

+1

Thank you so much for addressing all the review comments

Bipin Prasad added 2 commits August 7, 2020 12:16

[STORM-3685] Detect and prevent cycles when Topology is submitted.

ffcf116

[STORM-3685] Warn instead of rejecting Topology with cycles.

0139cbf

bipinprasad force-pushed the storm3685 branch from aefd3b3 to 0139cbf Compare August 7, 2020 17:58

bipinprasad closed this Aug 10, 2020

bipinprasad reopened this Aug 10, 2020

bipinprasad closed this Aug 11, 2020

bipinprasad reopened this Aug 11, 2020

Ethanlm reviewed Aug 12, 2020

View reviewed changes

[STORM-3685] Exclude diamonds and add a complex cycle(s) test case.

91f97dc

bipinprasad closed this Aug 13, 2020

bipinprasad reopened this Aug 13, 2020

Ethanlm reviewed Aug 31, 2020

View reviewed changes

storm-client/test/jvm/org/apache/storm/utils/UtilsTest.java Outdated Show resolved Hide resolved

storm-client/test/jvm/org/apache/storm/utils/UtilsTest.java Outdated Show resolved Hide resolved

storm-client/src/jvm/org/apache/storm/utils/Utils.java Outdated Show resolved Hide resolved

Ethanlm reviewed Aug 31, 2020

View reviewed changes

Bipin Prasad added 3 commits August 31, 2020 17:25

[STORM-3685] Fix NPE in a new test and add randomized 100 randomized …

4117a14

…topologies for cycle detection.

[STORM-3685] Missing spouts not signaled as error. Randomize spout/bo…

4a29dac

…lt connections counts. Fix topo comments.

[STORM-3685] Add new method StormCommon.validateCycleFree() and use i…

cc7a66d

…n StormSubmitter.submitTopologyAs().

Ethanlm reviewed Sep 2, 2020

View reviewed changes

Bipin Prasad added 2 commits September 2, 2020 13:48

[STORM-3685] Remove unused imports and unused private static method; …

77e79fc

…fix @throws tag; remove superfluous sysout..

[STORM-3685] Move validateCycleFree() to Utils and throw WrappedInval…

c34bf1b

…idTopologyException.

Ethanlm approved these changes Sep 4, 2020

View reviewed changes

Ethanlm merged commit 8399edc into apache:master Sep 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[STORM-3685] Detect and prevent cycles when Topology is submitted. #3322

[STORM-3685] Detect and prevent cycles when Topology is submitted. #3322

bipinprasad commented Aug 7, 2020

bipinprasad commented Aug 10, 2020

bipinprasad commented Aug 11, 2020

bipinprasad commented Aug 13, 2020

Ethanlm left a comment •

edited

Loading

Ethanlm Aug 31, 2020

bipinprasad Sep 1, 2020

Ethanlm Sep 2, 2020 •

edited

Loading

Ethanlm Sep 2, 2020 •

edited

Loading

bipinprasad Sep 3, 2020

bipinprasad Sep 3, 2020

Ethanlm Sep 2, 2020

bipinprasad Sep 2, 2020

Ethanlm Sep 2, 2020

bipinprasad Sep 2, 2020

Ethanlm Sep 2, 2020 •

edited

Loading

bipinprasad Sep 2, 2020

Ethanlm Sep 2, 2020

bipinprasad Sep 2, 2020

Ethanlm Sep 2, 2020

bipinprasad Sep 2, 2020

Ethanlm Sep 2, 2020

bipinprasad Sep 3, 2020

Ethanlm left a comment

[STORM-3685] Detect and prevent cycles when Topology is submitted. #3322

[STORM-3685] Detect and prevent cycles when Topology is submitted. #3322

Conversation

bipinprasad commented Aug 7, 2020

What is the purpose of the change

How was the change tested

bipinprasad commented Aug 10, 2020

bipinprasad commented Aug 11, 2020

bipinprasad commented Aug 13, 2020

Ethanlm left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ethanlm Sep 2, 2020 • edited Loading

Choose a reason for hiding this comment

Ethanlm Sep 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ethanlm Sep 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ethanlm left a comment

Choose a reason for hiding this comment

Ethanlm left a comment •

edited

Loading

Ethanlm Sep 2, 2020 •

edited

Loading

Ethanlm Sep 2, 2020 •

edited

Loading

Ethanlm Sep 2, 2020 •

edited

Loading