Improved Stream Router Engine #1305

dennisoelkers · 2015-07-16T11:56:20Z

This pull request adds the new stream router engine optimized for best guess minimum cost evaluation.

This reverts commit ce1433b. Reverting this as lists are used in all other places unfortunately.

joschi · 2015-07-16T12:31:35Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

+    protected final StreamFaultManager streamFaultManager;
+    protected final StreamMetrics streamMetrics;
+    protected final TimeLimiter timeLimiter;
+    protected final long streamProcessingTimeout;


Why did you change the visibility of those final fields? They're only being accessed in this very class.

joschi · 2015-07-16T12:32:24Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

+        final List<Rule> regexRules = Lists.newArrayList();
+
+        for (Stream stream : streams) {
+            final Boolean sufficient = stream.getMatchingType() == Stream.MatchingType.OR;


This could be a primitive boolean.

joschi · 2015-07-16T12:35:51Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

        private final Stream stream;
        private final StreamRule rule;
        private final StreamRuleMatcher matcher;
+        private final Boolean sufficient;


This could be a primitive boolean.

While sufficient is shorter, I think shortCircuit or shortCircuitRule describes better, what this flag denotes in reality.

joschi · 2015-07-16T12:42:50Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

@@ -180,7 +216,7 @@ public String getFingerprint() {

            for (final StreamRule streamRule : stream.getStreamRules()) {
                try {
-                    final Rule rule = new Rule(stream, streamRule);
+                    final Rule rule = new Rule(stream, streamRule, null);


What meaning would sufficient = null have in this case? There's also not a single check for Rule#isSufficient() == null.

It is not relevant for the reuse of this class in the scope of the test matches, as shortcuts are not taken here and all rules need to be evaluated.

In this case, the Rule#isSufficient() method would need to be annotated with @Nullable and null-checks being introduced on every call of that method.

TBH, I'd prefer simply using false here and making Rule#isSufficient() return a primitive boolean.

TBH, I'd prefer simply using false here and making Rule#isSufficient() return a primitive boolean.

Agreed, avoids null checks as well. Or is there a specific reason why you use Boolean instead of boolean?

The reason was because of the reuse in the test matching logic. It is misleading to pass any value there. Null checking is a pita too. I will change to a primitive and add a comment.

Actually I will now replace it with the matching type of the stream. The wording was annoying from the beginning and a boolean flag is just not able to properly represent what it's supposed to do.

joschi · 2015-07-16T12:52:27Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

@@ -83,33 +73,49 @@ public StreamRouterEngine(@Assisted List<Stream> streams,
        this.streamProcessingTimeout = streamFaultManager.getStreamProcessingTimeout();
        this.fingerprint = new StreamListFingerprint(streams).getFingerprint();

-        for (final Stream stream : streams) {
-            for (final StreamRule streamRule : stream.getStreamRules()) {
+        this.rulesList = Lists.newArrayList();


This list should be initialized with the (maximum) size presenceRules.size() + exactRules .size() + greaterRules.size() + smallerRules.size() + regexRules.size().

joschi · 2015-07-16T12:53:50Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

+        final List<Rule> smallerRules = Lists.newArrayList();
+        final List<Rule> regexRules = Lists.newArrayList();
+
+        for (Stream stream : streams) {


Maybe instead of sorting the stream rules manually, it would make sense to introduce some kind of complexity rating for the different types of rules, and use a comparator to sort the list of rules accordingly.

I decided against this, because the effort to sort element by element is much higher than doing it manually.

joschi · 2015-07-16T12:58:35Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

@@ -137,33 +143,63 @@ public String getFingerprint() {
     * @return the list of matching streams
     */
    public List<Stream> match(Message message) {


Finally changing the method return type to Set<Stream> would make sense now, I guess. This would also avoid having to copy the matching result at the end of this method.

Tried that. Not possible right now, as it would involve external changes of the API.

This is a bit unfortunate because right now this collection is at least copied twice for each message (one time in this method, another time in Message#setStreams()).

Indeed. We should address this. It is beyond me why a List was chosen in the first place.

joschi · 2015-07-16T13:01:33Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

+                }
+            }, streamProcessingTimeout, TimeUnit.MILLISECONDS, true);
+        } catch (Exception e) {
+            streamFaultManager.registerFailure(rule.getStream());


Logging the exception on DEBUG or TRACE level might help in case of errors.

Done in the StreamFaultManager.registerFailure() method.

The exception is not being logged in StreamFaultManager#registerFailure().

joschi · 2015-07-16T13:03:58Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

-    }
-
-    private class Rule {
+    protected class Rule {


Why did you change the visibility level to protected?

joschi · 2015-07-16T13:13:26Z

graylog2-server/src/test/java/org/graylog2/streams/StreamRouterEngineTest.java

@@ -58,7 +59,7 @@ public void setUp() throws Exception {
        when(streamFaultManager.getStreamProcessingTimeout()).thenReturn(250L);
    }

-    private StreamRouterEngine newEngine(List<Stream> streams) {
+    protected StreamRouterEngine newEngine(List<Stream> streams) {


Why was the visibility changed to protected?

joschi · 2015-07-16T13:20:07Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

+        return Lists.newArrayList(result);
+    }
+
+    private Stream matchWithTimeOut(final Message message, final Rule rule) {


Please add @Nullable.

joschi · 2015-07-16T13:20:21Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

+        public Boolean isSufficient() {
+            return sufficient;
+        }
+
        public Stream match(Message message) {


Please add @Nullable.

joschi · 2015-07-16T13:21:34Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

+            }
+        }
+
+


Unnecessary blank line. 😉

joschi · 2015-07-16T13:23:44Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

-            streamFaultManager.registerFailure(stream);
-        }
+        final Set<Stream> result = Sets.newHashSet();
+        final Set<Stream> blackList = Sets.newHashSet();


These sets could be initialized with the (maximum) number of items: Sets#newHashSetWithExpectedSize(streams.size())

Didn't do this, because it's misleading. We are not expecting it to have that size, it's is the maximum size they could have.

joschi · 2015-07-16T14:04:56Z

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java

                }
            }
        }
+
+        this.rulesList = new ImmutableList.Builder<Rule>()


Unfortunately this suffers the same problem: Having to resize the underlying array multiple times if the number of rules is high (at least once if the number of rules is greater than 4).

I prefer immutability in this case. Remember that it's done only during initialization of the stream router (which is done only when the set of stream rules changes).

Having looked at the surrounding code, this class is initialized every second (even if the rules haven't been changed) and rulesList is only used internally in this class, so I think initializing the list with the correct size would yield some improvements and is worth the disadvantage(?) over having a mutable collection instead of an immutable one.

…ons.

…ed-router-engine Improved StreamRouterEngine

dennisoelkers added 8 commits Jul 10, 2015

Adding router engine implementation optimized for OR-matching.

076f2e3

Correcting multiple stream evaluation/adding tests/optimizing blacklist

6027797

Making blacklist a set, using time limiter for regex matches.

d63a860

Changing result type of matcher method to Set.

a7b7c96

Revert "Changing result type of matcher method to Set."

73966ec

This reverts commit ce1433b. Reverting this as lists are used in all other places unfortunately.

Removing comments.

015b518

Recording incoming messages in stream metrics.

6ea1e13

Moving new stream routing engine from sub- to superclass.

Loading status checks…

525800a

dennisoelkers added the ready-for-review label Jul 16, 2015

joschi self-assigned this Jul 16, 2015

joschi added performance processing labels Jul 16, 2015

joschi added this to the 1.2.0 milestone Jul 16, 2015

joschi reviewed Jul 16, 2015
View changes

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java Outdated

}

private class Rule {

protected class Rule {

This comment has been minimized.

Sign in to view

joschi Jul 16, 2015
Contributor

Why did you change the visibility level to protected?

joschi reviewed Jul 16, 2015
View changes

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java Outdated

public Boolean isSufficient() {

return sufficient;

}

public Stream match(Message message) {

This comment has been minimized.

Sign in to view

joschi Jul 16, 2015
Contributor

Please add @Nullable.

joschi reviewed Jul 16, 2015
View changes

graylog2-server/src/main/java/org/graylog2/streams/StreamRouterEngine.java Outdated

}

}

This comment has been minimized.

Sign in to view

joschi Jul 16, 2015
Contributor

Unnecessary blank line. 😉

joschi reviewed Jul 16, 2015
View changes

dennisoelkers added 2 commits Jul 16, 2015

Changing scopes, making rules list immutable.

Loading status checks…

e4b76b6

Marking getter as nullable.

Loading status checks…

994b199

joschi reviewed Jul 16, 2015
View changes

dennisoelkers and others added 9 commits Jul 16, 2015

Adding timer to time limited matching, logging for unexpected excepti…

Loading status checks…

c198a01

…ons.

Replacing sufficient boolean with the matching type of the stream.

Loading status checks…

ff8e537

Mark StreamRouterEngine.Rule#match(Message) as nullable

be932a7

Remove unused local variable

Loading status checks…

229533e

Initialize rulesList in StreamRouterEngine with correct size

c0246e4

Make return value of StreamRouterEngine#match() immutable

066de1b

Mark StreamRouterEngine#matchWithTimeOut() as nullable

b338a2d

Avoid repeated hash lookup for StreamRule#getType()

cc9a113

Move StreamRouterEngine#matchWithTimeOut() into StreamRouterEngine.Rule

Loading status checks…

02a18ea

joschi added a commit that referenced this pull request Jul 17, 2015

Merge pull request #1305 from Graylog2/feature-stream-matcher-optimiz…

Loading status checks…

b26ee61

…ed-router-engine Improved StreamRouterEngine

joschi merged commit b26ee61 into master Jul 17, 2015
2 checks passed

2 checks passed

ci Jenkins build graylog2-server-integration-pr 60 has succeeded
Details

continuous-integration/travis-ci/push The Travis CI build passed
Details

joschi deleted the feature-stream-matcher-optimized-router-engine branch Jul 17, 2015

joschi removed the ready-for-review label Jul 17, 2015

Graylog2 / graylog2-server

Improved Stream Router Engine #1305

Improved Stream Router Engine #1305

Conversation

dennisoelkers commented Jul 16, 2015

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

bernd Jul 16, 2015 Member

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

bernd Jul 16, 2015 Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

dennisoelkers Jul 16, 2015 Author Member

This comment has been minimized.

joschi Jul 16, 2015 Contributor

This comment has been minimized.

joschi Jul 17, 2015 Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

bernd Jul 16, 2015
Member

dennisoelkers Jul 16, 2015
Author Member

dennisoelkers Jul 16, 2015
Author Member

bernd Jul 16, 2015
Member

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

joschi Jul 16, 2015
Contributor

dennisoelkers Jul 16, 2015
Author Member

joschi Jul 16, 2015
Contributor

joschi Jul 17, 2015
Contributor