Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

check lookup before processing (better error handling) #20

Closed
jalogisch opened this issue Jan 3, 2017 · 1 comment
Closed

check lookup before processing (better error handling) #20

jalogisch opened this issue Jan 3, 2017 · 1 comment

Comments

@jalogisch
Copy link
Contributor

If you have a pipeline that extract data (like DNS Logfiles) that you extract non IP Data.

The error handling should be improved to get only one line and not the following

2017-01-02T20:55:06.289+01:00 ERROR [GlobalIpLookupFunction] Could not run global lookup for IP [NODATA-IPv6] with prefix [query_answer].
java.lang.RuntimeException: Could not fetch intel from [org.graylog.plugins.threatintel.providers.spamhaus.SpamhausIpLookupProvider] as part of global lookup.
	at org.graylog.plugins.threatintel.providers.global.GlobalLookupProvider.lookup(GlobalLookupProvider.java:87) ~[graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.threatintel.providers.global.GlobalLookupProvider.lookupIp(GlobalLookupProvider.java:62) ~[graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.threatintel.providers.global.ip.GlobalIpLookupFunction.evaluate(GlobalIpLookupFunction.java:53) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.threatintel.providers.global.ip.GlobalIpLookupFunction.evaluate(GlobalIpLookupFunction.java:16) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.evaluateUnsafe(FunctionExpression.java:59) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.ast.expressions.Expression.evaluate(Expression.java:36) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.ast.statements.VarAssignStatement.evaluate(VarAssignStatement.java:33) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.ast.statements.VarAssignStatement.evaluate(VarAssignStatement.java:22) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.processForResolvedPipelines(PipelineInterpreter.java:357) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.processForPipelines(PipelineInterpreter.java:291) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:248) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:192) [graylog-plugin-threatintel-0.9.0.jar:?]
	at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:56) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:82) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:61) [graylog.jar:?]
	at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35) [graylog.jar:?]
	at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Could not parse [NODATA-IPv6]
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476) ~[graylog.jar:?]
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:435) ~[graylog.jar:?]
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79) ~[graylog.jar:?]
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:143) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2352) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2324) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3953) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875) ~[graylog.jar:?]
	at org.graylog.plugins.threatintel.providers.LocalCopyListProvider.lookup(LocalCopyListProvider.java:141) ~[?:?]
	at org.graylog.plugins.threatintel.providers.spamhaus.SpamhausIpLookupProvider.lookup(SpamhausIpLookupProvider.java:22) ~[?:?]
	at org.graylog.plugins.threatintel.providers.global.GlobalLookupProvider.lookup(GlobalLookupProvider.java:85) ~[?:?]
	... 18 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Could not parse [NODATA-IPv6]
	at org.graylog.plugins.threatintel.providers.LocalCopyListProvider$5.load(LocalCopyListProvider.java:90) ~[?:?]
	at org.graylog.plugins.threatintel.providers.LocalCopyListProvider$5.load(LocalCopyListProvider.java:83) ~[?:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3953) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875) ~[graylog.jar:?]
	at org.graylog.plugins.threatintel.providers.LocalCopyListProvider.lookup(LocalCopyListProvider.java:141) ~[?:?]
	at org.graylog.plugins.threatintel.providers.spamhaus.SpamhausIpLookupProvider.lookup(SpamhausIpLookupProvider.java:22) ~[?:?]
	at org.graylog.plugins.threatintel.providers.global.GlobalLookupProvider.lookup(GlobalLookupProvider.java:85) ~[?:?]
	... 18 more
Caused by: java.lang.IllegalArgumentException: Could not parse [NODATA-IPv6]
	at org.apache.commons.net.util.SubnetUtils.toInteger(SubnetUtils.java:287) ~[?:?]
	at org.apache.commons.net.util.SubnetUtils.access$400(SubnetUtils.java:27) ~[?:?]
	at org.apache.commons.net.util.SubnetUtils$SubnetInfo.isInRange(SubnetUtils.java:125) ~[?:?]
	at org.graylog.plugins.threatintel.providers.spamhaus.SpamhausIpLookupProvider.fetchIntel(SpamhausIpLookupProvider.java:66) ~[?:?]
	at org.graylog.plugins.threatintel.providers.spamhaus.SpamhausIpLookupProvider.fetchIntel(SpamhausIpLookupProvider.java:22) ~[?:?]
	at org.graylog.plugins.threatintel.providers.LocalCopyListProvider$5.load(LocalCopyListProvider.java:88) ~[?:?]
	at org.graylog.plugins.threatintel.providers.LocalCopyListProvider$5.load(LocalCopyListProvider.java:83) ~[?:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3953) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957) ~[graylog.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875) ~[graylog.jar:?]
	at org.graylog.plugins.threatintel.providers.LocalCopyListProvider.lookup(LocalCopyListProvider.java:141) ~[?:?]
	at org.graylog.plugins.threatintel.providers.spamhaus.SpamhausIpLookupProvider.lookup(SpamhausIpLookupProvider.java:22) ~[?:?]
	at org.graylog.plugins.threatintel.providers.global.GlobalLookupProvider.lookup(GlobalLookupProvider.java:85) ~[?:?]
	... 18 more

depending on the skill of the graylog user it might not be seen what the initial problem is!
@jalogisch jalogisch mentioned this issue Jan 23, 2017
Closed
@lennartkoopmann lennartkoopmann modified the milestones: 1.0, 1.1 Mar 25, 2017
@joschi joschi removed this from the 1.1 milestone Sep 26, 2017
@jalogisch
Copy link
Contributor Author

as the processing switches now to Lookup Tables and Processing Pipelines that handling is total different now and this can be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lennartkoopmann @joschi @jalogisch