New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data loss using cidr_match() in rule with IPv6 address vs IPv4 subnet #5405

Closed
mbunkus opened this Issue Dec 13, 2018 · 3 comments

Comments

Projects
None yet
4 participants
@mbunkus
Copy link

mbunkus commented Dec 13, 2018

The short version is: testing an IPv6 address against an IPv4 subnet with cidr_match() results in an exception being thrown and the message being lost.

I have a pipeline that tests an IP address (e.g. source address of an SSH login attempt) against various known IPv4 and IPv6 subnets. Think of something like this:

rule "ssh login from known-good addresses"
when cidr_match("10.0.0.0/8", to_ip($message.my_ssh_login_remote_ip))
then
  set_field("my_ssh_login_remote_ip_good", true);
end

This works as expected if my_ssh_login_remote_ip contains a valid IPv4 address or if it contains something that's neither a valid IPv4 nor a valid IPv6 address.

If, however, my_ssh_login_remote_ip contains a valid IPv6 address, the result is that there's a warning written to the server.log file:

2018-12-12T22:50:20.731+01:00 WARN  [ProcessBufferProcessor] Unable to process message <ebd959b1-fe57-11e8-b07f-000c2936a92d>: org.graylog.plugins.pipelineprocessor.ast.exceptions.FunctionEvaluationException: java.lang.IllegalArgumentException: This IPv6 address cannot be used in IPv4 context

Despite the message stating it's only a warning, the result is actually that the message being processed is lost and will not turn up in any stream, not even in All messages.

How to reproduce

Using the pipeline's simulator one can see that an exception is thrown when such a test happens. Here's how to reproduce that exception using the simulator:

  1. Create an empty stream, e.g. cidr_match test
  2. Create a new pipeline, e.g. cidr_match test, and connect it to the stream cidr_match test.
  3. Create a new rule with the content as shown below at [1].
  4. Add the rule to the first stage in the pipe line.
  5. Set up the simulator:
  6. Select the cidr_match test stream.
  7. Select any Syslog-type input.
  8. Select Syslog as the codec.
  9. Run a test with a valid IPv4 address:
  10. Enter the following raw message: <38>1 2018-12-13T12:50:34+01:00 dummy sshd 5420 - - 10.1.2.3
  11. Click on Load message. The simulation will work fine, the field my_cidr_match_test_result will be set to ok. This is good and expected.
  12. Run a test with garbage (something that's neither a valid IPv4 nor a valid IPv6 address):
  13. Enter the following raw message: <38>1 2018-12-13T12:50:34+01:00 dummy sshd 5420 - - dummy
  14. Click on Load message. The simulation will work fine, the field my_cidr_match_test_result will not be set to ok. This is as expected.
  15. Now the test leading to the exception with a valid IPv6 address:
  16. Enter the following raw message: <38>1 2018-12-13T12:50:34+01:00 dummy sshd 5420 - - 2001:db8::1
  17. Click on Load message. The simulation will not finish; the "waiting" spinner will continue to be shown, and the server.log will contain the traceback below under [2].

[1] Rule content:

rule "cidr_match test IPv4 subnet"
when cidr_match("10.0.0.0/8", to_ip($message.message))
then
  set_field("my_cidr_match_test_result", "ok");
end

[2] Traceback from server.log:

2018-12-13T14:27:28.771+01:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
org.graylog.plugins.pipelineprocessor.ast.exceptions.FunctionEvaluationException: java.lang.IllegalArgumentException: This IPv6 address cannot be used in IPv4 context
        at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.evaluateUnsafe(FunctionExpression.java:69) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.expressions.BooleanValuedFunctionWrapper.evaluateBool(BooleanValuedFunctionWrapper.java:37) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateRuleCondition(PipelineInterpreter.java:399) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.evaluateStage(PipelineInterpreter.java:299) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.processForResolvedPipelines(PipelineInterpreter.java:263) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter.process(PipelineInterpreter.java:143) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.rest.SimulatorResource.simulate(SimulatorResource.java:87) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_191]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_191]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: java.lang.IllegalArgumentException: This IPv6 address cannot be used in IPv4 context
        at org.jboss.netty.handler.ipfilter.CIDR.getIpV4FromIpV6(CIDR.java:213) ~[graylog.jar:?]
        at org.jboss.netty.handler.ipfilter.CIDR4.ipv4AddressToInt(CIDR4.java:144) ~[graylog.jar:?]
        at org.jboss.netty.handler.ipfilter.CIDR4.contains(CIDR4.java:100) ~[graylog.jar:?]
        at org.graylog.plugins.pipelineprocessor.functions.ips.CidrMatch.evaluate(CidrMatch.java:57) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.functions.ips.CidrMatch.evaluate(CidrMatch.java:30) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.evaluateUnsafe(FunctionExpression.java:63) ~[?:?]
        ... 33 more

The reverse situation (testing an IPv4 address against an IPv6 subnet) works just fine. All three test cases finish in the simulator if I use the following rule:

rule "cidr_match test IPv6 subnet"
when cidr_match("2001:db8::/32", to_ip($message.message))
then
  set_field("my_cidr_match_test_result", "ok");
end

I can work around this problem with a crude "is the address IPv6?" test before the cidr_match(), e.g. like this:

rule "cidr_match test IPv4 subnet"
when (regex(":", to_string($message.message)).matches == false)
  && cidr_match("10.0.0.0/8", to_ip($message.message))
then
  set_field("my_cidr_match_test_result", "ok");
end

An additional problem is the one of data loss due to the exception thrown. But that's most likely a completely separate issue, and I'll open a second issue for it.

Expected Behavior

The mentioned workaround should really not be necessary: the cidr_match() case should simply return false when comparing an IPv6 address object (that doesn't represent an IPv4 address) against an IPv4 subnet object.

Current Behavior

An exception is thrown, and the message is lost.

Your Environment

  • Graylog Version: 2.4.6-1 (Ubuntu package from Graylog's repo)
  • Elasticsearch Version: 5.6.13 (Ubuntu package from Elastico's repo)
  • MongoDB Version: 4.0.4 (Ubuntu package from MongoDB.org's repo)
  • Operating System: Ubuntu 18.04
@kroepke

This comment has been minimized.

Copy link
Member

kroepke commented Dec 19, 2018

Thanks for the report!

This is already fixed in master (which will become 3.0), but luckily the fix for 2.5 is trivial, because the necessary code is there as well.
We'll fix this for 2.5.1

kroepke added a commit to Graylog2/graylog-plugin-pipeline-processor that referenced this issue Dec 19, 2018

properly handle IPv6 addresses in IPv4 subnet check
the class from netty3 improperly threw an exception when checking an unmappable v6 address against a v4 cdir pattern
using the custom subnet class from Graylog fixes these issues

fixes Graylog2/graylog2-server#5405
@mbunkus

This comment has been minimized.

Copy link

mbunkus commented Dec 19, 2018

Great, thanks!

bernd added a commit to Graylog2/graylog-plugin-pipeline-processor that referenced this issue Dec 19, 2018

Properly handle IPv6 addresses in IPv4 subnet check (#254)
the class from netty3 improperly threw an exception when checking an unmappable v6 address against a v4 cdir pattern
using the custom subnet class from Graylog fixes these issues

Fixes Graylog2/graylog2-server#5405
@bernd

This comment has been minimized.

Copy link
Member

bernd commented Dec 19, 2018

This has been fixed in Graylog2/graylog-plugin-pipeline-processor#254 and will be in the upcoming 2.5.1 release.

@bernd bernd closed this Dec 19, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment