New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem like #24 #211

Closed
akiontke opened this Issue Sep 27, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@akiontke

akiontke commented Sep 27, 2017

Problem description

Can't save a rule with ( in grok pattern

Steps to reproduce the problem

  1. Create Rule
rule "extract_mac"
when
  contains(value: to_string($message.full_message), search: "DHCPREQUEST")
then
    let matches = grok(pattern: "DHCPREQUEST for %{IPV4:client_ip} from %{COMMONMAC:client_mac} \\(%{NOTSPACE:client_name}\\)", value: to_string($message.full_message));
    set_fields(matches);
end
  1. Try to save

Environment

  • Graylog Version: 2.3.1
  • Pipeline Processor plugin version: 2.3.1
  • Elasticsearch Version:
  • MongoDB Version:
  • Operating System: Debian 9
  • Browser version: Google Chrome Version 61.0.3163.100 (Official Build) (64-bit)
@joschi

This comment has been minimized.

Contributor

joschi commented Sep 27, 2017

@akiontke Please check the logs of your Graylog node(s) and attach the complete error messages which occur when trying to save the rule.

@akiontke

This comment has been minimized.

akiontke commented Sep 28, 2017

@joschi I get the following error message with the mentioned example.

2017-09-28T10:56:33.600+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.NullPointerException: null
        at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.lambda$getConstantArgs$0(FunctionArgs.java:57) ~[?:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[?:1.8.0_102]
        at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1691) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_102]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_102]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_102]
        at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.getConstantArgs(FunctionArgs.java:59) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.functions.Function.preprocessArgs(Function.java:54) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.<init>(FunctionExpression.java:44) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleAstBuilder.exitFunctionCall(PipelineRuleParser.java:411) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.RuleLangParser$FunctionCallContext.exitRule(RuleLangParser.java:1434) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:170) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:135) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.rest.RuleResource.update(RuleResource.java:174) ~[?:?]
        at sun.reflect.GeneratedMethodAccessor415.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
@joschi

This comment has been minimized.

Contributor

joschi commented Sep 28, 2017

@akiontke I just tried to reproduce this issue on Graylog 2.3.1 (Docker image) but everything worked as expected.

create_rule_1

create_rule_2

Please post the contents of the System / Nodes / Details page ("Installed plugins" specifically). Maybe you're running an old/incompatible version of the Pipeline Processor Plugin?

Also check the contents of the System / Grok Patterns page and make sure that all referenced Grok patterns in your rule (IPV4, COMMONMAC, etc.) do exist.

For reference:
docker-compose.yml

version: '2'
services:
  mongodb:
    image: mongo:3
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.2
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  graylog:
    image: graylog/graylog:2.3.1-2
    mem_limit: 4g
    environment:
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      - 9000:9000
@akiontke

This comment has been minimized.

akiontke commented Sep 28, 2017

These are the plugins in use

Name Version Author Description
Anonymous Usage Statistics 2.3.1 Graylog, Inc. A plugin for collecting anonymous usages statistics about Graylog nodes and clusters.  Website
Collector 2.3.1 Graylog, Inc. Collectors plugin  Website
Elastic Beats Input 2.3.1 Graylog, Inc. Input plugin for Elastic Beats (Beats/Lumberjack protocol).  Website
Enterprise Integration Plugin 2.3.1 Graylog, Inc Provides basic integration with Graylog Enterprise  Website
Internal Metrics InfluxDB Reporter 1.4.0 Graylog, Inc. A plugin for reporting internal Graylog metrics to InfluxDB.  Website
MapWidgetPlugin 2.3.1 Graylog, Inc. Map widget for Graylog  Website
NetFlow Plugin 2.3.0-rc.5 Graylog, Inc. Provides NetFlow inputs  Website
Pipeline Processor Plugin 2.3.1 Graylog, Inc Pluggable pipeline processing framework  Website
Slack 2.4.0 Graylog, Inc. Slack plugin to forward messages or write alarms to Slack chat rooms.  Website

All mentioned grok patterns are configured

@joschi joschi self-assigned this Sep 28, 2017

joschi added a commit that referenced this issue Sep 28, 2017

@joschi joschi added this to the 2.4.0 milestone Sep 28, 2017

@wafflebot wafflebot bot added the in progress label Sep 28, 2017

@bernd bernd closed this in dbd7981 Sep 28, 2017

@wafflebot wafflebot bot removed the in progress label Sep 28, 2017

bernd added a commit that referenced this issue Sep 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment