Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem like #24 #211

Closed
akiontke opened this issue Sep 27, 2017 · 4 comments
Closed

Problem like #24 #211

akiontke opened this issue Sep 27, 2017 · 4 comments
Assignees
Milestone

Comments

@akiontke
Copy link

@akiontke akiontke commented Sep 27, 2017

Problem description

Can't save a rule with ( in grok pattern

Steps to reproduce the problem

  1. Create Rule
rule "extract_mac"
when
  contains(value: to_string($message.full_message), search: "DHCPREQUEST")
then
    let matches = grok(pattern: "DHCPREQUEST for %{IPV4:client_ip} from %{COMMONMAC:client_mac} \\(%{NOTSPACE:client_name}\\)", value: to_string($message.full_message));
    set_fields(matches);
end
  1. Try to save

Environment

  • Graylog Version: 2.3.1
  • Pipeline Processor plugin version: 2.3.1
  • Elasticsearch Version:
  • MongoDB Version:
  • Operating System: Debian 9
  • Browser version: Google Chrome Version 61.0.3163.100 (Official Build) (64-bit)
@joschi
Copy link
Contributor

@joschi joschi commented Sep 27, 2017

@akiontke Please check the logs of your Graylog node(s) and attach the complete error messages which occur when trying to save the rule.

@akiontke
Copy link
Author

@akiontke akiontke commented Sep 28, 2017

@joschi I get the following error message with the mentioned example.

2017-09-28T10:56:33.600+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.NullPointerException: null
        at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.lambda$getConstantArgs$0(FunctionArgs.java:57) ~[?:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[?:1.8.0_102]
        at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1691) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_102]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_102]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_102]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_102]
        at org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs.getConstantArgs(FunctionArgs.java:59) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.functions.Function.preprocessArgs(Function.java:54) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression.<init>(FunctionExpression.java:44) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleAstBuilder.exitFunctionCall(PipelineRuleParser.java:411) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.RuleLangParser$FunctionCallContext.exitRule(RuleLangParser.java:1434) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:170) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:135) ~[?:?]
        at org.graylog.plugins.pipelineprocessor.rest.RuleResource.update(RuleResource.java:174) ~[?:?]
        at sun.reflect.GeneratedMethodAccessor415.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_102]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
@joschi
Copy link
Contributor

@joschi joschi commented Sep 28, 2017

@akiontke I just tried to reproduce this issue on Graylog 2.3.1 (Docker image) but everything worked as expected.

create_rule_1

create_rule_2

Please post the contents of the System / Nodes / Details page ("Installed plugins" specifically). Maybe you're running an old/incompatible version of the Pipeline Processor Plugin?

Also check the contents of the System / Grok Patterns page and make sure that all referenced Grok patterns in your rule (IPV4, COMMONMAC, etc.) do exist.

For reference:
docker-compose.yml

version: '2'
services:
  mongodb:
    image: mongo:3
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.2
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  graylog:
    image: graylog/graylog:2.3.1-2
    mem_limit: 4g
    environment:
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      - 9000:9000
@akiontke
Copy link
Author

@akiontke akiontke commented Sep 28, 2017

These are the plugins in use

Name Version Author Description
Anonymous Usage Statistics 2.3.1 Graylog, Inc. A plugin for collecting anonymous usages statistics about Graylog nodes and clusters.  Website
Collector 2.3.1 Graylog, Inc. Collectors plugin  Website
Elastic Beats Input 2.3.1 Graylog, Inc. Input plugin for Elastic Beats (Beats/Lumberjack protocol).  Website
Enterprise Integration Plugin 2.3.1 Graylog, Inc Provides basic integration with Graylog Enterprise  Website
Internal Metrics InfluxDB Reporter 1.4.0 Graylog, Inc. A plugin for reporting internal Graylog metrics to InfluxDB.  Website
MapWidgetPlugin 2.3.1 Graylog, Inc. Map widget for Graylog  Website
NetFlow Plugin 2.3.0-rc.5 Graylog, Inc. Provides NetFlow inputs  Website
Pipeline Processor Plugin 2.3.1 Graylog, Inc Pluggable pipeline processing framework  Website
Slack 2.4.0 Graylog, Inc. Slack plugin to forward messages or write alarms to Slack chat rooms.  Website

All mentioned grok patterns are configured

@joschi joschi self-assigned this Sep 28, 2017
joschi pushed a commit that referenced this issue Sep 28, 2017
Jochen Schalanda
Closes #211
@joschi joschi added this to the 2.4.0 milestone Sep 28, 2017
@ghost ghost added the in progress label Sep 28, 2017
@bernd bernd closed this in dbd7981 Sep 28, 2017
@ghost ghost removed the in progress label Sep 28, 2017
bernd added a commit that referenced this issue Sep 28, 2017
…212)

Closes #211

(cherry picked from commit dbd7981)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants