Grok Function Always Returns {} in Pipeline

[lystor]

Hello

It seems grok function doesn't work in Graylog 2.2.2.

• Graylog Version: graylog2/server:2.2.2-1
• Elasticsearch Version: 2.4.4
• MongoDB Version: 3.4

Clean installation with default'All messages' stream, single pipeline, default stage, single rule:

rule "r1 desc"
when
  has_field("message")
then
 let matches = grok(pattern: "%{GREEDYDATA: x}", value: to_string($message.message));
 set_fields(matches);
 
 set_field("grok_matches", matches);
 set_field("y", to_string($message.message));
 set_field("z", 111); 
end

Simulate:

- Stream: All messages
- Message Codec: Raw String
- Raw message: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
- Press "Load Message"

Simulation results:

message: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
y: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
source: 127.0.0.1
timestamp: 2017-03-14T15:50:05.545Z
z: 111
grok_matches: {}

As you can see - 'x' field is absent, 'grok_matches' field is {}.

Please fix the bug.

Thank you

[jalogisch]

As grok pattern are working and I assume that the space after the : is also in your rule you should fix that and then this should work.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

[lystor]

I have removed space.

Result: no effect, same behavior.

rule "r1 desc"
when
  has_field("message")
then
 let matches = grok(pattern: "%{GREEDYDATA:x}", value: to_string($message.message));
 set_fields(matches);
 
 set_field("grok_matches", to_string(matches));
 set_field("y", to_string($message.message));
 set_field("z", 222); 
end

Added fields:

grok_matches: {}
y: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
z: 222

[joschi]

@lystor How are you importing the messages for which the Grok extraction fails? Which character set are these messages using?

[lystor]

Hello, joschi

I have created minimal testcase.

Requirements:

• Docker Host: Ubuntu Server 16.04.2 amd64
• Docker CE: 17.03.0~ce-0~ubuntu-xenial
  (installation howto: https://docs.docker.com/engine/installation/linux/ubuntu/)
• Docker Compose: 1.11.2:
  (installation howto: https://github.com/docker/compose/releases)
• Browser: Google Chrome 57 on Windows 8 Ent 64-bit

Steps:

1. Create docker-compose.yml:

mongo:
  image: mongo:3.4

elasticsearch:
  image: elasticsearch:2.4
  command: elasticsearch -Des.cluster.name='graylog'

graylog:
  environment:
    GRAYLOG_WEB_ENDPOINT_URI: http://192.168.11.31:9000/api/
  image: graylog2/server:2.2.2-1
  links:
    - mongo
    - elasticsearch
  ports:
    - 9000:9000

2. Start Containers:
   docker-compose up

3. Login http://192.168.11.31:9000

4. System > Pipelines > Manage rules > Create rule:

• Rule source:

rule "r1"
when
  has_field("message")
then
 let matches = grok(pattern: "%{GREEDYDATA:x}", value: to_string($message.message));
 set_fields(matches);
 
 set_field("grok_matches", to_string(matches));
 set_field("y", to_string($message.message));
 set_field("z", 333); 
end

• Press 'Save'

2. System > Pipelines > Add new pipeline:

• Title: p1
• Press 'Save'

3. System > Pipelines > p1 > Edit connections:

• Streams: add 'All messages'
• Press 'Save'

4. System > Pipelines > p1 > Stage 0 > Edit:

• Stage rules: add r1
• Press 'Save'

5. System > Pipelines > Simulate processing:

• Stream: All messages
• Raw message: 111111111 222222222222 333333333333
• Message codec: Raw string
• Press 'Load message'

Simulation results:

Added fields:
message: 111111111 222222222222 333333333333
source: 127.0.0.1
timestamp: 2017-03-15T09:08:59.688Z
y: 111111111 222222222222 333333333333
z: 333
grok_matches: {}

Please reopen the bug and fix it.

Thank you

[jalogisch]

I just tried your rule and have the following result:

I run current latest stable Graylog 2.2.2+691b4b7 and did not see the issue

[joschi]

@lystor Is the used Grok pattern GREEDYDATA available in Graylog? You can check this on the System / Grok Patterns page.

[lystor]

@joschi

Is the used Grok pattern GREEDYDATA available in Graylog?

Sure.

I use OFFICIAL docker image without any configuration options.

The bug is reproducible in 100% cases using provided testcase.

Additional info:

1. Same Grok patterns works in extractors perfectly;
2. Grok patterns start to work in pipelines after single Graylog container restart.

Details:

1. Create docker-compose.yml:

2. Create Containers:
   # docker-compose up

3. Add pipeline with grok and simulate

Result: grok in pipeline doesn't work

4. Restart graylog container:

# docker-compose stop graylog
# docker-compose start graylog

5. Simulate pipeline

Result: grok in pipeline works perfectly.

[joschi]

I could reproduce the problem. Reopening the issue.