New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grok Function Always Returns {} in Pipeline #3610

Closed
lystor opened this Issue Mar 14, 2017 · 8 comments

Comments

Projects
None yet
3 participants
@lystor

lystor commented Mar 14, 2017

Hello

It seems grok function doesn't work in Graylog 2.2.2.

  • Graylog Version: graylog2/server:2.2.2-1
  • Elasticsearch Version: 2.4.4
  • MongoDB Version: 3.4

Clean installation with default'All messages' stream, single pipeline, default stage, single rule:

rule "r1 desc"
when
  has_field("message")
then
 let matches = grok(pattern: "%{GREEDYDATA: x}", value: to_string($message.message));
 set_fields(matches);
 
 set_field("grok_matches", matches);
 set_field("y", to_string($message.message));
 set_field("z", 111); 
end

Simulate:

- Stream: All messages
- Message Codec: Raw String
- Raw message: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
- Press "Load Message"

Simulation results:

message: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
y: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
source: 127.0.0.1
timestamp: 2017-03-14T15:50:05.545Z
z: 111
grok_matches: {}

As you can see - 'x' field is absent, 'grok_matches' field is {}.

Please fix the bug.

Thank you

@jalogisch

This comment has been minimized.

Member

jalogisch commented Mar 14, 2017

As grok pattern are working and I assume that the space after the : is also in your rule you should fix that and then this should work.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

@jalogisch jalogisch closed this Mar 14, 2017

@lystor

This comment has been minimized.

lystor commented Mar 14, 2017

I have removed space.

Result: no effect, same behavior.

rule "r1 desc"
when
  has_field("message")
then
 let matches = grok(pattern: "%{GREEDYDATA:x}", value: to_string($message.message));
 set_fields(matches);
 
 set_field("grok_matches", to_string(matches));
 set_field("y", to_string($message.message));
 set_field("z", 222); 
end

Added fields:

grok_matches: {}
y: 157.55.39.201 2017-03-10T03:19:43+02:00 "xxx.xxx" "GET /xxx/xxx-xxx/ HTTP/1.1" 200 6418 "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
z: 222
@joschi

This comment has been minimized.

Contributor

joschi commented Mar 14, 2017

@lystor How are you importing the messages for which the Grok extraction fails? Which character set are these messages using?

@lystor

This comment has been minimized.

lystor commented Mar 15, 2017

Hello, joschi

I have created minimal testcase.

Requirements:

Steps:

  1. Create docker-compose.yml:
mongo:
  image: mongo:3.4

elasticsearch:
  image: elasticsearch:2.4
  command: elasticsearch -Des.cluster.name='graylog'

graylog:
  environment:
    GRAYLOG_WEB_ENDPOINT_URI: http://192.168.11.31:9000/api/
  image: graylog2/server:2.2.2-1
  links:
    - mongo
    - elasticsearch
  ports:
    - 9000:9000
  1. Start Containers:
    docker-compose up

  2. Login http://192.168.11.31:9000

  3. System > Pipelines > Manage rules > Create rule:

  • Rule source:
rule "r1"
when
  has_field("message")
then
 let matches = grok(pattern: "%{GREEDYDATA:x}", value: to_string($message.message));
 set_fields(matches);
 
 set_field("grok_matches", to_string(matches));
 set_field("y", to_string($message.message));
 set_field("z", 333); 
end
  • Press 'Save'
  1. System > Pipelines > Add new pipeline:
  • Title: p1
  • Press 'Save'
  1. System > Pipelines > p1 > Edit connections:
  • Streams: add 'All messages'
  • Press 'Save'
  1. System > Pipelines > p1 > Stage 0 > Edit:
  • Stage rules: add r1
  • Press 'Save'
  1. System > Pipelines > Simulate processing:
  • Stream: All messages
  • Raw message: 111111111 222222222222 333333333333
  • Message codec: Raw string
  • Press 'Load message'

Simulation results:

Added fields:
message: 111111111 222222222222 333333333333
source: 127.0.0.1
timestamp: 2017-03-15T09:08:59.688Z
y: 111111111 222222222222 333333333333
z: 333
grok_matches: {}

Please reopen the bug and fix it.

Thank you

@jalogisch

This comment has been minimized.

Member

jalogisch commented Mar 15, 2017

I just tried your rule and have the following result:

banners_and_alerts_und_graylog_-_simulate_processing

I run current latest stable Graylog 2.2.2+691b4b7 and did not see the issue

@joschi

This comment has been minimized.

Contributor

joschi commented Mar 15, 2017

@lystor Is the used Grok pattern GREEDYDATA available in Graylog? You can check this on the System / Grok Patterns page.

@lystor

This comment has been minimized.

lystor commented Mar 15, 2017

@joschi

Is the used Grok pattern GREEDYDATA available in Graylog?

Sure.

I use OFFICIAL docker image without any configuration options.

The bug is reproducible in 100% cases using provided testcase.

Additional info:

  1. Same Grok patterns works in extractors perfectly;
  2. Grok patterns start to work in pipelines after single Graylog container restart.

Details:

  1. Create docker-compose.yml:

  2. Create Containers:
    # docker-compose up

  3. Add pipeline with grok and simulate

Result: grok in pipeline doesn't work

  1. Restart graylog container:
# docker-compose stop graylog
# docker-compose start graylog
  1. Simulate pipeline

Result: grok in pipeline works perfectly.

@joschi

This comment has been minimized.

Contributor

joschi commented Mar 15, 2017

I could reproduce the problem. Reopening the issue.

@joschi joschi reopened this Mar 15, 2017

@joschi joschi added the bug label Mar 15, 2017

joschi added a commit that referenced this issue Mar 15, 2017

Publish GrokPatternsChangedEvent when using content packs
The BundleImporter logic for creating Grok patterns from a content pack
wasn't publishing a GrokPatternsChangedEvent which caused the GrokService
to invalidate and reload its internal cache.

Fixes #3610

@joschi joschi self-assigned this Mar 15, 2017

@joschi joschi added the in progress label Mar 15, 2017

@joschi joschi added this to the 2.2.3 milestone Mar 15, 2017

@jalogisch jalogisch added the triaged label Mar 27, 2017

@wafflebot wafflebot bot removed the in progress label Mar 29, 2017

dennisoelkers added a commit that referenced this issue Mar 29, 2017

Publish GrokPatternsChangedEvent when using content packs (#3621)
The BundleImporter logic for creating Grok patterns from a content pack
wasn't publishing a GrokPatternsChangedEvent which caused the GrokService
to invalidate and reload its internal cache.

Fixes #3610

joschi added a commit that referenced this issue Mar 29, 2017

Publish GrokPatternsChangedEvent when using content packs (#3621)
The BundleImporter logic for creating Grok patterns from a content pack
wasn't publishing a GrokPatternsChangedEvent which caused the GrokService
to invalidate and reload its internal cache.

Fixes #3610
(cherry picked from commit cdffb53)

dennisoelkers added a commit that referenced this issue Mar 30, 2017

Publish GrokPatternsChangedEvent when using content packs (#3621) (#3669
)

The BundleImporter logic for creating Grok patterns from a content pack
wasn't publishing a GrokPatternsChangedEvent which caused the GrokService
to invalidate and reload its internal cache.

Fixes #3610
(cherry picked from commit cdffb53)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment