Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grok support for extractors #377

Closed
henrikjohansen opened this issue Jan 16, 2014 · 13 comments
Closed

Grok support for extractors #377

henrikjohansen opened this issue Jan 16, 2014 · 13 comments
Assignees
Labels
Milestone

Comments

@henrikjohansen
Copy link

@henrikjohansen henrikjohansen commented Jan 16, 2014

When it comes to parsing complex (read:crappy) log formats Jordan Sissel hit a homerun with the concepts and ideas behind Grok.

I have tried plain regex'es and drools but neither can match the ease, speed, and maintainability of grok patterns.

As a bonus it would enable a lot of transparency between Logstash and Graylog2.

Parsing crappy syslog like this :

srvesbmanxxxx api-log: 2014-01-15 14:46:33,382 INFO [cloud.api.ApiServer] (catalina-exec-13:null) (userId=x accountId=x sessionId=3A0E966E2xxxxxx) 10.x.x.x -- GET command=queryAsyncJobResult&jobId=b74d9ce1-0423-4a3e-b460-217787cf0681&response=json&sessionkey=0WfyV4trvb5ng6XrRLCAxHx3KOA%3D&_=1389793597907 200 { "queryasyncjobresultresponse" : {"accountid":"xxxxx-xxxx-11e3-bbe0-235130fb5cd9","userid":"xxxxx-60d0-xxxx-bbe0-235130fb5cd9","cmd":"org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd","jobstatus":1,"jobprocstatus":0,"jobresultcode":0,"jobresulttype":"object","jobresult":{"storagepool":{"id":"xxxxxx-f3ca-3ed7-9ae2-6d79a56d3e90","zoneid":"xxxxxx-99cc-4ab6-a009-04a15d3ccd0a","zonename":"DCE-POC1","podid":"xxxxxxx-b4f7-4994-bd2f-9391e8fbd6d5","podname":"RACK5-DCE_POC1","name":"PR_NP0","ipaddress":"xxx.xxx.xxx.xxx","path":"/ds0002_nfs_fc","created":"2014-01-15T14:46:19+0100","type":"NetworkFilesystem","clusterid":"xxxxx-b113-4dbc-8301-1fd8ce1f5a60","clustername":"xxx.xxx.3.250/DCE_Zone_1/DCE_POC1","disksizetotal":1099511627776,"disksizeallocated":0,"tags":"PR_NP_0","state":"Maintenance","scope":"CLUSTER","jobid":"xxxxx-0423-4a3e-b460-217787cf0681","jobstatus":0}},"created":"2014-01-15T14:46:30+0100","jobid":"xxxx-0423-4a3e-b460-217787cf0681"} }

took 2 minutes using something like this :

%{HOSTNAME:host} %{NOTSPACE:log-type} %{NOTSPACE:date} %{NOTSPACE:time} %{WORD:level} %{NOTSPACE:path} %{NOTSPACE:thread} %{NOTSPACE:userid} %{NOTSPACE:accountid} %{NOTSPACE:sessionid} %{IP:ip} -- %{WORD:method} %{NOTSPACE:command} %{INT:status_code} %{GREEDYDATA:request}
@hggh
Copy link

@hggh hggh commented Feb 6, 2014

👍

2 similar comments
@bonJoeV
Copy link

@bonJoeV bonJoeV commented Feb 9, 2014

+1

@ghost
Copy link

@ghost ghost commented Feb 17, 2014

+1

@jpmens
Copy link

@jpmens jpmens commented Apr 15, 2014

+∞

@sebclick
Copy link

@sebclick sebclick commented May 13, 2014

+1

@timukas
Copy link

@timukas timukas commented May 18, 2014

Grok conversion (int and float) could be also implemented. E.g. %{NUMBER:num:int} or %{NUMBER:amount:float}

@r-duran
Copy link
Contributor

@r-duran r-duran commented Nov 20, 2014

+1

@joschi joschi removed this from the mercury milestone Dec 8, 2014
@henrikjohansen henrikjohansen removed this from the mercury milestone Dec 8, 2014
@kroepke kroepke added this to the 0.93 milestone Jan 8, 2015
@kroepke kroepke added the processing label Jan 8, 2015
@kroepke kroepke self-assigned this Jan 8, 2015
@kroepke
Copy link
Member

@kroepke kroepke commented Jan 8, 2015

kroepke added a commit that referenced this issue Jan 8, 2015
remove obsolete pattern string

#377
kroepke added a commit that referenced this issue Jan 8, 2015
…e result fields, even if they happen to produce just one

#377
kroepke added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 8, 2015
those aren't supported right now so we turn them off

Graylog2/graylog2-server#377
kroepke added a commit that referenced this issue Jan 8, 2015
avoids npe in resource
#377
kroepke added a commit that referenced this issue Jan 9, 2015
kroepke added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 9, 2015
…do anything with them

* not showing is enough because the rest api won't give out any information and the page would be empty anyway,
  so it's not necessary to make the controller inaccessible

Graylog2/graylog2-server#377
@sebclick
Copy link

@sebclick sebclick commented Jan 12, 2015

Can you add default's patterns (as logstash do : https://github.com/elasticsearch/logstash/tree/v1.4.0/patterns) ?

kroepke added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 13, 2015
  * more minor layout cleanup
  * some other minor text changes

Graylog2/graylog2-server#377
@kroepke
Copy link
Member

@kroepke kroepke commented Jan 14, 2015

@sebclick we will likely not ship with default patters, but you can now import pattern files with two clicks.

that will be available in beta.2

@sebclick
Copy link

@sebclick sebclick commented Jan 15, 2015

@kroepke Nice 😄

@kroepke
Copy link
Member

@kroepke kroepke commented Jan 20, 2015

has been implemented in way too many commits.

@kroepke kroepke closed this Jan 20, 2015
@razvanphp
Copy link
Contributor

@razvanphp razvanphp commented Feb 26, 2015

Brilliant feature, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.