Skip to content

/ graylog2-server

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Grok support for extractors #377

Closed
henrikjohansen opened this issue Jan 16, 2014 · 13 comments
Closed

Grok support for extractors #377

henrikjohansen opened this issue Jan 16, 2014 · 13 comments
Assignees
@kroepke
Labels
processing
Milestone
1.0.0

Comments

@henrikjohansen
Copy link

@henrikjohansen henrikjohansen commented Jan 16, 2014

When it comes to parsing complex (read:crappy) log formats Jordan Sissel hit a homerun with the concepts and ideas behind Grok.

I have tried plain regex'es and drools but neither can match the ease, speed, and maintainability of grok patterns.

As a bonus it would enable a lot of transparency between Logstash and Graylog2.

Parsing crappy syslog like this :

srvesbmanxxxx api-log: 2014-01-15 14:46:33,382 INFO [cloud.api.ApiServer] (catalina-exec-13:null) (userId=x accountId=x sessionId=3A0E966E2xxxxxx) 10.x.x.x -- GET command=queryAsyncJobResult&jobId=b74d9ce1-0423-4a3e-b460-217787cf0681&response=json&sessionkey=0WfyV4trvb5ng6XrRLCAxHx3KOA%3D&_=1389793597907 200 { "queryasyncjobresultresponse" : {"accountid":"xxxxx-xxxx-11e3-bbe0-235130fb5cd9","userid":"xxxxx-60d0-xxxx-bbe0-235130fb5cd9","cmd":"org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd","jobstatus":1,"jobprocstatus":0,"jobresultcode":0,"jobresulttype":"object","jobresult":{"storagepool":{"id":"xxxxxx-f3ca-3ed7-9ae2-6d79a56d3e90","zoneid":"xxxxxx-99cc-4ab6-a009-04a15d3ccd0a","zonename":"DCE-POC1","podid":"xxxxxxx-b4f7-4994-bd2f-9391e8fbd6d5","podname":"RACK5-DCE_POC1","name":"PR_NP0","ipaddress":"xxx.xxx.xxx.xxx","path":"/ds0002_nfs_fc","created":"2014-01-15T14:46:19+0100","type":"NetworkFilesystem","clusterid":"xxxxx-b113-4dbc-8301-1fd8ce1f5a60","clustername":"xxx.xxx.3.250/DCE_Zone_1/DCE_POC1","disksizetotal":1099511627776,"disksizeallocated":0,"tags":"PR_NP_0","state":"Maintenance","scope":"CLUSTER","jobid":"xxxxx-0423-4a3e-b460-217787cf0681","jobstatus":0}},"created":"2014-01-15T14:46:30+0100","jobid":"xxxx-0423-4a3e-b460-217787cf0681"} }

took 2 minutes using something like this :

%{HOSTNAME:host} %{NOTSPACE:log-type} %{NOTSPACE:date} %{NOTSPACE:time} %{WORD:level} %{NOTSPACE:path} %{NOTSPACE:thread} %{NOTSPACE:userid} %{NOTSPACE:accountid} %{NOTSPACE:sessionid} %{IP:ip} -- %{WORD:method} %{NOTSPACE:command} %{INT:status_code} %{GREEDYDATA:request}
@hggh
Copy link

@hggh hggh commented Feb 6, 2014

👍
2 similar comments
@bonJoeV
Copy link

@bonJoeV bonJoeV commented Feb 9, 2014

+1
@ghost
Copy link

@ghost ghost commented Feb 17, 2014

+1
@jpmens
Copy link

@jpmens jpmens commented Apr 15, 2014

+∞
@sebclick
Copy link

@sebclick sebclick commented May 13, 2014

+1
@timukas
Copy link

@timukas timukas commented May 18, 2014

Grok conversion (int and float) could be also implemented. E.g. %{NUMBER:num:int} or %{NUMBER:amount:float}
@r-duran
Copy link
Contributor

@r-duran r-duran commented Nov 20, 2014

+1
@joschi joschi removed this from the mercury milestone Dec 8, 2014
@henrikjohansen henrikjohansen removed this from the mercury milestone Dec 8, 2014
@kroepke kroepke added this to the 0.93 milestone Jan 8, 2015
@kroepke kroepke added the processing label Jan 8, 2015
@kroepke kroepke self-assigned this Jan 8, 2015
@kroepke
Copy link
Member

@kroepke kroepke commented Jan 8, 2015

de30ca6
kroepke added a commit that referenced this issue Jan 8, 2015
@kroepke
fix grok tester resource
Loading status checks…
f273e1a 
remove obsolete pattern string

#377
kroepke added a commit that referenced this issue Jan 8, 2015
@kroepke
use the correct target field for extractors which can produce multipl…
Loading status checks…
8856c17 
…e result fields, even if they happen to produce just one

#377
kroepke added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 8, 2015
@kroepke
don't display converters and target field for grok extractor
Loading status checks…
eb08469 
those aren't supported right now so we turn them off

Graylog2/graylog2-server#377
kroepke added a commit that referenced this issue Jan 8, 2015
@kroepke
return empty list for grok tester instead of null
Loading status checks…
7ac45c9 
avoids npe in resource
#377
kroepke added a commit that referenced this issue Jan 9, 2015
@kroepke
use inputs restpermissions for grok pattern resource
Loading status checks…
a730fc7 
#377
kroepke added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 9, 2015
@kroepke
hide grok patterns link if the user won't have enough permissions to …
Loading status checks…
2038d7d 
…do anything with them

* not showing is enough because the rest api won't give out any information and the page would be empty anyway,
  so it's not necessary to make the controller inaccessible

Graylog2/graylog2-server#377
@sebclick
Copy link

@sebclick sebclick commented Jan 12, 2015

Can you add default's patterns (as logstash do : https://github.com/elasticsearch/logstash/tree/v1.4.0/patterns) ?
kroepke added a commit to graylog-labs/graylog2-web-interface that referenced this issue Jan 13, 2015
@kroepke
add file upload for grok pattern files
Loading status checks…
bbbf3c2 
  * more minor layout cleanup
  * some other minor text changes

Graylog2/graylog2-server#377
@kroepke
Copy link
Member

@kroepke kroepke commented Jan 14, 2015

@sebclick we will likely not ship with default patters, but you can now import pattern files with two clicks.

that will be available in beta.2
@sebclick
Copy link

@sebclick sebclick commented Jan 15, 2015

@kroepke Nice 😄
@kroepke kroepke mentioned this issue Jan 20, 2015
Closed
@kroepke
Copy link
Member

@kroepke kroepke commented Jan 20, 2015

has been implemented in way too many commits.
@kroepke kroepke closed this Jan 20, 2015
@razvanphp
Copy link
Contributor

@razvanphp razvanphp commented Feb 26, 2015

Brilliant feature, thank you!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kroepke kroepke

Labels
processing
Projects
None yet
Milestone
1.0.0
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
@joschi @kroepke @jpmens @hggh @henrikjohansen @timukas @r-duran @sebclick @razvanphp @bonJoeV
You can’t perform that action at this time.