-
Notifications
You must be signed in to change notification settings - Fork 33
Closed
Description
I'm trying to create something that auto generates groks and then dynamically creates a fluentd.conf file based on the discovered groks.
My question is. Can you preserve the original log message that the grok filter parses and add that as a field to your final record?
<source>
@type tail
path /root/Hadoop-2k
tag hadoop_grok_log
<parse>
@type none
</parse>
</source>
<filter **>
@type parser
key_name message
<parse>
@type grok
grok_name_key grok_name
grok_failure_key grokfailure
<grok>
name a1946ed9-1075-3216-a812-59dd49140d9b
pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[RMCommunicator Allocator\] %{JAVACLASS:javaOne}: Cannot assign container Container: \[ContainerId: .*NodeId: %{HOSTPORT:hostOne}, NodeHttpAddress: %{HOSTPORT:hostTwo}, Resource: <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}>, Priority: .*Token: Token { kind: ContainerToken, service: %{HOSTPORT:hostFive} }, \] for a map as either .*memory less than required <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}> or no pending map tasks - maps\.isEmpty=true
</grok>
<grok>
name 1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e
pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[IPC Server handler %{POSINT:numberOne} on %{POSINT:numberTwo}\] %{JAVACLASS:javaOne}: Diagnostics report from .*%{LOGLEVEL:loglevelTwo}: %{JAVACLASS:javaTwo}: No Route to Host from .*%{IPV4:hostTwo} to %{HOSTPORT:hostOne} failed on socket timeout exception: %{JAVACLASS:javaTwo}: No route to host: no further information; For more details see:
</grok>
</parse>
</filter>
<match **>
@type stdout
</match>
What is captured:
{"timeOne":"2015-10-18 18:04:10,002","loglevelOne":"INFO","javaOne":"org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator","hostOne":"MSRA-SA-39.fareast.corp.microsoft.com:28345","hostTwo":"MSRA-SA-39.fareast.corp.microsoft.com:8042","hostThree":"memory:1024","hostFour":"vCores:1","hostFive":"172.22.149.145:28345","grok_name":"a1946ed9-1075-3216-a812-59dd49140d9b"}
{"timeOne":"2015-10-18 18:06:26,029","loglevelOne":"INFO","numberOne":"13","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"}
{"timeOne":"2015-10-18 18:06:28,217","loglevelOne":"INFO","numberOne":"4","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"}
What would I add to my configuration to capture the original log message for the parsed logs?
Metadata
Metadata
Assignees
Labels
No labels