Skip to content

Question: Can you preserve original log message? #82

@Matt-Dee

Description

@Matt-Dee

I'm trying to create something that auto generates groks and then dynamically creates a fluentd.conf file based on the discovered groks.

My question is. Can you preserve the original log message that the grok filter parses and add that as a field to your final record?

<source>
  @type tail
  path /root/Hadoop-2k
  tag hadoop_grok_log
  <parse>
    @type none
  </parse>
</source> 

<filter **>
    @type parser
    key_name message
    <parse>
        @type grok
        grok_name_key grok_name
        grok_failure_key grokfailure

        <grok>
          name a1946ed9-1075-3216-a812-59dd49140d9b
          pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[RMCommunicator Allocator\] %{JAVACLASS:javaOne}: Cannot assign container Container: \[ContainerId: .*NodeId: %{HOSTPORT:hostOne}, NodeHttpAddress: %{HOSTPORT:hostTwo}, Resource: <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}>, Priority: .*Token: Token { kind: ContainerToken, service: %{HOSTPORT:hostFive} }, \] for a map as either .*memory less than required <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}> or no pending map tasks - maps\.isEmpty=true
         </grok>

        <grok>
          name 1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e
          pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[IPC Server handler %{POSINT:numberOne} on %{POSINT:numberTwo}\] %{JAVACLASS:javaOne}: Diagnostics report from .*%{LOGLEVEL:loglevelTwo}: %{JAVACLASS:javaTwo}: No Route to Host from .*%{IPV4:hostTwo} to %{HOSTPORT:hostOne} failed on socket timeout exception: %{JAVACLASS:javaTwo}: No route to host: no further information; For more details see:
        </grok>
    </parse>
</filter>

<match **>
  @type stdout
</match>

What is captured:

{"timeOne":"2015-10-18 18:04:10,002","loglevelOne":"INFO","javaOne":"org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator","hostOne":"MSRA-SA-39.fareast.corp.microsoft.com:28345","hostTwo":"MSRA-SA-39.fareast.corp.microsoft.com:8042","hostThree":"memory:1024","hostFour":"vCores:1","hostFive":"172.22.149.145:28345","grok_name":"a1946ed9-1075-3216-a812-59dd49140d9b"}
{"timeOne":"2015-10-18 18:06:26,029","loglevelOne":"INFO","numberOne":"13","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"}
 {"timeOne":"2015-10-18 18:06:28,217","loglevelOne":"INFO","numberOne":"4","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"}

What would I add to my configuration to capture the original log message for the parsed logs?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions