kafka appender for logback-access events

[dianedownie]

I am currently using logback-access with tomcat to write access log events to a daily log file.

 <appender name="dailyFileAppender" class="ch.qos.logback.core.rolling.RollingFileAppender
     <file>${catalina.base}/logs/localhost_access_log.txt</file>
     <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <fileNamePattern>localhost_access_log.%d{yyyy-MM-dd}.txt.gz</fileNamePattern>
      </rollingPolicy>
      <encoder>
        <pattern>%h %l %u %t{yyyy-MM-dd HH:mm:ss} "%r" %s %b %D :%i{X-BDS-AuthToken}:%i{X-BDS-CorrelationID} :%i{User-Agent}</pattern>
      </encoder>
    </appender>

I would like to send them directly to kafka and this project seems like a good solution for that problem. Unfortunately, the logback-kafka-appender doesn't support IAccessEvent layout patterns, only ILogEvent layout patterns. It seems like I could get this to work by creating an encoding class derived from KafkaMessageEncoderBase. I'm about to give that a try, but do you know of a better approach or do you already know this is not going to work for some reason?

Thanks for any suggestions,
Diane

[danielwegener]

Hey @dianedownie, thanks for this question.

A custom implementation of KafkaMessageEncoderBase would not work since its generic parameter is restricted in its type parameter to sub-types of ILoggingEvent. IAccessEvent unfortunately does not extent this interface. This limitation is currently required because some of the keying strategies actually require a real ILoggingEvent to determine a key.

I wasn't even aware of IAccessEvent but I think its a good idea to make logback-kafka-appender as generic as the logback-core infrastructure.

After these changes, you could use the following configuration for logback-access:

<appender name="kafkaAppender" class="your.package.access.KafkaAppender">
  <encoder class="com.github.danielwegener.logback.kafka.encoding.PatternLayoutKafkaMessageEncoder">  
    <layout class="ch.qos.logback.access.PatternLayout">
      <pattern>%h %l %u %t{yyyy-MM-dd HH:mm:ss} "%r" %s %b %D :%i{X-BDS-AuthToken}:%i{X-BDS-CorrelationID} :%i{User-Agent}</pattern>
    </layout>
  </encoder>
   <!-- kafka configuration -->
</appender>

I'll set up a branch with some initial experiments and have a look at the weekend.

[dianedownie]

Thanks much for your quick response and creating this branch. I will start working my extensions there.
Diane

[danielwegener]

I hope you won't need any extensions once this issue is solved :)

[dianedownie]

Fabulous! You did all the hard work. :-)

15:41:26,851 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [com.github.danielwegener.logback.kafka.KafkaAppender]
15:41:26,854 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [kafkaAppender]
15:41:26,870 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@30:26 - no applicable action for [charset], current ElementPath  is [[configuration][appender][encoder][layout][charset]]
15:41:26,871 |-INFO in com.github.danielwegener.logback.kafka.encoding.LayoutKafkaMessageEncoder@49daafb9 - No charset specified for PatternLayoutKafkaEncoder. Using default UTF8 encoding.
15:41:26,877 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [dailyFileAppender] to ch.qos.logback.access.tomcat.LogbackValve[localhost]
15:41:26,877 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [kafkaAppender] to ch.qos.logback.access.tomcat.LogbackValve[localhost]
15:41:26,877 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
15:41:26,878 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@66f42d3 - Registering current configuration as safe fallback point
:
15:46:51.139 [kafka-producer-network-thread | xxxxxx-CONTEXT_NAME_IS_UNDEFINED-accesslog] DEBUG o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node 1
15:46:51.139 [kafka-producer-network-thread | bds00931.bds-ad.lc-CONTEXT_NAME_IS_UNDEFINED-accesslog] DEBUG o.apache.kafka.clients.NetworkClient - Sending metadata request ClientRequest(expectResponse=true, payload=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=3,client_id=xxxxxx-CONTEXT_NAME_IS_UNDEFINED-accesslog}, body={topics=[access_test]})) to node 1

I have that one error to track down although I have not seen any ill effects in the logging.
Diane

[danielwegener]

Can you show your current configuration? the log looks like as if you put a charset-element into the layout-element while it should be one level up in the encoder-element.

[dianedownie]

I was seeing warning about that...so I tried specifying the charset in multiple places...

     <!-- send access log directory to Kafka.  This effectively replaces logstash as a log collector -->
     <appender name="kafkaAppender" class="com.github.danielwegener.logback.kafka.KafkaAppender" charset="UTF-8">
            <!-- This is the default encoder that encodes every log message to an utf8-encoded string  -->

            <encoder class="com.github.danielwegener.logback.kafka.encoding.LayoutKafkaMessageEncoder" charset="UTF-8">
                <layout class="ch.qos.logback.access.PatternLayout">
                <charset>UTF-8</charset>                
                    <pattern>"message":"%m","host":"${HOSTNAME}","client":"%h","date":"%t{yyyy-MM-dd HH:mm:ss}","verb":"%r","request":"%r","httpversion":"%reqAttribute{protocol}","response":"%s","bytes":"%b","requestTime":"%D","AuthToken":"%i{X-BDS-AuthToken}","CorrelationID":"%i{X-BDS-CorrelationID}","User-Agent":"%i{User-Agent}"</pattern>
                </layout>
            </encoder>

            <topic>kbmatch_access_test</topic>

            <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.RoundRobinKeyingStrategy" />

            <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy" />

But this config doesn't generate the error, so thanks for that tip.

     <appender name="kafkaAppender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
            <!-- This is the default encoder that encodes every log message to an utf8-encoded string  -->

            <encoder class="com.github.danielwegener.logback.kafka.encoding.LayoutKafkaMessageEncoder">
                <charset>UTF-8</charset>                
                <layout class="ch.qos.logback.access.PatternLayout">
                    <pattern>"message":"%m","host":"${HOSTNAME}","client":"%h","date":"%t{yyyy-MM-dd HH:mm:ss}","verb":"%r","request":"%r","httpversion":"%reqAttribute{protocol}","response":"%s","bytes":"%b","requestTime":"%D","AuthToken":"%i{X-BDS-AuthToken}","CorrelationID":"%i{X-BDS-CorrelationID}","User-Agent":"%i{User-Agent}"</pattern>
                </layout>
            </encoder>

            <topic>kbmatch_access_test</topic>

            <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.RoundRobinKeyingStrategy" />

            <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy" />

[danielwegener]

Cool, thanks for testing! We'll release this with the upcoming 0.1.0

[dianedownie]

I'm looking forward to the release of 0.1.0. Do you have an estimated timeframe for that?
Diane

[danielwegener]

I thought about having a look at #16 (which probably will introduce another breaking change) next weekend, and then roll up the 0.1.0.

[dianedownie]

Ok, thanks for letting me know.

[danielwegener]

@dianedownie There we go!

[dianedownie]

Thanks much for the quick turnaround, @danielwegener . I pulled 0.1.0 and it's working great with my configuration.