go_kafka_client is now able to send all log and metrics data as structured Avro messages to Kafka for later processing.
go_kafka_client now has an implementation for LogEmitter and MetricsEmitter interfaces that handle log and metrics data, transform them into Avro structures and produce to a given Kafka topic. Both of them can be used to transfer all logs and metrics generated by go_kafka_client consumers but you can also use them as standalone components to emit your log and metrics data.
KafkaLogEmitter also implements KafkaLogger if you wish your logs to go directly to a Kafka topic.
Usage example for KafkaLogEmitter:
emitterConfig := NewKafkaLogEmitterConfig() // creates a new config with log level set to Info
emitterConfig.SchemaRegistryUrl = "http://localhost:8081" // url to Confluent Schema Repository
emitterConfig.Topic = "kafka-logs" // Kafka topic to produce logs to
emitterConfig.Source = "go_kafka_client.log.emitter" // Source string to distinguish multiple log sources
emitterConfig.Tags = map[string]string{"origin": topic} // Optional tags that will be appended to each log message
emitterConfig.ProducerConfig = DefaultProducerConfig() // Embedded producer config
emitterConfig.ProducerConfig.BrokerList = []string{"localhost:9092"} // Broker address is the only required parameter, however you may configure the producer in the way you need. The only exception is the ValueEncoder which will be overridden to be KafkaAvroEncoder
emitter := NewKafkaLogEmitter(emitterConfig)
emitter.Info("I am a log message!") // this message should be sent to a Kafka topic if the log level is high enough
Usage example for CodahaleKafkaReporter:
metricsProducerConfig := DefaultProducerConfig() // Embedded producer config
metricsProducerConfig.BrokerList = []string{"localhost:9092"} // Broker address is the only required parameter, however you may configure the producer in the way you need. The only exception is the ValueEncoder which will be overridden to be KafkaAvroEncoder
reporter := NewCodahaleKafkaReporter("kafka-metrics", "http://localhost:8081", metricsProducerConfig)
go consumer.Metrics().WriteJSON(10 * time.Second, reporter)
// or you may set this reporter standalone
metrics.WriteJSONOnce(yourMetricsRegistry, reporter)