kong-kafka-log

This plugin publishes request and response logs to a Kafka topic or to file/stdout.

Supported Kong Releases

Kong >= 3.7.x

Installation

Recommended:

$ luarocks install kong-kafka-log

Other:

$ git clone https://github.com/Optum/kong-kafka-log.git /path/to/kong/plugins/kong-kafka-log
$ cd /path/to/kong/plugins/kong-kafka-log
$ luarocks make *.rockspec

Configuration

Enabling globally for Kafka Logging

$ curl -X POST http://kong:8001/plugins \
    --data "name=kong-kafka-log" \
    --data "config.bootstrap_servers=localhost:9092" \
    --data "config.topic=kong-log" \
    --data "config.ask_id=MYASKID-00000000" \
    --data "config.app_name=GatewayStageEnvironment" \
    --data "config.timeout=10000" \
    --data "config.keepalive=60000" \
    --data "config.ssl=false" \
    --data "config.ssl_verify=false" \
    --data "config.producer_request_acks=1" \
    --data "config.producer_request_timeout=2000" \
    --data "config.producer_request_limits_messages_per_request=200" \
    --data "config.producer_request_limits_bytes_per_request=1048576" \
    --data "config.producer_request_retries_max_attempts=10" \
    --data "config.producer_request_retries_backoff_timeout=100" \
    --data "config.producer_async=true" \
    --data "config.producer_async_flush_timeout=1000" \
    --data "config.producer_async_buffering_limits_messages_in_memory=50000"

Enabling globally for Stdout Logging

$ curl -X POST http://kong:8001/plugins \
    --data "name=kong-kafka-log" \
    --data "config.log_to_file=true" \
    --data "config.log_to_kafka=false" \
    --data "config.ask_id=MYASKID-00000000" \
    --data "config.app_name=GatewayStageEnvironment" \

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

Form Parameter	default	description
name		The name of the plugin to use, in this case kafka-log
config.log_to_file		Option to log messages to file specified in log_to_file_path
config.log_to_file_path		Location of file to output message to defaults to /dev/stdout
config.log_to_kafka		Option to disable kafka logging
config.bootstrap_servers		List of bootstrap brokers in host:port format
config.topic		Topic to publish to
config.ask_id		ASK Id
config.app_name		Application name using logging utility
config.timeout Optional	10000	Socket timeout in millis
config.keepalive Optional	60000	Keepalive timeout in millis
config.ssl Optional	false	Enable SSL Connection
config.ssl_verify Optional	false	Enable SSL Verification
config.producer_request_acks Optional	1	The number of acknowledgments the producer requires the leader to have received before considering a request complete. Allowed values: 0 for no acknowledgments, 1 for only the leader and -1 for the full ISR
config.producer_request_timeout Optional	2000	Time to wait for a Produce response in millis
config.producer_request_limits_messages_per_request Optional	200	Maximum number of messages to include into a single Produce request
config.producer_request_limits_bytes_per_request Optional	1048576	Maximum size of a Produce request in bytes
config.producer_request_retries_max_attempts Optional	10	Maximum number of retry attempts per single Produce request
config.producer_request_retries_backoff_timeout Optional	100	Backoff interval between retry attempts in millis
config.producer_async Optional	true	Flag to enable asynchronous mode
config.producer_async_flush_timeout Optional	1000	Maximum time interval in millis between buffer flushes in in asynchronous mode
config.producer_async_buffering_limits_messages_in_memory Optional	50000	Maximum number of messages that can be buffered in memory in asynchronous mode

Log Format

{
  "application": {
    "askId": "ASKID000-000000",
    "name": "GatewaySampleInstance"
  },
  "destHost": {
    "port": 443,
    "ipv4": 179347885,
    "path": "/Backend/api/path",
    "hostname": "api-service.company.com"
  },
  "name": "my.api.service.name",
  "device": {
    "vendor": "Optum",
    "ip4": 176365362,
    "hostname": "kong-507-g7rck",
    "product": "kong-kafka-log"
  },
  "receivedTime": 1588059064647,
  "msg": "STARGATE_PROXY_TX",
  "sourceHost": {
    "port": 443,
    "ip4": 179301896
  },
  "logClass": "SECURITY_SUCCESS",
  "request": {
    "out": 675,
    "method": "GET",
    "request": "https://gateway.company.com:443/api/proxy/path/service/v1",
    "Optum_CID_Ext": "27097e07-fa15-4bbb-9a9e-7cea46abc422#1",
    "in": 339
  },
  "eventClass": "200",
  "sourceUser": {
    "tokenHash": "3502a5bc96d4468c6974e8b415eb8899b501b1ca6dc717ee4e07ff655dbaebb6", --> sha256(authentication token)
    "name": "consumer.name"
  }
}

As the log format is custom for our requirements for Kafka logging, you can fork and replace the /src/basic.lua with your desired logging format.

Known issues and limitations

Known limitations:

1. There is no support for Authentication (Would like to see MTLS Auth or SASL written into underlying dependency library)
2. There is no support for message compression

Quickstart

1. Install kong-kafka-log via luarocks:

   luarocks install kong-kafka-log
   

2. Load the kong-kafka-log in Kong:

   KONG_PLUGINS=bundled,kong-kafka-log bin/kong start
   

3. Add kong-kafka-log plugin globally:

   curl -X POST http://localhost:8001/plugins \
       --data "name=kong-kafka-log" \
       --data "config.bootstrap_servers=localhost:9093" \
       --data "config.ask_id=testaskid" \
       --data "config.app_name=gatewayappname" \
       --data "config.ssl=true" \
       --data "config.topic=example-topic"
   

4. As requests are made, check your Kafka instance!

Implementation details

Original source rewritten from kong-plugin-kafka-log by yskopets, Big Thanks!
This plugin makes use of lua-resty-kafka as a dependency. Need version >= v0.09

Maintainers

jeremyjpj0916
rsbrisci