The Log4J Appender for Amazon Kinesis enables customers to publish logs from their Java applications into an Amazon Kinesis stream. We also provide a publisher application which uses the Log4J Appender for Amazon Kinesis to publish data from a file into an Amazon Kinesis stream.
The Log4J Appender for Amazon Kinesis:
- Buffers log messages in memory
- Uses the Amazon Kinesis AsyncClient from the AWS SDK for Java
- Uses a ThreadPoolExecutor backed by a LinkedBlockingQueue. In most cases, this results in the log message simply being added to the in-memory queue. However, if the queue is full, the logging call will block until there is room in the buffer (e.g. some log entries are successfully sent to Amazon Kinesis and removed from the queue).
- Uses an exponential backoff strategy with a configurable number of maxRetries in case of failures.
If a message is not successfully sent to Amazon Kinesis even after the retry attempts, then it is sent to the fallback handler (and no longer sent to Amazon Kinesis).
Note that the current implementation doesn't insert records in the same order as they are logged. One way to get ordering is by including a timestamp in the log entries and then sorting them at the consumer side of the Amazon Kinesis stream.
| Configuration | Default | Description |
|---|---|---|
| log4j.appender.[APPENDER_NAME].streamName | Stream name to which data is to be published | |
| log4j.appender.[APPENDER_NAME].encoding | UTF-8 | Encoding used to convert log message strings into bytes before sending |
| log4j.appender.[APPENDER_NAME].maxRetries | 3 | Maximum number of retries when calling Kinesis APIs to publish a log message. |
| log4j.appender.[APPENDER_NAME].threadCount | 20 | Number of parallel threads for publishing logs to configured Kinesis stream |
| log4j.appender.[APPENDER_NAME].bufferSize | 2000 | Maximum number of outstanding log messages to keep in memory |
| log4j.appender.[APPENDER_NAME].shutdownTimeout | 30 | Seconds to send buffered messages before application JVM quits normally |
| log4j.appender.[APPENDER_NAME].endpoint | kinesis.us-east-1.amazonaws.com | Amazon Kinesis endpoint to make requests to, if configured this overrides default endpoint for the configured region |
| log4j.appender.[APPENDER_NAME].region | us-east-1 | Amazon Kinesis endpoint in this configured region will be used for making requests unless overridden by log4j.appender.[APPENDER_NAME].endpoint |
We recommend that customers:
- Configure parameters such as buffer size, number of threads, and JVM heap size based on the needs of their use case.
- Configure an additional appender (for example: DailyRollingFileAppender), so that they will retain logs even in the rare cases when the log entries may not be successfully sent to Amazon Kinesis.
- Pick a shutdownTimeout that allows the appender to send all outstanding log entries to Amazon Kinesis before exiting.
-
Download and install Apache Maven 3.x
-
Clone this github repository to a new directory (e.g. by default, this is
kinesis-log4j-appender) by using followinggitcommand on shell prompt:git clone https://github.com/awslabs/kinesis-log4j-appender
This would download the repository to
kinesis-log4j-appenderdirectory. -
On the shell prompt, run
mvn clean installinkinesis-log4j-appenderdirectory and wait for packages to build. If the build is successful, you should see a message like the following:------ [INFO] ------ [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.091s [INFO] Finished at: Wed Feb 12 17:35:03 GMT 2014 [INFO] Final Memory: 52M/185M [INFO] ------------------------------------------------------------------------
This command creates a
targetdirectory insidekinesis-log4j-appenderdirectory.
-
After building this repository, copy kinesis-log4j-appender-1.0.0.jar from
targetdirectory to some other directory (such as KinesisLoggingTest). -
Create an Amazon Kinesis stream (such as testStream).
-
Copy sample log4j configuration file from here and save it as log4j.properties in KinesisLoggingTest directory.
-
Create the credentials file, AwsCredentials.properties in KinesisLoggingTest directory. It must contain following keys:
accessKey=<AWS_ACCESS_KEY> secretKey=<AWS_SECRET_KEY>
-
Run the following command on command prompt from KinesisLoggingTest directory:
On Mac OS X/Linux/Unix:
${JAVA_HOME}/bin/java -cp .:./kinesis-log4j-appender-1.0.0.jar com.amazonaws.services.kinesis.log4j.FilePublisher <path_to_sample_log_file>
On Windows:
%JAVA_HOME%/bin/java -cp .;./kinesis-log4j-appender-1.0.0.jar com.amazonaws.services.kinesis.log4j.FilePublisher <path_to_sample_log_file>
If Java is already in the environment configuration, you can use the following command directly (without reference to JAVA_HOME):
java -cp ...
The application begins publishing records to the configured Amazon Kinesis stream, with the following output:
INFO [main] (FilePublisher.java:64) - Started reading: access_log_1 [continues...] DEBUG [main] (FilePublisher.java:79) - Total 6100 records written to logger DEBUG [main] (FilePublisher.java:79) - Total 6200 records written to logger DEBUG [main] (FilePublisher.java:79) - Total 6300 records written to logger DEBUG [main] (FilePublisher.java:79) - Total 6400 records written to logger DEBUG [main] (FilePublisher.java:79) - Total 6500 records written to logger DEBUG [main] (FilePublisher.java:79) - Total 6600 records written to logger [continues...] DEBUG [pool-1-thread-12] (AsyncPutCallStatsReporter.java:62) - Appender (KINESIS) made 12000 successful put requests out of total 12000 in 1 minute, 14 seconds and 250 milliseconds since start [continues...]