Amazon Kinesis Agent
The Amazon Kinesis Agent is a stand-alone Java software application that offers an easier way to collect and ingest data into Amazon Kinesis services, including Amazon Kinesis Streams and Amazon Kinesis Firehose.
- Monitors file patterns and sends new data records to delivery streams
- Handles file rotation, checkpointing, and retry upon failure
- Delivers all data in a reliable, timely, and simpler manner
- Emits Amazon CloudWatch metrics to help you better monitor and troubleshoot the streaming process
- Sign up for AWS — Before you begin, you need an AWS account. For more information about creating an AWS account and retrieving your AWS credentials, see AWS Account and Credentials in the AWS SDK for Java Developer Guide.
- Sign up for Amazon Kinesis — Go to the Amazon Kinesis console to sign up for the service and create a Amazon Kinesis stream or Firehose delivery stream. For more information, see Create an Amazon Kinesis Stream in the Amazon Kinesis Streams Developer Guide or Create an Amazon Kinesis Firehose Delivery Stream in the Amazon Kinesis Firehose Developer Guide.
- Minimum requirements — To start the Amazon Kinesis Agent, you need Java 1.7+.
- Using the Amazon Kinesis Agent — For more information about using the Amazon Kinesis Agent to deliver data to Streams and Firehose, see Writing to Amazon Kinesis with Agents and Writing to Delivery Streams with Agents.
Installing Amazon Kinesis Agent
After you've downloaded the code from GitHub, you can install the Amazon Kinesis Agent with the following command:
# Optionally, you can set DEBUG=1 in your environment to enable massively # verbose output of the script sudo ./setup --install
This setup script downloads all the dependencies and bootstraps the environment for running the Java program.
Configuring and starting Amazon Kinesis Agent
After the agent is installed, the configuration file can be found in
/etc/aws-kinesis/agent.json. You need to modify this configuration file to set the data destinations and AWS credentials, and to point the agent to the files to push. After you complete the configuration, you can make the agent start automatically at system startup with the following command:
sudo chkconfig aws-kinesis-agent on
If you do not want the agent running at system startup, turn it off with the following command:
sudo chkconfig aws-kinesis-agent off
To start the agent manually, use the following command:
sudo service aws-kinesis-agent start
You can make sure the agent is running with the following command:
sudo service aws-kinesis-agent status
You may see messages such as
aws-kinesis-agent (pid [PID]) is running...
To stop the agent, use the following command:
sudo service aws-kinesis-agent stop
Viewing the Amazon Kinesis Agent log file
The agent writes its logs to
Uninstalling Amazon Kinesis Agent
To uninstall the agent, use the following command:
sudo ./setup --uninstall
Building from Source
The installation done by the setup script is only tested on the following OS Disributions:
- Red Hat Enterprise Linux version 7 or later
- Amazon Linux AMI version 2015.09 or later
- Ubuntu Linux version 12.04 or later
- Debian Linux version 8.6 or later
For other distributions or platforms, you can build the Java project with the following command:
sudo ./setup --build
or by using Ant target as you would build any Java program:
If you use Ant command, you need to download all the dependencies listed in pom.xml before building the Java program. DEPENDENCY_DIR is the directory where you download and store the dependencies. By default, the Amazon Kinesis Agent reads the configuration file from /etc/aws-kinesis/agent.json. You need to create such a file if it does not already exist. A sample configuration can be found at ./configuration/release/aws-kinesis-agent.json
To start the program, use the following command:
java -cp CLASSPATH "com.amazon.kinesis.streaming.agent.Agent"
CLASSPATH is the classpath to your dependencies and the target JAR file that you built from the step above.
Release 1.1 (April 11, 2016)
- Pre-process data before sending it to destinations — Amazon Kinesis Agent now supports to pre-process the records parsed from monitored files before sending them to your streams. The processing capability can be enabled by adding dataProcessingOptions configuration to file flow. There are three available options for now: SINGLELINE, CSVTOJSON, and LOGTOJSON. For more information, see Writing to Amazon Kinesis with Agents and Writing to Delivery Streams with Agents.
- Ingore tailing compressed files — Compressed file extensions, e.g. .gz, .bz2, and .zip, are ignored for tailing.
- Force to kill the program on out-of-memory error — The program will be killed when it's out of memory.
Release 1.0 (October 7, 2015)
- This is the first release.