omnibus-td-agent is for td-agent 2 and td-agent 3. For td-agent 4, check td-agent-builder repository.
The event collector daemon, for Treasure Data. This daemon collects various types of logs/events via various way, and transfer them to the cloud. For more about Treasure Data, see the official website, and the documentation.
td-agent is open sourced as Fluentd project. In other words, td-agent is a distribution package of Fluentd.
td-agent package is based on Omnibus
If you want to know the td-agent changelog, check td-agent changelog article.
We'll assume you have Ruby 2.2 and Bundler installed. First ensure all required gems are installed and ready to use:
$ bundle install --binstubs
At first, you should download dependent gems using downloder. This is for avoding broken gem download and reduce the build time by avoiding internet access.
$ bin/gem_downloader core_gems.rb $ bin/gem_downloader plugin_gems.rb
Create required directory and add permission
$ sudo mkdir -p /opt/td-agent /var/cache/omnibus $ sudo chown [USER] /opt/td-agent $ sudo chown [USER] /var/cache/omnibus
After that, you create a platform-specific package using the
build project command:
$ bin/omnibus build td-agent3
The platform/architecture type of the package created will match the platform
build project command is invoked. So running this command on say a
MacBook Pro will generate a Mac OS X specific package. After the build
completes packages will be available in
Build td-agent 2.5
BUILD_TD_AGENT_VERSION=2_5 to each command.
$ BUILD_TD_AGENT_VERSION=2_5 bin/gem_downloader core_gems.rb $ BUILD_TD_AGENT_VERSION=2_5 bin/gem_downloader plugin_gems.rb $ BUILD_TD_AGENT_VERSION=2_5 bin/gem_downloader ui_gems.rb
td-agent2_5 instead of
td-agent3 for omnibus command.
$ bin/omnibus build td-agent2_5
Build on CentOS 5
td-agent build doesn't work on CentOS 5 by OpenSSL related issues. Follow this setup instruction before use omnibus: https://gist.github.com/repeatedly/97d4746e83a5ec135abf3eb77f46ff30
You can clean up all temporary files generated during the build process with
$ bin/omnibus clean
--purge purge option removes ALL files generated during the
build including the project install directory (
the package cache directory (
$ bin/omnibus clean --purge
Full help for the Omnibus command line interface can be accessed with the
$ bin/omnibus help
Build with customized plugins, files and package name
Follow steps below to build packages with customized gem lists, configuration files and original package name.
- make plugin list for your own environment
your_plugin_gems.rbto define the plugin list for your package
- make your default configuration file
- comment out
td-agent-uiif you want not to install them
- build package by
bin/omnibus build YOUR_PACKAGE_NAME
- test your package file
Build script generates file paths with
YOUR_PACKAGE_NAME, from templates. Leave file names with
project_name in Vagrantfile if required.
Vagrant-based Virtualized Build Lab
- CentOS 6.9 64-bit
- CentOS 7.2 64-bit
- Ubuntu 14.04 64-bit
- Ubuntu 16.04 64-bit
- Ubuntu 18.04 64-bit
- Debian 9.3 64-bit
- Debian 10.0 64-bit
NOTE: CentOS 8.0 is not supported because chef client has a problem. For CentOS8, need to build the package on CentOS8 environment.
Please note this build-lab is only meant to get you up and running quickly; there's nothing inherent in Omnibus that restricts you to just building CentOS or Ubuntu packages. See the Vagrantfile to add new platforms to your build lab.
The only requirements for standing up this virtualized build lab are:
- VirtualBox - native packages exist for most platforms and can be downloaded from the VirtualBox downloads page.
- Vagrant 1.2.1+ - native packages exist for most platforms and can be downloaded from the Vagrant downloads page.
$ vagrant plugin install vagrant-berkshelf $ vagrant plugin install vagrant-omnibus $ vagrant plugin install vagrant-vbguest
berks command to setup cookbooks
$ berks vendor cookbooks
Once the pre-requisites are installed you can build your package across all platforms with the following command:
$ vagrant up
If you would like to build a package for a single platform the command looks like this:
$ vagrant up PLATFORM
The complete list of valid platform names can be viewed with the
vagrant status command.
Amazon Linux build
You need to install vagrant-aws 0.5.0 and add
--provider option to
AWS_SSH_KEY_PATH=/path/to/your_aws_key_file vagrant up amazon --provider=aws
After build package, you need to copy rpm file from ec2 instances. No automatic sync for now.
You can build all environments via
After that, each package is stored in
% ls td_agent2_pkg/ centos-6.7/ centos-7.2/ debian-7.10/ debian-8.4/ ubuntu-12.04/ ubuntu-14.04/ ubuntu-16.04/
% vagrant --version Vagrant 1.7.4 % VBoxHeadless --version Oracle VM VirtualBox Headless Interface 5.0.4 (C) 2008-2015 Oracle Corporation All rights reserved. 5.0.4r102546 % ruby --version ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]
Vagrant syncs current directory in each platform. Downloaded gems are also installed automatically.
And you should not use
rbenv local in project root because Ruby environment is built on top of rbenv in Vagrant.
So if you set different Ruby verion in
.ruby-version, running ruby code will fail during pacakging process.
Kitchen-based Build Environment
Every Omnibus project ships will a project-specific
Berksfile and Vagrantfile
that will allow you to build your omnibus projects on all of the projects listed
.kitchen.yml. You can add/remove additional platforms as needed by
changing the list found in the
platforms YAML stanza.
This build environment is designed to get you up-and-running quickly. However, there is nothing that restricts you to building on other platforms. Simply use the omnibus cookbook to setup your desired platform and execute the build steps listed above.
The default build environment requires Test Kitchen and VirtualBox for local development. Test Kitchen also exposes the ability to provision instances using various cloud providers like AWS, DigitalOcean, or OpenStack. For more information, please see the Test Kitchen documentation.
Once you have tweaked your
.kitchen.local.yml) to your
liking, you can bring up an individual build environment using the
$ bundle exec kitchen converge default-ubuntu-1204
Then login to the instance and build the project as described in the Usage section:
$ bundle exec kitchen login default-ubuntu-1204 [vagrant@ubuntu...] $ source load-omnibus-toolchain.sh [vagrant@ubuntu...] $ cd omnibus-td-agent [vagrant@ubuntu...] $ bundle install [vagrant@ubuntu...] $ ... [vagrant@ubuntu...] $ sudo install -d /opt/td-agent -o vagrant [vagrant@ubuntu...] $ ./bin/omnibus build td-agent3
For a complete list of all commands and platforms, run
kitchen list or
Treasure Agent package is released under the Apache2 license.