Packaging scripts for google-fluentd
Shell Ruby NSIS PowerShell HTML
Pull request Compare This branch is 172 commits ahead, 232 commits behind treasure-data:master.
Failed to load latest commit information.
bats Use same impl to stop td-agent between Debian and RedHat Feb 23, 2016
bin Fetch pre-compiled gem native extensions instead of building them fro… Feb 20, 2018
config Bump google-fluentd version to 1.5.33-1. Jun 28, 2018
docker Bump repo suffix to 20180629-1 in Dockerfile. Jun 29, 2018
lib Fetch pre-compiled gem native extensions instead of building them fro… Feb 20, 2018
resources/td-agent Fix packaging script to build user custom package: Dec 10, 2014
spec Add serverspec file to test Dec 9, 2014
templates Enforce retry limit by default and let users opt in forever retries i… Feb 14, 2018
windows-installer Bump fluent-plugin-google-cloud gem version to 0.6.21. (#96) Jun 28, 2018
.gitignore Add Gemfile.lock to follow working version Mar 30, 2016
.kitchen.yml Update kitchen file from latest omnibus Feb 11, 2016
.travis.yml Fix bats tests working on Jun 16, 2015
Berksfile Update Berksfile to use omnibus cookbook 4 Aug 31, 2016
Berksfile.lock Update Berksfile to use omnibus cookbook 4 Aug 31, 2016
Gemfile Add open_uri_redirections to patch URI redirection errors in omnibus (#… Jul 28, 2017
Gemfile.lock Update omnibus to 5.5 Aug 31, 2016
LICENSE (re-) Add top-level LICENSE Feb 13, 2015 Fix an old link to Vagrant download site Jul 25, 2016
Rakefile Fix bats tests working on Jun 16, 2015
Vagrantfile Update chef to 12.10.24 to use latest omnibus Aug 31, 2016
core_gems.rb Fetch pre-compiled gem native extensions instead of building them fro… Feb 20, 2018
omnibus.rb add build option append_timestamp May 20, 2015
omnibus.rb.example First commit. Not finished yet Feb 28, 2014
plugin_gems.rb Bump fluent-plugin-google-cloud gem version to 0.6.21. (#96) Jun 28, 2018
ui_gems.rb Move globalid to after activesupport to avoid rails 5 Jul 4, 2016


Build Status

The event collector daemon, for Treasure Data. This daemon collects various types of logs/events via various way, and transfer them to the cloud. For more about Treasure Data, see the homepage, and the documentation.

td-agent is open sourced as Fluentd project. In other words, td-agent is a distribution package of Fluentd.

td-agent package is based on Omnibus-ruby


We'll assume you have Ruby 2.1 and Bundler installed. First ensure all required gems are installed and ready to use:

$ bundle install --binstubs



At first, you should download dependent gems using downloder. This is for avoding broken gem download and reduce the build time by avoiding internet access.

$ bin/gem_downloader core_gems.rb
$ bin/gem_downloader plugin_gems.rb
$ bin/gem_downloader ui_gems.rb

Create required directory and add permission

$ sudo mkdir -p /opt/td-agent /var/cache/omnibus
$ sudo chown [USER] /opt/td-agent
$ sudo chown [USER] /var/cache/omnibus

After that, you create a platform-specific package using the build project command:

$ bin/omnibus build td-agent2

The platform/architecture type of the package created will match the platform where the build project command is invoked. So running this command on say a MacBook Pro will generate a Mac OS X specific package. After the build completes packages will be available in pkg/.

Build on CentOS 5

td-agent build doesn't work on CentOS 5 by OpenSSL related issues. Follow this setup instruction before use omnibus:


You can clean up all temporary files generated during the build process with the clean command:

$ bin/omnibus clean

Adding the --purge purge option removes ALL files generated during the build including the project install directory (/opt/td-agent) and the package cache directory (/var/cache/omnibus/pkg):

$ bin/omnibus clean --purge


Full help for the Omnibus command line interface can be accessed with the help command:

$ bin/omnibus help

Build with customized plugins, files and package name

Follow steps below to build packages with customized gem lists, configuration files and original package name.

  1. make plugin list for your own environment
  • add your_plugin_gems.rb to define the plugin list for your package
  • rm plugin_gems/*
  • bin/gem_downloader your_plugin_gems.rb
  1. make your default configuration file
  • edit templates/etc/td-agent/td-agent.conf and td-agent.conf.tmpl
  1. copy config/projects/td-agent2.rb to config/projects/YOUR_PACKAGE_NAME.rb
  2. edit config/projects/YOUR_PACKAGE_NAME.rb
  • fix name, maintainer, homepage and description
  • change install_dir as /opt/YOUR_PACKAGE_NAME
  • change build_version and build_iteration
  • comment out td and td-agent-ui if you want not to install them
  1. build package by bin/omnibus build YOUR_PACKAGE_NAME
  2. test your package file

Build script generates file paths with YOUR_PACKAGE_NAME, from templates. Leave file names with td-agent in templates.

NOTE: edit project_name in Vagrantfile if required.

Vagrant-based Virtualized Build Lab

td-agent omnibus ships will a project-specific Berksfile and Vagrantfile that will allow you to build your projects on the following platforms:

  • CentOS 6 64-bit
  • CentOS 6 32-bit
  • CentOS 7 64-bit
  • Ubuntu 12.04 64-bit
  • Ubuntu 12.04 32-bit
  • Ubuntu 14.04 64-bit
  • Ubuntu 16.04 64-bit
  • Debian 7.10 64-bit
  • Debian 8.4 64-bit
  • Amazon Linux 2016.03 64-bit

Please note this build-lab is only meant to get you up and running quickly; there's nothing inherent in Omnibus that restricts you to just building CentOS or Ubuntu packages. See the Vagrantfile to add new platforms to your build lab.

The only requirements for standing up this virtualized build lab are:

The vagrant-berkshelf and vagrant-omnibus Vagrant plugins are also required and can be installed easily with the following commands:

$ vagrant plugin install vagrant-berkshelf
$ vagrant plugin install vagrant-omnibus
$ vagrant plugin install vagrant-vbguest

Exceute berks command to setup cookbooks

$ berks vendor cookbooks

Once the pre-requisites are installed you can build your package across all platforms with the following command:

$ vagrant up

If you would like to build a package for a single platform the command looks like this:

$ vagrant up PLATFORM

The complete list of valid platform names can be viewed with the vagrant status command.

Amazon Linux build

You need to install vagrant-aws 0.5.0 and add --provider option to vagrant up.

AWS_SSH_KEY_PATH=/path/to/your_aws_key_file vagrant up amazon --provider=aws

After build package, you need to copy rpm file from ec2 instances. No automatic sync for now.

pkg_build command

You can build all environments via bin/pkg_build command.


After that, each package is stored in td_agent2_pkg directory.

% ls td_agent2_pkg/
centos-6.7/   centos-7.2/   debian-7.10/  debian-8.4/   ubuntu-12.04/ ubuntu-14.04/ ubuntu-16.04/

Tested environment

% vagrant --version
Vagrant 1.7.4

% VBoxHeadless --version
Oracle VM VirtualBox Headless Interface 5.0.4
(C) 2008-2015 Oracle Corporation
All rights reserved.


% ruby --version
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]


Vagrant syncs current directory in each platform. Downloaded gems are also installed automatically.

And you should not use rbenv local in project root because Ruby environment is built on top of rbenv in Vagrant. So if you set different Ruby verion in .ruby-version, running ruby code will fail during pacakging process.

Kitchen-based Build Environment

Every Omnibus project ships will a project-specific Berksfile and Vagrantfile that will allow you to build your omnibus projects on all of the projects listed in the .kitchen.yml. You can add/remove additional platforms as needed by changing the list found in the .kitchen.yml platforms YAML stanza.

This build environment is designed to get you up-and-running quickly. However, there is nothing that restricts you to building on other platforms. Simply use the omnibus cookbook to setup your desired platform and execute the build steps listed above.

The default build environment requires Test Kitchen and VirtualBox for local development. Test Kitchen also exposes the ability to provision instances using various cloud providers like AWS, DigitalOcean, or OpenStack. For more information, please see the Test Kitchen documentation.

Once you have tweaked your .kitchen.yml (or .kitchen.local.yml) to your liking, you can bring up an individual build environment using the kitchen command.

$ bundle exec kitchen converge default-ubuntu-1204

Then login to the instance and build the project as described in the Usage section:

$ bundle exec kitchen login default-ubuntu-1204
[vagrant@ubuntu...] $ cd td-agent
[vagrant@ubuntu...] $ bundle install
[vagrant@ubuntu...] $ ...
[vagrant@ubuntu...] $ ./bin/omnibus build td-agent2

For a complete list of all commands and platforms, run kitchen list or kitchen help.


Treasure Agent package is released under the Apache2 license.