V3 to v4 migration guide

Since version 4.* is another major version than version 3.*, the two versions are not directly compatible. Why this is the case is explained in the FAQ entry What do the version numbers mean?. This doesn't mean you cannot use v4 if you've already used v3, but you will need to migrate your data and configuration file. This document is here to guide you along the necessary steps.

There are two major steps: First, you should migrate your configuration. Once that's done your RRD-files need to be migrated, possibly deleting the data that was collected when the first step was tested.

Migrating the configuration

The layout of the configuration file hasn't changed dramatically, but the name of options may have changed and in some cases functionality has been moved into plugins. We recommend you use the vanilla configfile installed with make install, open your working configuration file next to it, and transfer your changes "by hand".

You will need to make the following changes:

• Quote strings. All strings need to be quoted in double quotes, «"», even paths and IP-addresses. The only exception are words, i. e. strings that only consist of the characters 'A' to 'Z' and 'a' to 'z'.
• Replace the modes. Version 3 used to have the modes "Local", "Client", "Server" and "Log", all of which don't exist anymore. If you used the Local or Log modes you will need to load the RRDtool plugin (if you used Local) or the CSV plugin (if you used Log). You can, of course, load both if that is any use for you. If you used the Client or Server mode you will need to load the Network plugin and configure the Server (if you used Client) or Listen (if you used Server) option. Since the ex-server is supposed to save the data to somewhere, you will need to load the RRDtool plugin there, too.
• Enable logging. Since even logging is done by plugins in version 4 you will need to load one of them if you want to have diagnostic or informative output. We recommend you load the SysLog plugin and set the log-level to at least "warning". An alternative is the LogFile plugin which writes the output to a given file or STDOUT.
• Remove unused plugins. When running a daemon in mode Server it usually loaded next to all plugins in order to be able to save the values submitted by these plugins. This is not true anymore. The description of the various "data sets" (usually called "types") is now loaded from a file called "TypesDB", see the types.db(5) manual page. So if you load the Network plugin and configure it to listen on an interface, only load those plugins the daemon is supposed to collect from the host it's running on. If, for example, the server instance shall receive the MySQL statistics of your database server, you do not need to load the MySQL plugin. But if you want to see the interface traffic of the machine that's running the server instance you need to load the Interface plugin (formerly called "traffic") on that machine.

Once you have changed your configuration, let the daemon run for a couple of minutes. If you get warnings like "read-function of plugin foobar failed", you have either made a mistake when configuring that plugin or that plugin isn't appropriate for your system, e. g. if you have loaded the CPUFreq plugin on a desktop machine. Check if all the data you collected using version 3 is being collected with version 4, too. If all looks good, you can start migrating the data.

Migrating existing RRD files

Migrating existing RRD files is not an easy task, because there are many of them, they're hard to change and given their nature it's kind of time-critical. There are two scripts in the contrib/ directory which should assist you with that task.

The script contrib/migrate-3-4.px scans your data directory and prints a bash-script to STDOUT that is supposed to do the migration for you. It does not execute the commands right away to give you the possibility to look over the commands and make your own adjustments.

The script contrib/rrd_filter.px is used by the code generated by migrate-3-4.px to split one RRD file with many DSes into multiple RRD files with only one or two DSes. You don't have to worry about it's usage much, but it must be found when the migration script is being executed.

We recommend to use those scripts along this lines:

 ~/collectd-4.0.0 $ contrib/migrate-3-4.px \
   -i /var/lib/collectd \
   -o /tmp/collectd-4.0.0/var/lib/collectd/rrd \
   >/tmp/migration.sh
 ~/collectd-4.0.0 $ cp contrib/rrd_filter.px /tmp/
 ~/collectd-4.0.0 $ cd /tmp/
 /tmp $ vim migration.sh
 /tmp $ ./migration.sh

When the script is done start the (version 4) daemon and see if all files that were updated before are still being updated. If some filenames didn't get translated correctly (the Disk plugin is a likely candidate for this) make the appropriate changes to the /tmp/migration.sh (or however you named it) and try again.

Explore the possibilities

We put much effort into the creation of collectd 4, so this new major version provides many new and, hopefully, interesting possibilities. Here are some examples to get you started:

• The RRDtool plugin provides caching to improve IO-performance. This allows very large setups. Inside-the-RRDtool-plugin has the juicy details.
• The Network plugin allows you to forward the data it has received. This allows for multicast-aggregation or multilayered network layouts.
• The Perl plugin provides the internal API to Perl-modules, so you can write your own plugins in Perl if C isn't flexible enough for you.
• The Exec and UnixSock plugins provide more means of custom interaction with the daemon.

Now that you're done: Welcome to the next generation of statistics collection. Enjoy! :)