Install and configure Apache Flume a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.
See the full Salt Formulas installation and usage instructions.
Download, install, configure Apache Flume.
Main important sections are
One or more agents could be added under
And each agent could have many sources, channels, sinks.
flume: agents: agent01: sources: httpSource: type: http port: 8080 channels: httpChannel01: type: memory capacity: 1000000 transactionCapacity: 100000 nullChannel01: type: memory capacity: 1000 transactionCapacity: 1000 sinks: httpFileSink01: type: file_roll channel: httpChannel01 sink: directory: /var/log/flume rollInterval: 300 nullSink01: type: 'null' channel: nullChannel01 sinkgroups: sinkgroup01: processor: type: failover backoff: 'false' selector: failover
More Flume plugins (i.e. external libs) could be added under
If the plugin doesn't follow Flume convention,
then the non-standard dirs should be exposed in
If the plugin does follow the convention, then no need to
flume: plugins: hadoop: source: https://archive.apache.org/dist/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz hash: False paths: # No need to expose dirs here. native: src: lib/native dest: native hdfs: src: share/hadoop/hdfs dest: hdfs # Expose classpath dirs, # in case the plugin doesn't follow Flume plugins convention. classpath: - '*' - 'lib/*' common: src: share/hadoop/common dest: common classpath: - '*' - 'lib/*'