This project's intent is to demonstrate the use of Apache NiFi (and NiFi MiNiFi) to collect and make available application and device information.
The containers are pre-configured to run:
- A basic Python HTTP server
- nmon for generating snapshots of host performance and configuration data
- Apache NiFi and Apache NiFi subproject MiNiFi to gather metadata from Consul, logs from HTTP server, nmon, 'netstat', and 'ps' output
- Apache Zeppelin to display a basic operational view of data gathered from containers by NiFi and MiNiFi
Overview Diagram:
This project uses features from unreleased NiFi 1.2.0. To run this yourself, you'll need to build it from master:
git clone https://github.com/apache/nifi
cd nifi
mvn clean package -DskipTests
cp nifi-assembly/target/nifi-1.2.0-SNAPSHOT-bin.tar.gz ~/projects/TechOps/containers/monitor
Build and run containers:
docker build -t app containers/app
docker-compose up
Build will take 20ish minutes. About 30 seconds after successful start, the following web UIs should be accessible:
- NiFi at localhost:8080/nifi
- Zeppelin at localhost:8081
Data generated by the containers is saved in the 'data' directory of this project.
To increase the number of web servers:
docker-compose scale web-service.dev=10
If you want to avoid running the full stack, you may use sample data provided for convenience:
rm -rf data/
tar -xzvf data.tgz
docker-compose up zeppelin.dev
You may then explore the sample data in Zeppelin.
Use Zeppelin's "Run All" feature to run all notes in DeviceRegistry-Setup before you can generate the charts in the DeviceRegistry notebook.
Sample Graphs:
