Apache NiFi Streaming Examples

Collection of examples integrating NiFi with stream process frameworks.

Initial Setup

• Download the latest Apache NiFi release

• Extract the tar and create two instances of NiFi:

    
    tar xzvf nifi-1.0.0-bin.tar.gz
    mv nifi-1.0.0 nifi-edge
    tar xzvf nifi-1.0.0-bin.tar.gz
    mv nifi-1.0.0 nifi-core
  

• Configure the edge instance by editing nifi-edge/conf/nifi.properties and setting the following properties:

    
    nifi.remote.input.socket.port=7088
    nifi.remote.input.secure=false
    nifi.web.http.port=7080
  

• Configure the core instance by editing nifi-core/conf/nifi.properties and setting the following properties:

   
    nifi.remote.input.socket.port=8088
    nifi.remote.input.secure=false
    nifi.web.http.port=8080
  

• Start both instances

   
    ./nifi-core/bin/nifi.sh start
    ./nifi-edge/bin/nifi.sh start
  

• Open the UI for both instances in a browser

   
    http://localhost:7080/nifi/
    http://localhost:8080/nifi/
  

• Setup initial dictionary files

  
    mkdir nifi-edge/data
    mkdir nifi-edge/data/dictionary
    mkdir nifi-core/data
    mkdir nifi-core/data/dictionary
  

• In each of the above dictionary directories, create a file called levels.txt with the content:

    ERROR
    WARN

• Import nifi-streaming-examples/templates/nifi-log-example-edge.xml into the the edge instance (http://localhost:7080/nifi)

• Import nifi-streaming-examples/templates/nifi-log-example-core.xml into the the core instance (http://localhost:8080/nifi)

• Start everything on the core instance (http://localhost:8080/nifi)

• To start sending logs, starting everything on the edge instance (http://localhost:8080/nifi) EXCEPT the TailFile processor, the "Generate Test Logs" process group will send fake log messages

• To tail a real file, stop the "Generate Test Logs" process group, configure TailFile to point to your log file of choice, and start the TailFile processor

Flink - WindowLogLevelCount - Setup

• For local testing, run a standalone Flink streaming job

  cd nifi-flink-examples
  mvn clean package -PWindowLogLevelCount
  java -jar target/nifi-flink-examples-0.0.1-SNAPSHOT.jar

Apex - LogLevelApplicationRunner - Setup

• For local testing, run LogLevelApplicationRunner from your favorite IDE:

  nifi-apex-examples/src/test/java/nifi/apex/examples/logs/LogLevelApplicationRunner.java

Storm - LogLevelCountTopology - Setup

• For local testing, run a standalone local Storm topology

  cd nifi-storm-examples
  mvn clean package -PLogLevelCountTopology
  java -jar target/nifi-storm-examples-0.0.1-SNAPSHOT.jar