flume-to-nifi
Examples for Apache Flume to Apache NiFi
Remember Flume? Me neither. Let's try to do these things in Apache NiFi.
Paul Vidal Asked for a Read me
Articles
- https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-jms-to-x-and-x-to-jms-1g02
- https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_15.html
- https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_42.html
Use Cases - NiFi Flows
- Logs to HDFS: Flume_Log_to_HDFS.xml
- Kafka: Kafka_Example.xml
- JDBC/RDBMS/Database: RDBMSFlume.xml
- Twitter to Kafka: TwitterToKafka.xml
- Kafka to Parquet: kafkatoparquet.xml. (could be HDFS, Files, S3, ADLS)
- PutParquet: putparquet.xml
Flows
Log Files to HDFS
NiFi File Filter / Reg Ex To Match Input Name Patterns: (wifi).*log
UpdateAttribute: Set File Name: ${filename:replaceAll(':', ''):replaceAll('|','')}
Also remove invalid characters from HDFS filename output
https://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/filesystem/model.html
NiFi Output: Attribute Values absolute.path /var/log/ file.creationTime 2019-08-12T10:08:53-0400 file.group wheel file.lastAccessTime 2019-08-12T10:08:53-0400 file.lastModifiedTime 2019-08-12T10:08:53-0400 file.owner root file.permissions rw-r--r-- file.size 271413 filename wifi-08-12-2019__10:08:53.107.log path ./ uuid 6693546e-1241-4625-bbd8-b9dd5da9281a
hdfs dfs -ls /flume Found 7 items -rw-r--r-- 1 tspann hdfs 120824 2019-08-13 19:49 /flume/wifi-08-12-2019__100647.406.log -rw-r--r-- 1 tspann hdfs 125888 2019-08-13 19:49 /flume/wifi-08-12-2019__100651.863.log -rw-r--r-- 1 tspann hdfs 252917 2019-08-13 19:49 /flume/wifi-08-12-2019__100839.901.log -rw-r--r-- 1 tspann hdfs 255873 2019-08-13 19:49 /flume/wifi-08-12-2019__100845.958.log -rw-r--r-- 1 tspann hdfs 271315 2019-08-13 19:49 /flume/wifi-08-12-2019__100848.189.log -rw-r--r-- 1 tspann hdfs 271413 2019-08-13 19:49 /flume/wifi-08-12-2019__100853.107.log -rw-r--r-- 1 tspann hdfs 17078 2019-08-13 19:49 /flume/wifi.log