Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.adoc
Random_User_Kudu.xml

README.adoc

Apache NiFi Quickstart

Below is a brief example using Apache NiFi to ingest data in Apache Kudu.

Start the Kudu Quickstart Environment

See the Apache Kudu quickstart documentation to setup and run the Kudu quickstart environment.

Run Apache NiFi

Use the following command to run the latest Apache NiFi Docker image:

docker run --name kudu-nifi --network="docker_default" -p 8080:8080 apache/nifi:latest

You can view the running NiFi instance at localhost:8080/nifi.

Note
--network="docker_default" is specified to connect the container the same network as the quickstart cluster.
Note
You can include the -d flag to run the container in the background.

Create the Kudu table

Create the random_user Kudu table that matches the expected Schema.

In order to do this without any dependencies on your host machine, we will use the jshell REPL in a Docker container to create the table using the Java API. First setup the Docker container, download the jar, and run the REPL:

docker run -it --rm --network="docker_default" maven:latest bin/bash
# Download the kudu-client-tools jar which has the kudu-client and all the dependencies.
mkdir jars
mvn dependency:copy \
    -Dartifact=org.apache.kudu:kudu-client-tools:1.10.0 \
    -DoutputDirectory=jars
# Run the jshell with the jar on the classpath.
jshell --class-path jars/*
Note
--network="docker_default" is specified to connect the container the same network as the quickstart cluster.

Then, once in the jshell REPL, create the table using the Java API:

import org.apache.kudu.client.CreateTableOptions
import org.apache.kudu.client.KuduClient
import org.apache.kudu.client.KuduClient.KuduClientBuilder
import org.apache.kudu.ColumnSchema.ColumnSchemaBuilder
import org.apache.kudu.Schema
import org.apache.kudu.Type

KuduClient client =
  new KuduClientBuilder("kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251").build();

if(client.tableExists("random_user")) {
  client.deleteTable("random_user");
}

Schema schema = new Schema(Arrays.asList(
  new ColumnSchemaBuilder("ssn", Type.STRING).key(true).build(),
  new ColumnSchemaBuilder("firstName", Type.STRING).build(),
  new ColumnSchemaBuilder("lastName", Type.STRING).build(),
  new ColumnSchemaBuilder("email", Type.STRING).build())
);
CreateTableOptions tableOptions =
  new CreateTableOptions().setNumReplicas(3).addHashPartitions(Arrays.asList("ssn"), 4);
client.createTable("random_user", schema, tableOptions);

Once complete, you can use CTRL + D to exit the REPL and exit to exit the container.

Load the Dataflow Template

The Random_User_Kudu.xml template downloads randomly generated user data from http://randomuser.me and then pushes the data into Kudu. The data is pulled in 100 records at a time and then split into individual records. The incoming data is in JSON Format.

Next, the user’s social security number, first name, last name, and e-mail address are extract from the JSON into FlowFile Attributes and the content is modified to become a new JSON document consisting of only 4 fields: ssn, firstName, lastName, and email. Finally, this smaller JSON is then pushed to Kudu as a single row, each field being a separate column in that row.

To load the template follow the NiFi "Importing a Template" documentation to load Random_User_Kudu.xml.

Then follow the NiFi "Instantiating a Template" documentation to add the Random User Kudu template to the canvas.

Once the template is added to the canvas you need to start the JsonTreeReader controller service. You can do this via the PutKudu processor configuration or via the Nifi Flow configuration in the Operate panel. See the Nifi "Controller Service" documentation for more details.

Now you can start individual processors by right-clicking each processor and selecting Start. You can also explore the configuration, queue contents, and more by right-clicking on each element. Alternatively you can use the Operate panel and start the entire flow at once. More about starting and stopping NiFi components can be read in the NiFi "Starting a Component" documentation.

Shutdown NiFi

Once you are done with the NiFi container you can shutdown in a couple of ways. If you ran NiFi without the -d flag, you can use ctrl + c to stop the container.

If you ran NiFi with the -d flag, you can use the following to gracefully shutdown the container:

docker stop kudu-nifi

To permanently remove the container run the following:

docker rm kudu-nifi

Next steps

The above example showed how to ingest data into Kudu using Apache NiFi. Next explore the other quickstart guides to learn how to query or process the data using other tools.

For example, the Spark quickstart guide will walk you through how to setup and query Kudu tables with the spark-kudu integration.

If you have already run through the Spark quickstart the following is a brief example of the code to allow you to query the random_user table:

spark-shell --packages org.apache.kudu:kudu-spark2_2.11:1.10.0
:paste
val random_user = spark.read
	.option("kudu.master", "localhost:7051,localhost:7151,localhost:7251")
	.option("kudu.table", "random_user")
	// We need to use leader_only because Kudu on Docker currently doesn't
	// support Snapshot scans due to `--use_hybrid_clock=false`.
	.option("kudu.scanLocality", "leader_only")
	.format("kudu").load
random_user.createOrReplaceTempView("random_user")
spark.sql("SELECT count(*) FROM random_user").show()
spark.sql("SELECT * FROM random_user LIMIT 5").show()
You can’t perform that action at this time.