Skip to content
Reactive Hive Toolkit for Streaming Platforms
Kotlin Java Scala Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
gradle/wrapper
rxhive-components
rxhive-core
rxhive-parquet
rxhive-schemas
.gitignore
.travis.yml
LICENSE
README.md
build.gradle
docker-compose.yml
gradlew
gradlew.bat
hadoop-hive.env
settings.gradle
snapshot_deploy.sh

README.md

rxhive

Build Status License

Introduction

A set of components for reading and writing to Apache Hive and Apache Parquet using popular open source streaming frameworks. Supported frameworks and libraries:

  • Akka-Streams
  • Kotlin Flows

Akka-Streams

Source

To create a source for reading from Hive:

Hive.source("mydb", "mytab")

You must also bring into scope an implicit HiveMetaStoreClient and FileSystem instances. These are both created from configs backed by the hive site files.

val hiveConf: HiveConf = new HiveConf()
implicit val client: HiveMetaStoreClient = new HiveMetaStoreClient(hiveConf)

val conf: Configuration = new Configuration()
implicit val fs: FileSystem = FileSystem.get(conf)

Sink

To create a sink for writing data to Hive:

Hive.sink("mydb", "mytab", HiveSinkSettings())

Again this requires an implicit client and filesystem.

Background

To create a source for reading from Hive:

Processing Steps for writing struct data to hive:

Presteps:

  • If the table exists and overwrite mode is used, delete the table
  • If the table does not exist and overwrite/create mode is used, create the table from the supplied schema or derive it from the first record. Partition columns must be specified

For each struct:

  • Resolve the schema in the struct with the schema in the metastore using a SchemaResolver.
  • If the table is partitioned:
    • Calculate the partition for this struct
    • Using a partitioner: locate the partition write directory and ensure that the partition exists in the metastore and on the filesystem
  • If the table is not partitioned:
    • Return the base table location
  • Calculate the write schema from the input schema, the metastore schema, and the partitions (if any)
  • Derive an updated struct using the incoming struct and the write schema
  • Get or create a writer for the write location
    • If the writer has not been created, create a new file using the file manager.
  • Write the aligned struct using a struct writer

After all structs:

  • Close each writer to flush any data
  • For each file, complete the file via the file manager.
You can’t perform that action at this time.