MemSQL Streamliner is a deprecated feature and will be removed in MemSQL 6.0
This is a starter repository that you can use to build pipelines for MemSQL Spark Streamliner.
MemSQL Spark Streamliner lets you build custom Spark pipelines to:
- extract from real-time data sources such as Kafka,
- transform data structures such as CSV, JSON, or Thrift in table rows,
- load your data into MemSQL.
Check out the MemSQL Spark Streamliner Examples repository for more example Extractors and Transformers.
-
Clone this repository
-
Modify
Extractors.scala
andTransformers.scala
-
Build the JAR with:
make build
-
The JAR will be placed in
target/scala-<version>/
. Upload the JAR to MemSQL Ops and create a pipeline using your custom code.
Read more on how to create custom Spark Interface JARs in our docs.
Run:
make test