Datasource-Receiver

Datasource-Receiver is a library that allows the user to read data with SparkSQL from Datasources and then insert the data in Spark Streaming

Requirements

This library requires Spark 1.5+, Scala 2.10+

Using the library

There are two ways of using Datasource-Receiver library:

The first one is to add the next dependency in your pom.xml:

<dependency>
  <groupId>com.stratio.receiver</groupId>
  <artifactId>datasource</artifactId>
  <version>LATEST</version>
</dependency>

The other one is to clone the full repository and build the project:

git clone https://github.com/Stratio/Datasource-Receiver.git
mvn clean install

The basic idea is receive data in Spark Streaming with Spark Sql:

Some use cases are monitoring tables, migrating data, generate streaming data from batch data... The use of SparkSQL have the advantage that the library is polyglot and can read data from multiple databases, like JDBC, Cassandra, ElasticSearch, MongoDB ... The user must add the java jar with the connector library and then is possible to use it in the initial query. Per example:

• Reading data from MongoDB with the Stratio Library:

InputSentences(
   query = "select * from tableName",
   initialStatements = Seq(
     "CREATE TEMPORARY TABLE tableName USING com.stratio.datasource.mongodb OPTIONS (host 'localhost', database 'sparta', collection 'streams'")
   )
) 

Distributed Approach with SparkSQL

The distributed approach is based in SparkSQL, on each streaming window the receiver execute one query with SparkSQL and generate one DataFrame and the partitions of this DataFrame corresponds with the DStream partitions.

With the receiver is possible to monitoring tables in the datasources, the user specify one offset field and on each window the offset is updated to generate the new incremental query.

The receiver have implemented one Object for specify the offset options, limit the results and generate stop conditions. One example:

InputSentences(
      query = "select * from tableName",
      offsetConditions = OffsetConditions(OffsetField(name = "followersCount")),
      stopConditions = StopConditions(stopWhenEmpty = true, finishContextWhenEmpty = true),
      initialStatements = Seq("CREATE TEMPORARY TABLE tableName USING com.stratio.datasource.mongodb OPTIONS (host 'localhost', database 'sparta', collection 'streamtrigger')")
)

Build

There are two modules to package with different versions of Spark

• To package it with Spark-1.5:

  mvn clean package -pl com.stratio.receiver:spark-datasource_1.5

• To package it with Spark-1.6 (default):

  mvn clean package -pl com.stratio.receiver:spark-datasource_1.6

Scala API

• Implicitly use SparkSQLContext

  val receiverStream = DatasourceUtils.createStream(sparkStreamingContext, inputSentences, datasourceParams)
  

• Using a generic Context

  val receiverStream = DatasourceUtils.createStream(sparkStreamingContext, inputSentences, datasourceParams, sqlContext)
  

Java API

• Implicitly use SparkSQLContext

  JavaReceiverInputDStream receiverStream = DatasourceUtils.createJavaStream(javaSparkStreamingContext, inputSentences datasourceParams);
  

• Using a generic Context

  JavaReceiverInputDStream receiverStream = DatasourceUtils.createJavaStream(javaSparkStreamingContext, inputSentences datasourceParams, sqlContext);
  

Spark Parameters Options

Parameter	Description	Optional
stopGracefully	Stop Streaming Context gracefully	Yes (default: true)
stopSparkContext	Stop SparkContext when stop Streaming	Yes (default: false)
storageLevel	Max time to receive messages	Yes (default: MEMORY_ONLY)
rememberDuration	Remember duration for Spark Dstreams	Yes (default: 60s)

License

Licensed to STRATIO (C) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The STRATIO (C) licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.