Easy access to big things. Library for Apache Spark extending and improving its capabilities
Permalink
Failed to load latest commit information.
.github Update PULL_REQUEST_TEMPLATE.md Jun 6, 2016
cassandra Prepare for next version Dec 13, 2016
common RowSerializer: Safe guard against double types with Decimal schema ta… Dec 22, 2016
core [CROSSDATA-832] Make Elasticsearch Crossdata connector comply with th… Jan 27, 2017
dist Prepare for next version Dec 13, 2016
docker Cherry picks before branch 1.8 (#794) Dec 9, 2016
driver Executing query from the shell (#800) Jan 19, 2017
elasticsearch [CROSSDATA-832] Make Elasticsearch Crossdata connector comply with th… Jan 27, 2017
examples [CROSSDATA-546][Driver] Add shutdown method (#628) Jul 27, 2016
mongodb Prepare for next version Dec 13, 2016
postgresql [Postgresql Connector] TestAT (#812) Feb 8, 2017
project Added sbt-scalafmt thus enabling `sbt scalafmt` tool to fix scala fil… Nov 25, 2016
sandbox Fix Sandbox Feb 5, 2016
scripts Minor changes Jan 27, 2016
server Postgresql connector (#804) Jan 24, 2017
streaming Prepare for next version Dec 13, 2016
testsAT [Postgresql Connector] TestAT (#812) Feb 8, 2017
testsIT [Postgresql Connector] Support union, intersect, except and fix SQLBu… Jan 25, 2017
zeppelin Prepare for next version Dec 13, 2016
.gitignore [CROSSDATA-324] Session managament (#624) Jul 22, 2016
.gitmodules updating sphinx configuration and theme Mar 25, 2015
.jenkins.yml Postgresql connector (#804) Jan 24, 2017
.travis.yml [CROSSDATA-553] Set scala binary version to 2.11 by default (#633) Aug 18, 2016
CHANGELOG.md Prepare for next version Dec 13, 2016
Jenkinsfile proper variable name (#831) May 3, 2017
LICENSE Change license to Apache V2 Aug 4, 2014
NOTICE Initial Commit for New Crossdata Jul 9, 2015
README.md Remove references to private docs (#835) Sep 21, 2017
build.sbt Added sbt-scalafmt thus enabling `sbt scalafmt` tool to fix scala fil… Nov 25, 2016
cleanDerbyDBs.sh Merge with branch 1.4 (#600) Jul 4, 2016
pom.xml Update Stratio Parent to 0.9.0 version May 5, 2017

README.md

Introduction

GitterIL Coverage Status

Crossdata is a distributed framework and a fast and general-purpose computing system powered by Apache Spark. It unifies the interaction with different sources supporting multiple datastore technologies thanks to its generic architecture and a custom SQL-like language using SparkSQL as the core of the project. In addition, Crossdata supports batch and streaming processing so that you can mix data from both input technologies. Supporting multiple architectures imposes two main challenges: how to normalize the access to the datastores, and how to cope with datastore limitations. Crossdata provides connectors which can access to multiple datastores natively, speeding up the queries by avoiding the overhead and the block of resources of the Spark Cluster when possible. We offer a shell, Java and Scala APIs, JDBC and ODBC for BI tools.

This project is aimed for those who want to manage only one API to access to multiple datastores with different nature, get rid of the drawbacks of Apache Spark, perform analytics from a BI tool and speed up your queries effortlessly.

Crossdata is broken up into the following components:

  • Crossdata Core: It is a library that you can deploy in any existent system using Spark with no changes, just add the Crossdata jar file. SparkSQL extension with improvements in the DataSource API and new features. Crossdata expands the functionalities of Apache Spark in order to provide a richer SQL-like language, to improve some aspects (metastore, execution trees, ...)
  • Crossdata Server: Provides a multi-user environment to SparkSQL, giving a reliable architecture with high-availability and scalability out of the box.
  • Crossdata Driver: Entry point with an API for both Scala and Java. Crossdata ODBC/JDBC uses this driver. - Crossdata Connectors: Take advantage of the Crossdata DataSource API to speed up the queries in specific datasources and provide new features.

We include some Spark connectors optimized to access to each datasource, but Crossdata is fully compatible with any connector developed by the Spark community.

  • Apache Cassandra connector powered by Datastax-Spark-Connector
  • MongoDB connector powered by Stratio-Spark-Connector
  • ElasticSearch connector powered by Elastic-Spark-Connector

Moreover, some datasources are already included, avoiding to import them manually:

  • Spark-CSV
  • Spark-Avro

Main Crossdata's advantages over other options:

  • JDBC/ODBC self-contained. Other solutions require Hive.
  • Faster queries using native access (including subdocuments and array elements).
  • Streaming queries from a SQL-like interface.
  • Metadata discovery.
  • Datasource functions (Spark only can execute its own UDFs).
  • High-availability and load balancing.
  • Logical views.
  • Full SQL interface for documents with nested subdocuments and nested arrays.
  • Persistent metadata catalog.
  • Common interface for datasources management.
  • Creation of tables in the datastores.
  • Drop of tables from the datastores.
  • Insert with values queries as in typical SQL.
  • Service Discovery

=================== Spark Compatibility

Crossdata Version Spark Version
1.7.X 1.6.X
1.6.X 1.6.X
1.5.X 1.6.X
1.4.X 1.6.X
1.3.X 1.6.X
1.2.X 1.5.X
1.1.X 1.5.X
1.0.X 1.5.X

=========== Get support

You can send us issues in https://crossdata.atlassian.net.

You can also find help in https://groups.google.com/forum/#!forum/crossdata-users.

There is also a gitter channel available: https://gitter.im/Stratio/Crossdata.

Alternatively, you can try to reach us at gitter or our IRC channel #stratio-crossdata. Feel free to ask, if we are available we'll try to help you.

============= Release notes

Features and changes are detailed in the changelog.

======= License

Stratio Crossdata is licensed as Apache2

Licensed to STRATIO (C) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The STRATIO (C) licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.