Scala Java Other
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.circleci publish ScioBenchmark results to Datastore Aug 15, 2018
logo add logo Jul 26, 2017
project bump plugins (#1285) Aug 9, 2018
scio-avro/src add username to TypeProvider path, fix #1205 (#1244) Jul 19, 2018
scio-bench update scio-bench May 31, 2018
scio-bigquery/src fix some deprecation warnings (#1256) Jul 19, 2018
scio-bigtable/src consolidate GCP project for IT Aug 15, 2018
scio-cassandra2/src add notes about cassandra sinks Aug 16, 2018
scio-cassandra3/src add notes about cassandra sinks Aug 16, 2018
scio-core/src/main Remove duplicate registration of AllScalaRegistrar (#1302) Aug 15, 2018
scio-elasticsearch2/src Add elasticsearch ensureIndex helper methods Nov 15, 2017
scio-elasticsearch5/src Add elasticsearch ensureIndex helper methods Nov 15, 2017
scio-examples/src Improved arguments parsing Aug 15, 2018
scio-extra/src Fix OOM when using sparkey side input, fix #1269 (#1280) Aug 8, 2018
scio-jdbc/src Add basic linting scalac options (see: #1097) (#1147) May 8, 2018
scio-jmh/src/test/scala/com/spotify/scio add FunctionsBenchmark #1183 (#1291) Aug 13, 2018
scio-parquet/src fix ParquetAvroSink scaladoc error Mar 11, 2018
scio-repl/src fix #867, REPL hangs on closeAndCollect for scala 2.12 (#1277) Aug 6, 2018
scio-schemas/src/main Add array field to avro records used for testing. (#999) Jan 5, 2018
scio-tensorflow/src Set presence on TF schema + fix bug fixed feature on missing features Aug 15, 2018
scio-test/src pretty print benchmark results Aug 15, 2018
scripts also bump beamVersion in bump_scio.sh Aug 16, 2018
site refactor site layout Nov 8, 2017
.gitignore Fix kryo performance Oct 6, 2017
.jvmopts Use xlarge resource class (#1200) Jun 7, 2018
CONTRIBUTING.md Fix grammar in contribution guidelines (#906) Oct 30, 2017
LICENSE Add license file Feb 19, 2016
NOTICE add NOTICE Feb 23, 2016
README.md Update README with reference to scio.g8 (#1072) Mar 10, 2018
apache.sbt add nightly build with Beam SNAPSHOTs, fix #512 Jan 5, 2018
build.sbt bump scio-cassandra2 to Cassandra 2.2.12, fix #1257 (#1263) Aug 16, 2018
scalastyle-config.xml set all ScalaStyle check levels to error Mar 28, 2016
version.sbt Setting version to 0.5.8-SNAPSHOT Aug 15, 2018

README.md

Scio

Build Status codecov.io GitHub license Maven Central Join the chat at https://gitter.im/spotify/scio

Scio Logo

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]

Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.

Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.

Features

  • Scala API close to that of Spark and Scalding core APIs
  • Unified batch and streaming programming model
  • Fully managed service*
  • Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable
  • HDFS, JDBC, TensorFlow TFRecords, Cassandra, Elasticsearch and Parquet I/O
  • Interactive mode with Scio REPL
  • Type safe BigQuery
  • Integration with Algebird and Breeze
  • Pipeline orchestration with Scala Futures
  • Distributed cache

* provided by Google Cloud Dataflow

Quick Start

Use our giter8 template to quickly setup a project:

sbt new spotify/scio.g8

Compile it:

sbt pack

Run the included word count example:

target/pack/bin/word-count --output=wc

Inspect the results:

cat wc/part-00000-of-00001.txt

Documentation

Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.

Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.

Artifacts

Scio includes the following artifacts:

  • scio-core: core library
  • scio-test: test utilities, add to your project as a "test" dependency
  • scio-avro: add-on for Avro, included in scio-core but can also be used standalone
  • scio-bigquery: add-on for BigQuery, included in scio-core but can also be used standalone
  • scio-bigtable: add-on for Bigtable
  • scio-cassandra2: add-on for Cassandra 2.x
  • scio-cassandra3: add-on for Cassandra 3.x
  • scio-elasticsearch2: add-on for Elasticsearch 2.x
  • scio-elasticsearch5: add-on for Elasticsearch 5.x
  • scio-extra: extra utilities for working with collections, Breeze, etc.
  • scio-hdfs: add-on for HDFS IO
  • scio-jdbc: add-on for JDBC IO
  • scio-parquet: add-on for Parquet
  • scio-tensorflow: add-on for TensorFlow TFRecords IO and prediction

License

Copyright 2016 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0