Neville Li edited this page Oct 16, 2018 · 123 revisions
Scio Logo

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]
Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.

Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.

Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.

Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.

See Scio Scaladocs for current API documentation.

See scio-contrib for community-supported add-ons for Scio.

Getting help

  • Join the chat at https://gitter.im/spotify/scio
  • Google Group
  • Stack Overflow

Documentation

Further Readings

Presentations

Scio Development

Projects using or related to Scio

  • scio-contrib - Community-supported add-ons for Scio
  • Featran - A Scala feature transformation library for data science and machine learning
  • Big Data Rosetta Code - Code snippets for solving common big data problems in various platforms. Inspired by Rosetta Code
  • Ratatool - A tool for random data sampling and generation, which includes BigDiffy, a Scio library for pairwise field-level statistical diff of data sets (slides)
  • scio-deep-dive - Building Scio from scratch step by step for an internal training session
  • scala-flow - A lightweight Scala wrapper for Google Cloud Dataflow from Zendesk
  • clj-headlights - Clojure API for Apache Beam, also from Zendesk
  • datasplash - A Clojure API for Google Cloud Dataflow
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.