Reading List for Stream Processing

Table of Contents

• Reading List for Stream Processing
  • Table of Contents
  • Courses
  • Books
  • Surveys
  • Stream Processing Engines
  • Micro-batch
  • Scheduling
    • Tuple Scheduling
  • Elasticity
    • Queuing Theory
    • State Management
  • Sketches
  • Stream Join
  • Hardware
  • Benchmarking
  • Misc
  • Related Topics
    • Batch Processing
    • Scheduling
      • Cluster Resource Management
      • Deep Learning
    • Resource Provisioning
      • Autoscaling
      • Overcommitment
    • Storage

Courses

• CS 591 K1: Data Stream Processing and Analytics, Spring 2021, Brown University

Books

• Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing, O'Reilly Media, Inc., 2018
• Stream Processing with Apache Flink: Fundamentals, Implementation, and Operation of Streaming Applications, O'Reilly Media, Inc., 2019

Surveys

• A Comprehensive Survey on Parallelization and Elasticity in Stream Processing, ACM Computing Surveys, 2019
• Hardware-Conscious Stream Processing: A Survey, ACM SIGMOD Record, 2020
• Resource Management and Scheduling in Distributed Stream Processing Systems: A Taxonomy, Review, and Future Directions, ACM Computing Surveys, 2020
• A Survey on the Evolution of Stream Processing Systems, arXiv, 2020

Stream Processing Engines

• S4: Distributed Stream Computing Platform, ICDMW, 2010, Yahoo
• Discretized Streams: An Efficient and Fault-Tolerant Model for Stream Processing on Large Clusters, HotCloud, 2012
• Naiad: a timely dataflow system, SOSP, 2013, Microsoft
• MillWheel: fault-tolerant stream processing at internet scale, VLDB, 2013, Google
• Storm@twitter, SIGMOD, 2014
• Twitter Heron: Stream Processing at Scale, SIGMOD, 2015, Twitter
• The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing, VLDB, 2015, Google
• Apache Flink™: Stream and Batch Processing in a Single Engine, Bulletin of the TCDE, 2015
• StreamScope: Continuous Reliable Distributed Processing of Big Data Streams, NSDI, 2016, Microsoft
• Samza: stateful scalable stream processing at LinkedIn, VLDB, 2017, LinkedIn
• State management in Apache Flink®: consistent stateful distributed stream processing, VLDB, 2017
• Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark, SIGMOD, 2018
• A Cloud Native Platform for Stateful Streaming, arXiv, 2020, IBM

Micro-batch

• Drizzle: Fast and Adaptable Stream Processing at Scale, SOSP, 2017
• Prompt: Dynamic Data-Partitioning for Distributed Micro-batch Stream Processing Systems, SIGMOD, 2020

Scheduling

Tuple Scheduling

• POTUS: Predictive Online Tuple Scheduling for Data Stream Processing Systems, TCC, 2020

Elasticity

• ChronoStream: Elastic Stateful Stream Computation in the Cloud, ICDE, 2015
• Stela: Enabling Stream Processing Systems to Scale-in and Scale-out On-demand, IC2E, 2016
• Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows (DS2), OSDI, 2018
• Minimizing cost by reducing scaling operations in distributed stream processing, VLDB, 2019
• Elasticutor: Rapid Elasticity for Realtime Stateful Stream Processing, SIGMOD, 2019
• Turbine: Facebook’s Service Management Platform for Stream Processing, ICDE, 2020, Facebook
• Auto-sizing for Stream Processing Applications at LinkedIn (Sage), HotCloud, 2020, LinkedIn
• Move Fast and Meet Deadlines: Fine-grained Real-time Stream Processing with Cameo, NSDI, 2021

Queuing Theory

• Elastic Stream Processing with Latency Guarantees (Nephele), ICDCS, 2015
• DRS: Auto-Scaling for Real-Time Stream Analytics, IEEE ACM Trans Netw, 2017

State Management

• Integrating scale out and fault tolerance in stream processing using operator state management (SEEP), SIGMOD, 2013
• Megaphone: latency-conscious state migration for distributed streaming dataflows, VLDB, 2019
• In support of workload-aware streaming state management, HotStorage, 2020

Sketches

• Cold Filter: A Meta-Framework for Faster and More Accurate Stream Processing, SIGMOD, 2018

Stream Join

• Providing streaming joins as a service at Facebook, VLDB, 2018
• AJoin: ad-hoc stream joins at scale, VLDB, 2019

Hardware

• Analyzing efficient stream processing on modern hardware, VLDB, 2019

Benchmarking

• Nexmark benchmark suite
  • nexmark/nexmark: Benchmarks for queries over continuous data streams.
• Benchmarking cloud serving systems with YCSB, SoCC, 2010
• Benchmarking Distributed Stream Data Processing Systems, ICDE, 2018
• Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures, Big Data Research, 2021

Misc

• TerseCades: Efficient Data Compression in Stream Processing, USENIX ATC, 2018 (compression)

---

Related Topics

Batch Processing

• Riffle: optimized shuffle service for large-scale data analytics, EuroSys, 2018

Scheduling

• Elasecutor: Elastic Executor Scheduling in Data Analytics Systems, SOCC, 2018

Cluster Resource Management

• Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center, NSDI, 2011
• Omega: flexible, scalable schedulers for large compute clusters, EuroSys, 2013, Google
• Apache Hadoop YARN: yet another resource negotiator, SoCC, 2013
• Apollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing, OSDI, 2014, Microsoft
• Large-scale cluster management at Google with Borg, EuroSys, 2015, Google
• Hydra: a federated resource manager for data-center scale analytics, NSDI, 2019, Microsoft
• Twine: A Unified Cluster Management System for Shared Infrastructure, OSDI, 2020, Facebook

Deep Learning

• HiveD: Sharing a GPU Cluster for Deep Learning with Guarantees, OSDI, 2020, Microsoft

Resource Provisioning

Autoscaling

• Autopilot: workload autoscaling at Google, EuroSys, 2020, Google

Overcommitment

• Take it to the limit: peak prediction-driven resource overcommitment in datacenters, EuroSys, 2021, Google

Storage

• Optimizing space amplification in RocksDB, CIDR, 2017
• FASTER: A Concurrent Key-Value Store with In-Place Updates, SIGMOD, 2018