Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
17 lines (11 sloc) 1.06 KB

Spark sessionization

This repo contains some examples of different ways to divide records into sessions by timestamps and a maximum interval between records of a session.

For a detailed discussion, see the blog post at http://www.janvsmachine.net/2017/09/sessionization-with-spark.html.

Implementation notes

The code reads example click data based on the Outbrain Click Prediction dataset from Kaggle. This comes as a very large zip-compressed CSV file (page_views.csv.zip). I converted this to Parquet format and partitioned it for efficiency. Thus, the code assumes Parquet format with a suitable schema for its input. Otherwise the data is untouched.

The /scripts folder contains scripts that create a suitable EMR cluster in AWS, and that runs the three different implementations as EMR steps. No attempt has been made to make these completely general, so you'll have to hack these to match your own AWS setup if you want to run these.