Skip to content

The Storage Loader

Jump to bottom Edit New page
Yali Sassoon edited this page Oct 22, 2013 · 3 revisions

HOME > [SNOWPLOW TECHNICAL DOCUMENTATION](Snowplow technical documentation) > [Storage](storage documentation) > The Storage Loader

An overview of how the StorageLoader instruments the loading of data from S3 into Redshift

  • Data from enriched Snowplow event files generated by the Scalding process on EMR is read adn written to Amazon Redshift
  • The enriched event files are then moved from teh in-bucket (which was the archive bucket for the EmrEtlRunner) to the archive bucket (for the StorageLoader)

The StorageLoader is configured via its config file. (Example config files for Redshift users can be found [here] redshift.yml.sample and Postgres users [here] postgres.yml.sample.) For more information, see the guide to setting up the StorageLoader.

Home | About | Project | Setup Guide | Technical Docs | Copyright © 2012-2013 Snowplow Analytics Ltd

HOME > [TECHNICAL DOCUMENTATION](Snowplow technical documentation)

1. Trackers
Overview
Javascript Tracker
No-JS Tracker
Lua Tracker
Arduino Tracker

A. Snowplow Tracker Protocol

2. Collectors
Overview
Cloudfront collector
Clojure collector (Elastic Beanstalk)
Scala Stream Collector

B. Collector logging formats

3. Enrich
Overview
EmrEtlRunner
Scala Kinesis Enrich

C. Canonical Snowplow event model

4. Storage
Overview
[Storage in S3](S3 storage)
Storage in Redshift
Storage in PostgreSQL
Storage in Infobright (deprecated)
The StorageLoader

D. Snowplow storage formats (to write)

5. Analytics
Analytics documentation

Common
Artifact repositories

Clone this wiki locally