The Enrichment Process
Enrichment process parses raw Snowplow events and performs the following:
- Extracts data
- Validates data against Snowplow Tracker Protocol and JSON schema
- Enriches data (adds extra value derived from the tracked/captured data), so called "dimension widening"
- Writes enriched data out
Therefore feeding in a raw Snowplow event will produce either the enriched event with additional data (context), modified (enriched) data or a
JSON schema specifies a JSON-based format to define the structure of JSON data for validation, documentation, and interaction control.
We distinguish 3 types of enrichment:
- Hardcoded enrichments loading
- Configurable enrichments loading
- Configurable enrichments adding new contexts to the
Legacy enrichments are those which populate
atomic.events table as opposed to enrichment’s dedicated tables. The hardcoded legacy enrichments normally take place as part of common enrichment process and they precede configurable enrichments.
Configurable enrichments are those controlled with
--enrichments option passed to the ETL (Extract, Transform, Load) runner. They often depend on the data produced by the common enrichment process.
ETL stands for Extract, Transform, Load.
During the common enrichment process the data received from collector(s) is mapped according to our Canonical Event Model.
The raw data undergoing "dimension widening" (enrichment) listed as per following:
The following fields are populated depending on whether the tracker provided the corresponding value or not.
|Raw Parameter||Enriched Parameter||Purpose|
||The unique event identifier (UUID). Assigned during enrichment if not provided with
||User ID set by Snowplow using 3rd party cookie. Overwriten with tracker-set
||Snowplow collectors log IP address as standard. However, you can override the value derived from the collector by populating this value in the tracker.|
||Raw useragent (browser string). Could be overwritten with
The following fields are populated depending on the collector and ETL (Extract, Transform, Load) utilized in the pipeline.
||Host ETL version|
||Timestamp event began ETL|
||Time stamp for the event recorded by the collector|
The raw parameter
res (if present) representing the screen/monitor resolution and coming in as a combination of width and height (ex.
1280x1024) is broken up into separate entities.
||Screen / monitor width|
||Screen / monitor height|
url parameter provides the value for
atomic.events, which represents the current page's URL. The following parts are extracted and populate separate fields as outlined below.
||Scheme (protocol), ex. "http"|
||Host (domain), ex. "www.snowplowanalytics.com"|
||Port if specified, 80 if not|
||Path to page, ex. "/product/index.html"|
||Querystring, ex. "id=GTM-DLRG"|
||Fragment (anchor), ex. "4-conclusion"|
page_referrer gets the value from
refr, which represents the referer’s URL, and the following parts are extracted and populate separate fields as shown below.
||Port if specified, 80 if not|
||Path to page|
Additionally the derived timestamp is calculated,
derived_tstamp. See this blog post for more details.
Finally, contexts, unstructured events and the relevant configurable enrichments (if enabled) are validated against their corresponding JSON schemas and the array of the derived contexts is assembled.
The configurable enrichment are listed below. Follow the corresponding links to find out more.
The enrichments which write the data into
atomic.events table (legacy enrichments):
- IP anonymization enrichment
- IP lookups enrichment
- Campaign attribution enrichment
- Currency conversion enrichment
- referer-parser enrichment
- user-agent-utils enrichment
- Event fingerprint enrichment
The below list contains the enrichment which create a separate context and thus are loaded into their dedicated tables (as opposed to