Release 82 Tawny Eagle

Adds the ability to use HTTP endpoints in the Kinesis Elaticsearch Sink.

• Blog post

Common

• Common: publish each Kinesis app individually to Bintray (#2492)

Kinesis Elasticsearch Sink

• Kinesis Elasticsearch Sink: bump to 0.7.0 (#2816)
• Kinesis Elasticsearch Sink: configure transport port (#2102)
• Kinesis Elasticsearch Sink: add support for HTTP protocol (#2092)
• Kinesis Elasticsearch Sink: unify logger configuration (#1699)

Release 81 Kangaroo Island Emu

Reboots the Hadoop Event Recovery project (formerly Hadoop Bad Rows), which allows you to fix up Snowplow bad rows and make them ready for reprocessing

• Blog post

Documentation

• Fix broken link in Thrift Schemas' README.md (#2498)

Common

• Add encrypted S3 credentials to .travis.yml (#2673)
• Delete publish-kinesis-release.bash (#2711)

Android Tracker

• Bump git submodule to 0.5.4 (#2710)

JavaScript Tracker

• Bump git submodule to 2.6.1 1. (#2708)

Objective-C Tracker

• Bump git submodule to 0.6.1 (#2709)

Golang Tracker

• Add git submodule (#2619)

Scala Common Enrich

• Bump to 0.23.1 (#2699)
• Bump commons codec to 1.10 (#2691)

Stream Enrich

• Bump to 0.8.1 (#2701)
• Bump Scala Common Enrich to 0.23.1 (#2700)

Hadoop Event Recovery

• Update README instructions (#2348)
• Add continuous deployment (#2692)
• Rename from Scala Hadoop Bad Rows (#2694)
• Allow source row to be transformed with JavaScript (#2223)
• Capitalize Snowplow correctly in copyright notices (#2641)

StorageLoader

• Write JSON path for com.clearbit/person (#2631)
• Write JSON path for com.clearbit/company (#2632)
• Write JSON path for com.amazon.aws.lambda/java_context (#2560)

Redshift

• Add Redshift DDL for com.clearbit/person/jsonschema/1-0-0 (#2633)
• Add Redshift DDL for com.clearbit/company/jsonschema/1-0-0 (#2634)
• Add Redshift DDL for com.amazon.aws.lambda/java_context/jsonschema/1-0-0 (#2559)

Release 80 Southern Cassowary

Real-time pipeline release which improves stability and brings the real-time pipeline up-to-date with our Hadoop pipeline

• Blog post

Common

• Add CI/CD for Kinesis apps (#2621)
• Add Bintray credentials to .travis.yml (#2618)
• Change Kinesis pipeline status from "Beta" to "Production-ready" in READMEs (#2629)

Config

• Update config/iglu_resolver.json version to 1-0-1 (#2479)

Scala Stream Collector

• Bump to 0.7.0 (#2595)
• Increase tolerance of timings in tests (#2614)
• Send nonempty response to POST requests (#2606)
• Crash when unable to find stream instead of hanging (#2583)
• Stop using deprecated Config.getMilliseconds method (#2570)
• Move example configuration file to examples folder (#2566)
• Upgrade the log level for reports of stream nonexistence from INFO to ERROR (#2384)
• Crash rather than hanging when unable to bind to the supplied port (#2551)
• Bump Spray version to 1.3.3 (#2522)
• Bump Scala version to 2.10.5 (#2565)
• Fix omitted string interpolation (#2561)

Stream Enrich

• Bump to 0.8.0 (#2596)
• Bump Common Enrich to 0.23.0 (#2612)
• Bump Iglu Scala Client to 0.4.0 (#2688)
• Add configuration setting for MaxRecords (#2610)
• Use nonEmpty method to check whether lists are empty (#2608)
• Refactor functions to avoid return keyword (#2607)
• Upgrade the log level for reports of stream nonexistence from INFO to ERROR (#2598)
• Crash when unable to find stream instead of hanging (#2584)
• Add standard copyright notice to AbstractSourceSpec.scala (#2580)
• Make logging more succinct in case of failure (#1723)
• Move example configuration file to examples folder (#2567)
• Remove src/main/resolver.json.sample (#1932)
• Use json4s to combine the enrichment configuration JSONs (#2259)

Kinesis Elasticsearch Sink

• Bump to 0.6.0 (#2597)
• Add configuration setting for MaxRecords (#2611)
• Crash when unable to find stream instead of hanging (#2585)
• Move example configuration file to examples folder (#2568)

Release 79 Black Swan

Introduces our powerful new API Request Enrichment, plus a new HTTP Header Extractor Enrichment and several other improvements on the enrichments side

• Blog post

Documentation

• Removed closes from CHANGELOG tickets for R78 (#2534)

Common

• Changed Vagrantfile to use NFS and extra CPU cores by default (#2482)

Config

• Removed duplicated enabled property in ua_parser_config.json (#2424)
• Enabled switched to false in currency_conversion_config.json (#2327)
• Enabled switched to false in weather_enrichment_config.json (#2326)

EmrEtlRunner

• Bumped AMI version in example config to 4.5.0 (#2604)
• Updated hadoop_enrich version in config.yml.sample to 1.7.0 (#2661)
• Updated hadoop_shred version in config.yml.sample to 0.9.0 (#2662)

Scala Common Enrich

• Bumped user-agent-utils version to latest (#2516)
• Transaction item quantity type changed to JInteger (#2157)
• Bumped to 0.23.0 (#2486)
• Improved OWM error if user doesn't have historical weather (#2325)
• Added API Request Enrichment (#2051)
• Bumped Iglu Scala Client to 0.4.0 (#2333)
• Added HTTP Header Extractor Enrichment (#1373)

Scala Hadoop Enrich

• Bumped to 1.7.0 (#2446)
• Bumped Scala Common Enrich to 0.23.0 (#2485)
• Bumped Iglu Scala Client to 0.4.0 (#2478)
• Added test for API Request Enrichment (#2603)

Scala Hadoop Shred

• Bumped to 0.9.0 (#2480)
• Bumped Scala Common Enrich to 0.23.0 (#2481)
• Bumped Iglu Scala Client to 0.4.0 (#2449)

Release 78 Great Hornbill

Brings our Kinesis pipeline functionally up-to-date with our Hadoop pipeline, and makes various further improvements to the Kinesis pipeline.

• Blog post

Common

• Removed openjdk7 from .travis.yml (#2533)

Scala Common Enrich

• Bumped to 0.22.0
• Added handling for bad rows which are too long to print in full (closes #2419)

Kinesis

• Updated publish-kinesis-release.bash (closes #2477)

Scala Stream Collector

• Bumped to 0.6.0
• Added Scala Common Enrich as a library dependency (closes #2153)
• Added click redirect mode (closes #549)
• Configured the ability to use IP address as partition key (closes #2331)
• Converted bad rows to new format (closes #2006)
• Shared a single thread pool for all writes to Kinesis (closes #2369)
• Specified UTF-8 encoding everywhere (closes #2147)
• Made cookie name customizable, thanks @kazjote! (closes #2474)
• Added boolean collector.cookie.enabled setting (closes #2488)
• Made backoffPolicy fields macros (closes #2518)
• Updated AWS credentials to support iam/env/default not cpf (closes #1518)

Scala Kinesis Enrich

• Bumped to 0.7.0
• Renamed to Stream Enrich (closes #2418)
• Bumped Kinesis Client Library to 1.6.1 (closes #1823)
• Bumped Scala Common Enrich to 0.21.0 (closes #2033)
• Bumped Iglu Scala Client to 0.3.1 (closes #2080)
• Configured the ability to use IP address as partition key (closes #2332)
• Started emitting KCL metrics to CloudWatch (closes #2357)
• Converted bad rows to new format (closes #1207)
• Removed outdated comment about ClasspathPropertiesFileCredentialsProvider from sample config file (closes #1519)
• Removed redundant documentation from README (closes #2032)
• Updated test suite with valid self-describing JSONs (closes #2151)
• Updated Scala Tracker to 0.2.0 and enabled EC2 context (closes #2109)
• Updated to use new EtlPipeline (closes #1933)
• Specified UTF-8 encoding everywhere (closes #2148)

Kinesis Elasticsearch Sink

• Bumped to 0.5.0
• Bumped Kinesis Client Library to 1.6.1 (closes #1824)
• Bumped Scala Common Enrich to 0.22.0 (closes #2152)
• Added mixed output mode (closes #2412)
• Added new canonical event fields (closes #2089)
• Moved the stream-type setting into the main sink configuration object (closes #2490)
• Made source and sink fields macros (closes #2519)
• Renamed Build object to match project (closes #2002)
• Converted bad rows to new format (closes #1208)
• Updated schema regular expression in line with Iglu Central (fixes #1998)
• Cached the mapping of field name to field type (closes #2090)
• Specified UTF-8 encoding everywhere (closes #2149)
• Stopped sending timestamp instead of failure count (fixes #1951)
• Made performance of conversion from TSV to JSON linear (closes #1847)
• Updated to latest version of EnrichedEvent (closes #2089)

Release 77 Great Auk

Updates Snowplow to run on the new 4.x series of Elastic MapReduce releases

• Blog post

Documentation

• Updated tracker status table (#1999)
• Fixed incorrect entries in CHANGELOG (#2443)

Common

• Made optionality of Lingual and HBase in config.yml clearer (#2206)
• Fixed OpenJDK build in Travis CI (#2447)

Scala Hadoop Enrich

• Bumped to 1.6.0
• Bumped Scala Common Enrich to 0.21.0 (#2442)

Scala Common Enrich

• Bumped to 0.21.0
• Fixed exception for invalid API key in currency conversion (#2441)
• Fixed exception on same currency conversion (#2437)
• Switched from javax.script to org.mozilla.javascript for JavaScriptEnrichment (#2453)

Scala Hadoop Shred

• Bumped to 0.8.0
• Bumped Iglu Scala Client to 0.3.2 (#2319)

EmrEtlRunner

• Bumped to 0.21.0
• Attached monitoring tags to jobflow (#425)
• Now throwing exception if processing thrift with --skip s3distcp or AMI 2.x.x (#1648)
• Added bootstrap action to prepare AMI >= 3.8.0 (#2320)
• Bumped Elasticity to 6.0.7 (#2400)
• Added support for Amazon EMR 4.x.x series (#1926)
• Prevented bad CLI options from throwing stack trace (#1930)
• Made error for nonempty processing bucket collector-agnostic (#1961)
• Bumped Ruby Tracker to 0.5.2 (#2143)
• Improved retry logic for EMR bootstrap timeouts (#2150)
• Excluded previously-built executables from the build (#2163)
• Added support for additional_info in EMR section of configuration (#2211)
• Added Elasticsearch stage to help message (#2323)
• Updated hadoop_enrich version in config.yml.sample to 1.6.0 (#2459)
• Updated hadoop_shred version in config.yml.sample to 0.8.0 (#2370)
• Removed snowplow-emr-etl-runner.sh (#2445)

StorageLoader

• Bumped to 0.7.0
• Added support for supplying config file as Base64-encoded string (#2227)
• Added ability to retrieve AWS credentials from EC2 role (#2226)
• Excluded previously-built executables from the build (#2164)
• Started printing stack trace for failures not caused by bad configuration (#2160)
• Bumped Ruby Tracker to 0.5.2 (#2144)
• Moved ANALYZE statements after VACUUM statements (#1361)
• Added resolver config option to snowplow-runner-and-loader.sh (#2170)
• Updated snowplow-runner-and-loader.sh to use JRuby binaries (#2233)
• Removed snowplow-storage-loader.sh (#2444)
• Wrote JSON Path file for com.optimizely/visitor_dimension event (#2436)
• Wrote JSON Path file for com.optimizely/visitor_audience event (#2435)
• Wrote JSON Path file for com.optimizely/visitor event (#2434)
• Wrote JSON Path file for com.optimizely/variation event (#2433)
• Wrote JSON Path file for com.optimizely/state event (#2432)
• Wrote JSON Path file for com.optimizely/experiment event (#2431)
• Wrote JSON Path file for io.augur.snowplow/identity_lite (#1958)

Redshift

• Wrote Redshift DDL for com.optimizely/visitor_dimension event (#2430)
• Wrote Redshift DDL for com.optimizely/visitor_audience event (#2429)
• Wrote Redshift DDL for com.optimizely/visitor event (#2428)
• Wrote Redshift DDL for com.optimizely/variation event (#2427)
• Wrote Redshift DDL for com.optimizely/state event (#2426)
• Wrote Redshift DDL for com.optimizely/experiment event (#2425)
• Added Redshift DDL for io.augur.snowplow/identity_lite (#1957)

Release 76 Changeable Hawk-Eagle

Introduces an event de-duplication process which runs on Hadoop, plus an important bug fix for our recent SendGrid webhook support

• Blog post

Scala Hadoop Enrich

• Bumped to 1.5.1
• Bumped Scala Common Enrich to 0.20.1 (#2338)

Scala Common Enrich

• Bumped to 0.20.0
• Now using only base MIME type in content-type check for SendGrid Adapter (#2328)

Scala Hadoop Shred

• Bumped to 0.7.0
• Fixed good tests' checks for empty paths (#2278)
• Now deduplicating event_id and event_fingerprint pairs (#2246)
• Fixed incorrect event in SchemaValidationFailed1Spec (#2355)
• Updated tests to check atomic-events output (#2264)
• Now only writes atomic-events if JSONs shred successfully (#2245)
• Removed empty SchemaValidationFailed2Spec (#2271)
• Fixed test suite issue with multiple input lines (#2270)

EmrEtlRunner

• Updated hadoop_enrich version in config.yml.sample to 1.5.1 (#2339)
• Changed in bucket example in config.yml.sample to s3://my-in-bucket (#2358)
• Updated archive bucket examples in config.yml (#2368)
• Updated hadoop_shred version in config.yml.sample to 0.7.0 (#2360)

StorageLoader

• Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/action (#2136)
• Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/actionFieldObject (#2135)
• Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/impressionFieldObject (#2134)
• Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/productFieldObject (#2133)
• Wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/promotionFieldObject (#2132)

Redshift

• Added Redshift DDL for com.google.analytics.enhanced-ecommerce/promotionFieldObject (#2131)
• Added Redshift DDL for com.google.analytics.enhanced-ecommerce/productFieldObject (#2130)
• Added Redshift DDL for com.google.analytics.enhanced-ecommerce/impressionFieldObject (#2129)
• Added Redshift DDL for com.google.analytics.enhanced-ecommerce/actionFieldObject (#2128)
• Added Redshift DDL for com.google.analytics.enhanced-ecommerce/action (#2127)

Release 75 Long-Legged Buzzard

Add support for Urban Airship and SendGrid webhooks

• Blog post

Scala Hadoop Enrich

• Bumped to 1.5.0
• Bumped Scala Common Enrich to 0.20.0 (#2200)
• Added test for loading Urban Airship Connect ndjson files (#2168)
• Added test for SendGrid Adapter (#2194)

Scala Common Enrich

• Bumped to 0.20.0
• Added JsonLoader for Urban Airship, Mixpanel et al (#2210)
• Added Adapter to pre-process Urban Airship events (#2167)
• Abstracted Mandrill reformatParameters function into Adapter (#2171)
• Added Adapter to pre-process SendGrid events (#1161)

EmrEtlRunner

• Bumped to 0.20.0
• Updated hadoop_enrich version in config.yml.sample to 1.5.0 (#2282)
• Added raw s3 -> hdfs step with group by (#2253)
• Added directory flattening code (#2232)
• Added support for ndjson loader format (#2251)
• Improved test coverage of runner.rb (#2250)

Redshift

• Added Redshift DDL for a com.sendgrid/processed event (#2172)
• Added Redshift DDL for a com.sendgrid/dropped event (#2173)
• Added Redshift DDL for a com.sendgrid/delivered event (#2174)
• Added Redshift DDL for a com.sendgrid/deferred event (#2175)
• Added Redshift DDL for a com.sendgrid/bounce event (#2176)
• Added Redshift DDL for a com.sendgrid/open event (#2177)
• Added Redshift DDL for a com.sendgrid/click event (#2178)
• Added Redshift DDL for a com.sendgrid/spamreport event (#2179)
• Added Redshift DDL for a com.sendgrid/unsubscribe event (#2180)
• Added Redshift DDL for a com.sendgrid/group_unsubscribe event (#2181)
• Added Redshift DDL for a com.sendgrid/group_resubscribe event (#2182)
• Added Redshift DDL for com.urbanairship.connect/UNINSTALL event (#2283)
• Added Redshift DDL for com.urbanairship.connect/TAG_CHANGE event (#2284)
• Added Redshift DDL for com.urbanairship.connect/SEND event (#2285)
• Added Redshift DDL for com.urbanairship.connect/RICH_READ event (#2286)
• Added Redshift DDL for com.urbanairship.connect/RICH_DELIVERY event (#2287)
• Added Redshift DDL for com.urbanairship.connect/RICH_DELETE event (#2288)
• Added Redshift DDL for com.urbanairship.connect/REGION event (#2289)
• Added Redshift DDL for com.urbanairship.connect/PUSH_BODY event (#2290)
• Added Redshift DDL for com.urbanairship.connect/OPEN event (#2291)
• Added Redshift DDL for com.urbanairship.connect/LOCATION event (#2292)
• Added Redshift DDL for com.urbanairship.connect/IN_APP_MESSAGE_RESOLUTION event (#2293)
• Added Redshift DDL for com.urbanairship.connect/IN_APP_MESSAGE_EXPIRATION event (#2294)
• Added Redshift DDL for com.urbanairship.connect/IN_APP_MESSAGE_DISPLAY event (#2295)
• Added Redshift DDL for com.urbanairship.connect/FIRST_OPEN event (#2296)
• Added Redshift DDL for com.urbanairship.connect/CUSTOM event (#2297)
• Added Redshift DDL for com.urbanairship.connect/CLOSE event (#2298)

StorageLoader

• Added JSON Path file for com.sendgrid/processed event (#2183)
• Added JSON Path file for com.sendgrid/dropped event (#2184)
• Added JSON Path file for com.sendgrid/delivered event (#2185)
• Added JSON Path file for com.sendgrid/deferred event (#2186)
• Added JSON Path file for com.sendgrid/bounce event (#2187)
• Added JSON Path file for com.sendgrid/open event (#2188)
• Added JSON Path file for com.sendgrid/click event (#2189)
• Added JSON Path file for com.sendgrid/spamreport event (#2190)
• Added JSON Path file for com.sendgrid/unsubscribe event (#2191)
• Added JSON Path file for com.sendgrid/group_unsubscribe event (#2192)
• Added JSON Path file for com.sendgrid/group_resubscribe event (#2193)
• Added JSON Path file for com.urbanairship.connect/UNINSTALL event (#2299)
• Added JSON Path file for com.urbanairship.connect/TAG_CHANGE event (#2300)
• Added JSON Path file for com.urbanairship.connect/SEND event (#2301)
• Added JSON Path file for com.urbanairship.connect/RICH_READ event (#2302)
• Added JSON Path file for com.urbanairship.connect/RICH_DELIVERY event (#2303)
• Added JSON Path file for com.urbanairship.connect/RICH_DELETE event (#2304)
• Added JSON Path file for com.urbanairship.connect/REGION event (#2305)
• Added JSON Path file for com.urbanairship.connect/PUSH_BODY event (#2306)
• Added JSON Path file for com.urbanairship.connect/OPEN event (#2307)
• Added JSON Path file for com.urbanairship.connect/LOCATION event (#2308)
• Added JSON Path file for com.urbanairship.connect/IN_APP_MESSAGE_RESOLUTION event (#2309)
• Added JSON Path file for com.urbanairship.connect/IN_APP_MESSAGE_EXPIRATION event (#2310)
• Added JSON Path file for com.urbanairship.connect/IN_APP_MESSAGE_DISPLAY event (#2311)
• Added JSON Path file for com.urbanairship.connect/FIRST_OPEN event (#2312)
• Added JSON Path file for com.urbanairship.connect/CUSTOM event (#2313)
• Added JSON Path file for com.urbanairship.connect/CLOSE event (#2314)

Data modeling

• Removed events enriched from web-recalculate (#2275)
• Added cookie-to-user-id map to web-recalculate (#2274)

Release 74 European Honey Buzzard

Adds a weather enrichment

• Blog post

Common

Common: added encrypted OWM API key to .travis.yml (#2243)

Scala Hadoop Enrich

Bumped to 1.4.0
Bumped Scala Common Enrich to 0.19.0 (#2255)

Scala Common Enrich

Bumped to 0.19.0
Added weather enrichment (#456)
Fixed issue with BC timestamp in ExtractEventTypeSpec (#2257)
Fixed currency conversion enrichment's test for invalid API key (#2258)

StorageLoader

Wrote JSON path file for org.openweathermap/weather (#2240)

Redshift

Added Redshift DDL for org.openweathermap/weather (#2241)

Release 73 Cuban Macaw

Loads bad rows in batch pipeline into Elasticsearch, and formally separates the Snowplow enriched event format from the TSV format used to load Redshift.

• Blog post

EmrEtlRunner

• Bumped to 0.19.0
• Added hadoop_elasticsearch to config.yml.sample (#2124)
• Added support for Elasticsearch in targets section of config (#826)
• Bumped Elasticity to 6.0.5 (#2026)
• Stopped skipping the whole job just because enrich and shred are being skipped (#2049)

Scala Common Enrich

• Bumped Iglu Scala Client to 0.3.1 (#2079)
• Bumped version to 0.18.0
• Moved ScalazArgs into shared library (#2010)
• Removed executable bit from Scala source files (#2022)
• Removed JSON length checks (#2041)
• Removed truncation code (#2044)
• Stopped attempting to catch fatal errors (#2045)

Scala Hadoop Enrich

• Bumped to 1.3.0
• Bumped Scala Common Enrich to 0.18.0 (#2015)
• Added Iglu Scala Client as an explicit dependency (#2115)
• Added .forceToDisk to speed up run (#859)
• Started using Scala Common Enrich's version of ScalazArgs (#2013)

Scala Hadoop Shred

• Bumped to 0.6.0
• Added .forceToDisk to common to speed up run (#2039)
• Bumped Iglu Scala Client to 0.3.1 (#2081)
• Bumped Scala Common Enrich to 0.18.0 (#2016)
• Applied truncation logic to atomic-events TSV (#2042)
• Processed enriched events for atomic.events removing JSON fields (#1731)
• Started using Scala Common Enrich's version of ScalazArgs (#2014)

Storage

• Fixed README's link to architecture image, thanks @miike! (#2156)

Hadoop Elasticsearch Sink

• Added. (#824)

StorageLoader

• Bumped to 0.6.0
• Added tcpKeepAlive=true to JDBC for long-running COPYs via NAT (#2145)
• Fixed setup guide link in README, thanks @diamondo25! (#2025)
• Loaded atomic.events from shredded folder (#1795)

Postgres

• Added migration script for 0.6.0 to 0.7.0 (#2047)
• Removed JSON fields from atomic.events (#1949)

Redshift

• Added migration script for 0.4.0 to 0.8.0 (#2155)
• Added migration script for 0.5.0 to 0.8.0 (#2119)
• Added migration script for 0.6.0 to 0.8.0 (#2120)
• Added migration script for 0.7.0 to 0.8.0 (#2048)
• Removed JSON fields from atomic.events (#1849)

Data Modeling

• Added separators to custom fingerprint in deduplication queries (#2198)
• Renamed dvce_tstamp to dvce_created_tstamp in basic recipes (#2166)
• Removed JSON fields from deduplication queries (#2197)