Crash aggregates

[Uberi]

Intended to have behaviour equivalent to https://github.com/mozilla/moz-crash-rate-aggregates, but faster.

It's also a little bit simpler, since the Spark Scala API is somewhat richer and more robust than PySpark.

[codecov-io]

Current coverage is 54.26%

Merging #56 into master will increase coverage by +0.08%

1. 2 files (not in diff) in ...streams/main_summary were deleted. more
2. 4 files (not in diff) in ...c/main/scala/streams were deleted. more
3. 2 files (not in diff) in .../src/main/scala/heka were deleted. more
4. 8 files (not in diff) in ...-view/src/main/scala were deleted. more
5. File ...lientCountView.scala (not in diff) was modified. more
   • Misses +2
   • Partials 0
   • Hits 0

@@             master        #56   diff @@
==========================================
  Files            32         17    -15   
  Lines          1402       1539   +137   
  Methods        1338       1481   +143   
  Messages          0          0          
  Branches         48         55     +7   
==========================================
+ Hits            759        835    +76   
- Misses          643        704    +61   
  Partials          0          0          

Powered by Codecov. Last updated by d260b7b...35e3ea1

[vitillo]

We are trying to move away from Avro schema in new datasets and use only the one provided by SparkSQL. Basically you would have to port over the same code you used in your Python version.

[Uberi]

After a lot of debugging, I found out that Parquet-Avro 1.8.1 (the most recent version) entirely breaks DataFrame.write.parquet:

libraryDependencies += "org.apache.parquet" % "parquet-avro" % "1.8.1",

Basically, if we even install parquet-avro, writing Parquet files from dataframes no longer works:

java.lang.NoSuchMethodError: org.apache.parquet.schema.Types$MessageTypeBuilder.addFields([Lorg/apache/parquet/schema/Type;)Lorg/apache/parquet/schema/Types$GroupBuilder;
(...16 more lines...)

There's a resolved issue in Parquet for this, but it's only for 1.8.2-SNAPSHOT, so it isn't really considered stable yet.

We should (1) move this into a separate project (so we don't need to depend on parquet-avro at all), or (2) move away from the Avro stuff all at once.

[vitillo]

Comment is incomplete.

[Uberi]

Changed:

• Added comment about ping aggregates.
• Clean up code a bit.
• Use list with single element to identify top-level fields.

[vitillo]

This looks good, did you compare the output generated by this job with the one generated by the Python one?

[vitillo]

Could you please also add some documentation about the metrics collected in this job similarly to what you have written down for the Python one?

[Uberi]

Changes:

• Add some better docs.
• Test the scala aggregator against the crash aggregator on nightly, aurora, and beta:
  • Scala output: https://console.aws.amazon.com/s3/home?region=us-west-2#&bucket=telemetry-test-bucket&prefix=crash_aggregates/v1/submission_date%3D2016-04-10/
  • In the Scala one, there's 10 files since it was set to coalesce(10). In this PR it's set back to 1.
  • Python output: https://console.aws.amazon.com/s3/home?region=us-west-2#&bucket=telemetry-test-bucket&prefix=crash_aggregates/vPy/submission_date%3D2016-04-10/
• The differences can be explained by the Scala version being slightly more strict: it doesn't allow crash pings that have subsession lengths or main pings with no subsession lengths. In Python these are still processed.
• Upload path for parquet files updated to match the Python one.

[vitillo]

Looks good, thanks!