Update Main Events schema to match event ping events #447
Conversation
|
circleci is showing a compilation error: |
sunahsuh
force-pushed
the
update-main-events
branch
from
f625eef
to
2ce7f04
Compare
| .select("document_id", "client_id", "normalized_channel", "country", "locale", "app_name", "app_version", "os", | ||
| "os_version", "e10s_enabled", "subsession_start_date", "subsession_length", "sync_configured", | ||
| "sync_count_desktop", "sync_count_mobile", "timestamp", "sample_id", "active_experiment_id", | ||
| "active_experiment_branch", "experiments", "events") |
There was a problem hiding this comment.
Why is this select going away? Is it simply redundant with the exploded.selectExpr below? Is Spark smart enough to prune the unused columns early if we don't select down the column list here?
There was a problem hiding this comment.
Theoretically it shouldn't matter since this method only does transformations and the triggering action is the write in the calling method.
| @@ -84,5 +78,7 @@ class MainEventsViewTest extends FlatSpec with Matchers with DataFrameSuiteBase | |||
| sampledEvents | |||
| .where("document_id = '72062950-3daf-450e-adfd-58eda3151a97'") | |||
| .count should be(1) | |||
|
|
|||
| events.select("session_start_time").collect.head.getLong(0) should be(1528394400000) | |||
There was a problem hiding this comment.
Probably need an L suffix on the literal here.
sunahsuh
force-pushed
the
update-main-events
branch
from
2ce7f04
to
71a3a07
Compare
Codecov Report
@@ Coverage Diff @@
## master #447 +/- ##
=======================================
Coverage 69.85% 69.85%
=======================================
Files 45 45
Lines 4011 4011
Branches 83 81 -2
=======================================
Hits 2802 2802
Misses 1209 1209
Continue to review full report at Codecov.
|
As mentioned in mozilla/telemetry-streaming#149, this is the update to main ping events output to match the new schema from events ping output. The only really interesting part is the new session_start_time field, which matches the format of the session_start_time payload field in the events ping.