Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add type-safe Event API #53

Closed
chuwy opened this issue May 28, 2018 · 3 comments
Closed

Add type-safe Event API #53

chuwy opened this issue May 28, 2018 · 3 comments
Assignees
Milestone

Comments

@chuwy
Copy link
Contributor

chuwy commented May 28, 2018

Currently return type for most functions in EventTransformer is a String, which represents enriched event turned into a JSON object. While this is very unopionated, minimalistic and exactly what most Spark users need - it usually involves a lot of post-processing, e.g:

  1. We always know that result is a JSON object, but in order to pull something out we need to do parse(result) and even parse(result).asInstanceOf[JObject]
  2. Multiple fields in enriched event are required and thus must be there by definition, but we need to access them carefully: parsedJson.map("event_id").getOrElse(throw new RuntimeException("event_id is not present in enriched event")
  3. To get list of shredded types we have a separate function jsonifyWithInventory.

I think all this information should be in one typesafe container with asString helper function, something like:

case class EnrichedEvent(json: JObject, shreddedTypes: Set[IgluUri], collectorTstamp: Instant, eventId: UUID) {
  lazy val asString: String = compact(json)
}

This will also give us #43 for free as this is a matter of one more asStringWithoutNulls function (but with better name).

@chuwy chuwy added this to the Version 0.4.0 milestone May 28, 2018
@chuwy
Copy link
Contributor Author

chuwy commented May 28, 2018

Will be very useful for Snowflake Transformer and almost vital for BigQuery Loader.

@chuwy chuwy self-assigned this May 28, 2018
@chuwy
Copy link
Contributor Author

chuwy commented May 28, 2018

Actually, this is vital for any Loader (including RDB Shredder) and for atomic events refactoring.

@chuwy
Copy link
Contributor Author

chuwy commented May 29, 2018

chuwy added a commit that referenced this issue Dec 28, 2018
@chuwy chuwy mentioned this issue Dec 28, 2018
@rzats rzats changed the title Make EventTransformer API type-safe Add type-safe Event API Feb 1, 2019
rzats added a commit that referenced this issue Feb 1, 2019
rzats added a commit that referenced this issue Feb 1, 2019
rzats added a commit that referenced this issue Feb 4, 2019
rzats added a commit that referenced this issue Feb 4, 2019
rzats added a commit that referenced this issue Feb 6, 2019
rzats added a commit that referenced this issue Feb 13, 2019
@rzats rzats closed this as completed in 3b5e1a1 Feb 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant