Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scala Common Enrich: remove shred function #4233

Closed
chuwy opened this issue Nov 5, 2019 · 4 comments
Closed

Scala Common Enrich: remove shred function #4233

chuwy opened this issue Nov 5, 2019 · 4 comments
Assignees
@chuwy
Milestone
R118 Morgantina (...

Comments

@chuwy
Copy link
Contributor

chuwy commented Nov 5, 2019

We used to have a big Shred object with logic related to extraction and validation of self-describing data, but only very few methods were used in actual enrichers, everything else was used in RDB Shredder. Since R31 we use Scala Analytics SDK for these purposes and many shredding methods are unused.
@chuwy chuwy self-assigned this Nov 5, 2019
@benjben
Copy link
Contributor

benjben commented Nov 5, 2019

By remove shredding you mean removed unused methods in Shredder right?
We still need Shredder to validate the contexts and the unstructured event.

@benjben
Copy link
Contributor

benjben commented Nov 5, 2019

Maybe we could do it at the same time as #3795

@chuwy
Copy link
Contributor Author

chuwy commented Nov 5, 2019 •
edited

By remove shredding you mean removed unused methods in Shredder right?

Yes, sorry I didn't know how to phrase it better. The actual shred method is not used anywhere, but extractAndValidate-like are necessary for enrichment (though can be simplified).

@chuwy chuwy changed the title Scala Common Enrich: remove shredding Scala Common Enrich: remove shred function Nov 6, 2019
chuwy added a commit that referenced this issue Nov 6, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
042b1ec
@chuwy chuwy mentioned this issue Nov 6, 2019
Closed
13 tasks
@chuwy
Copy link
Contributor Author

chuwy commented Nov 6, 2019

Also removed unnecessary TypeHierarchy class, added some tests and simplified types in few related functions.

chuwy added a commit that referenced this issue Nov 6, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
1ae070b
chuwy added a commit that referenced this issue Nov 6, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
4ce86c8
chuwy added a commit that referenced this issue Nov 6, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
cc515f5
chuwy added a commit that referenced this issue Nov 17, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
2bd8881
chuwy added a commit that referenced this issue Nov 17, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
711c911
chuwy added a commit that referenced this issue Nov 22, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
bf01cd4
chuwy added a commit that referenced this issue Nov 23, 2019
@chuwy
Scala Common Enrich: remove shred function (close #4233)
23f0908
benjben pushed a commit that referenced this issue Dec 4, 2019
@chuwy @benjben
Scala Common Enrich: remove shred function (close #4233)
c3016e3
benjben pushed a commit that referenced this issue Dec 6, 2019
@chuwy @benjben
Scala Common Enrich: remove shred function (close #4233)
eea8ff2
benjben pushed a commit that referenced this issue Dec 9, 2019
@chuwy @benjben
Scala Common Enrich: remove shred function (close #4233)
8d06709
benjben pushed a commit that referenced this issue Dec 9, 2019
@chuwy @benjben
Scala Common Enrich: remove shred function (close #4233)
adccd23
benjben pushed a commit that referenced this issue Dec 12, 2019
@chuwy @benjben
Scala Common Enrich: remove shred function (close #4233)
01dd12b
oguzhanunlu pushed a commit to snowplow/common-enrich that referenced this issue May 29, 2020
@chuwy @oguzhanunlu
Remove shred function (close snowplow/snowplow#4233)
9f2fb0e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chuwy chuwy

Labels
None yet
Projects
None yet
Milestone
R118 Morgantina (New bad row format)
Development

No branches or pull requests
2 participants
@chuwy @benjben