Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EmrEtlRunner: add support for shredded TSV data #4074

Closed
chuwy opened this issue May 8, 2019 · 1 comment
Closed

EmrEtlRunner: add support for shredded TSV data #4074

chuwy opened this issue May 8, 2019 · 1 comment
Assignees
@chuwy
Milestone
R116 Madara Rider...

Comments

@chuwy
Copy link
Contributor

chuwy commented May 8, 2019

In Redshift auto-migrations project (snowplow/snowplow-rdb-loader#7) we're planning to implement a tabular shredding blacklisting that needs to be passed to the RDB Shredderl. In general, I think whole configuration file should be sent as we extract manifest-related properties via EER anyway.

if storage.versions.rdb_shredder < "0.16.0" and storageTarget != null:
  job.addOption("target", base64(storageTarget))
@chuwy chuwy mentioned this issue May 8, 2019
Closed
@chuwy chuwy self-assigned this Jun 24, 2019
chuwy added a commit that referenced this issue Jun 24, 2019
@chuwy
EmrEtlRunner: pass storage target configuration to RDB Shredder (close …
6f65fa4 
…#4074)
This was referenced Jun 24, 2019
Closed
Closed
@chuwy chuwy added this to the R116 Madara Rider milestone Jun 25, 2019
@chuwy chuwy changed the title EmrEtlRunner: pass storage target configuration to RDB Shredder EmrEtlRunner: add support for shredded TSV data Aug 22, 2019
@chuwy
Copy link
Contributor Author

chuwy commented Aug 22, 2019

Rephrased the title because it needs more than just passing corresponding config.

chuwy added a commit that referenced this issue Aug 22, 2019
@chuwy
EmrEtlRunner: add support for shredded TSV data (close #4074)
7422462
chuwy added a commit that referenced this issue Aug 22, 2019
@chuwy
EmrEtlRunner: add support for shredded TSV data (close #4074)
d66c237
dilyand pushed a commit that referenced this issue Sep 2, 2019
@chuwy @dilyand
EmrEtlRunner: add support for shredded TSV data (close #4074)
d44cce1 
(cherry picked from commit d66c237)
dilyand pushed a commit that referenced this issue Sep 4, 2019
@chuwy @dilyand
EmrEtlRunner: add support for shredded TSV data (close #4074)
a88803f 
(cherry picked from commit d66c237)
dilyand pushed a commit that referenced this issue Sep 12, 2019
@chuwy @dilyand
EmrEtlRunner: add support for shredded TSV data (close #4074)
b05451e 
(cherry picked from commit d66c237)
dilyand pushed a commit that referenced this issue Sep 12, 2019
@chuwy @dilyand
EmrEtlRunner: add support for shredded TSV data (close #4074)
c2bcb04 
(cherry picked from commit d66c237)
peel pushed a commit to snowplow/emr-etl-runner that referenced this issue May 26, 2020
@chuwy @peel
Add support for shredded TSV data (close snowplow/snowplow#4074)
0dff3b2
peel pushed a commit to snowplow/emr-etl-runner that referenced this issue May 26, 2020
@chuwy @peel
Add support for shredded TSV data (close snowplow/snowplow#4074)
8063ae3
@stdfalse stdfalse mentioned this issue May 27, 2020
Closed
peel pushed a commit to snowplow/emr-etl-runner that referenced this issue May 28, 2020
@chuwy @peel
Add support for shredded TSV data (close snowplow/snowplow#4074)
6c5791e
peel pushed a commit to snowplow/emr-etl-runner that referenced this issue May 28, 2020
@chuwy @peel
Add support for shredded TSV data (close snowplow/snowplow#4074)
16947e0
peel pushed a commit to snowplow/emr-etl-runner that referenced this issue May 28, 2020
@chuwy @peel
Add support for shredded TSV data (close snowplow/snowplow#4074)
450ee76
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chuwy chuwy

Labels
None yet
Projects
None yet
Milestone
R116 Madara Rider (SSC and EER with a...
Development

No branches or pull requests
2 participants
@chuwy @BenFradet