EmrEtlRunner: fix srcPattern for copying stream enriched data to HDFS

[chuwy]

Related to #3717

When we're resuming from shred in stream enrich mode, S3DistCp tries to copy data from enriched/good, not from enriched/good/run=2018... and fails because of that.

Due this bug, we can recover R102 stream enrich mode only by re-staging enriched data back to enriched.stream.

[alexanderdean]

What's our chance of getting this into R103?

[chuwy]

This is already implemented, starting to test it. So question is for @BenFradet.

[BenFradet]

103 has been in code freeze for over a week, so I'm 👎

[BenFradet]

plus it'd make sense to integrate at least #3719 with this

[alexanderdean]

Makes sense - let's start sketching out the next high priority batch release. @chuwy can you compose that milestone please.

[chuwy]

Ok, will do.

[chuwy]

Turns out fix should (can) be different from one described in title. When we're resuming from shred in batch-enrich mode, S3DistCp knows nothing about run-folder and also uses enriched/good, but with --srcPattern .*part-.*, which seems to handle files from subfolders. With stream-enrich mode we use --srcPattern .+, which somehow doesn't handle subfolders in the same way.

[chuwy]

Don't understand why, but --srcPattern .*\.gz did the trick.

[BenFradet]

this is locking us down to a particular file format (gz) which I don't really like :(

[chuwy]

Do you mean because we can support other formats in future? AFAIK gz right now is only format for enriched data produced by S3 Loader.

[chuwy]

I'll try to test it with --srcPattern .* (which I guess can be different from initial .+), but also not sure if this is more bullet-proof option.

[chuwy]

Ok, .* also worked. @BenFradet can you confirm .* seems like a better option for you than .*\.gz?

[BenFradet]

As long as there are no other files being unnecessarily moved, yes

[chuwy]

Ok, no unnecessarily files were moved. Pushing rc then.

[chuwy]

Sorry, I was wrong this regex handles $folder$ files and tries to move them from enriched/good to HDFS for shredding. I cannot find any evidences that S3DistCp ignores empty files, so I think it still would be better to stick with .gz.