Fix HDFS copy logic

[xiangfu0]

Current hdfs cp will copy all the files from src dir to dest dir as a flatten layout.
This fix will ensure the dest directory could retain the same structure as src directory.

For HadoopPinotFS.copy(src,dest) behavior:

Before:

src/2014/01/01/myTable-20140101.avro
src/2014/01/02/myTable-20140102.avro
src/2014/01/03/myTable-20140103.avro
src/2014/01/04/myTable-20140104.avro

=>

dest/myTable-20140101.avro
dest/myTable-20140102.avro
dest/myTable-20140103.avro
dest/myTable-20140104.avro

New logic:

src/2014/01/01/myTable-20140101.avro
src/2014/01/02/myTable-20140102.avro
src/2014/01/03/myTable-20140103.avro
src/2014/01/04/myTable-20140104.avro

=>

dest/2014/01/01/myTable-20140101.avro
dest/2014/01/02/myTable-20140102.avro
dest/2014/01/03/myTable-20140103.avro
dest/2014/01/04/myTable-20140104.avro

[jackjlli]

I saw copy method is used in HadoopSegmentGenerationJobRunner, where it copies the newly generated segments from staging dir to the final output dir. But why do the new segments store in different sub-dirs that makes you make this change?

[xiangfu0]

I saw copy method is used in HadoopSegmentGenerationJobRunner, where it copies the newly generated segments from staging dir to the final output dir. But why do the new segments store in different sub-dirs that makes you make this change?

For all other pinot fs, the behavior keeps same as directory copying retains sub-dir structure. Otherwise, what if there are files with same name?

If user wants to just copy all the files recursively from a directory and flatten to a new directory, then user should list all the files then do file copy to dest dir.

[jackjlli]

Thanks for the explanation! Minor comments but LGTM.

[codecov-io]

Codecov Report

Merging #5218 into master will increase coverage by 0.25%.
The diff coverage is 73.86%.

@@             Coverage Diff              @@
##             master    #5218      +/-   ##
============================================
+ Coverage     65.90%   66.16%   +0.25%     
============================================
  Files          1052     1067      +15     
  Lines         54170    54311     +141     
  Branches       8078     8074       -4     
============================================
+ Hits          35702    35935     +233     
+ Misses        15819    15741      -78     
+ Partials       2649     2635      -14     

Impacted Files	Coverage Δ	Complexity Δ
...e/pinot/broker/api/resources/PinotBrokerDebug.java	76.66% <ø> (ø)	0.00 <0.00> (ø)
.../BrokerResourceOnlineOfflineStateModelFactory.java	55.81% <ø> (ø)	0.00 <0.00> (ø)
.../pinot/broker/broker/helix/HelixBrokerStarter.java	71.97% <ø> (ø)	0.00 <0.00> (ø)
...thandler/SingleConnectionBrokerRequestHandler.java	92.68% <ø> (ø)	0.00 <0.00> (ø)
...ting/instanceselector/InstanceSelectorFactory.java	71.42% <ø> (ø)	0.00 <0.00> (ø)
...er/routing/segmentpruner/SegmentPrunerFactory.java	83.33% <ø> (ø)	0.00 <0.00> (ø)
...outing/segmentselector/SegmentSelectorFactory.java	60.00% <ø> (ø)	0.00 <0.00> (ø)
...oker/routing/timeboundary/TimeBoundaryManager.java	87.50% <ø> (ø)	0.00 <0.00> (ø)
...mmon/assignment/InstanceAssignmentConfigUtils.java	67.50% <ø> (ø)	0.00 <0.00> (?)
...not/common/assignment/InstancePartitionsUtils.java	73.17% <ø> (ø)	0.00 <0.00> (ø)
... and 283 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 79c0838...a208fb0. Read the comment docs.