[SPARK-18960][SQL][SS] Avoid double reading file which is being copied.

[uncleGen]

What changes were proposed in this pull request?

In HDFS, when we copy a file into target directory, there will a temporary ._COPY_ file for a period of time. The duration depends on file size. If we do not skip this file, we will may read the same data for two times.

How was this patch tested?

update unit test

[uncleGen]

cc @zsxwing @liancheng

[SparkQA]

Test build #70456 has started for PR 16370 at commit 1d248c3.

[uncleGen]

@AmplabJenkins retest it please

[viirya]

retest this please.

[SparkQA]

Test build #70473 has finished for PR 16370 at commit 1d248c3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[uncleGen]

unrelated errors, retest this please.

[SparkQA]

Test build #70476 has finished for PR 16370 at commit 1d248c3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

That sounds reasonable to me.

[viirya]

LGTM.

[zsxwing]

A similar PR got refused as copy is not the correct way to stream files: #3203

You should always move files instead of copying them.

[uncleGen]

@zsxwing Thanks for your reminder!!
In some ways, we really can evade this issue, just like not use -cp. But this is an user-side behaviour, we can not ensure everyone knows and uses correct ways to move data. It may confuse users if they do not know this issue. Besides, current change is so tiny that does not have any harm to codes. So, IMHO, it is OK to add the check for protection. What do you think？

[srowen]

I don't see a harm in ignoring these files, other comments notwithstanding. In fact, I think it's mandatory. The HDFS copy mechanism basically is copy (to a .COPYING file), then move (rename). This is exactly what anyone copying from a different FS would have to do anyway manually as you can't rename a file that isn't already on the FS, and don't necessarily have ability to write anywhere else.

[uncleGen]

@zsxwing Is there any farther feedback？

[uncleGen]

@srowen It looks like @zsxwing has no time to feed back.

[srowen]

Merged to master

[zsxwing]

Sorry for the delay. LGTM.