[SPARK-20448][DOCS] Document how FileInputDStream works with object storage

[steveloughran]

Change-Id: I88c272444ca734dc2cbc2592607c11287b90a383

What changes were proposed in this pull request?

The documentation on File DStreams is enhanced to

1. Detail the exact timestamp logic for examining directories and files.
2. Detail how object stores different from filesystems, and so how using them as a source of data should be treated with caution, possibly publishing data to the store differently (direct PUTs as opposed to stage + rename)

How was this patch tested?

n/a

[SparkQA]

Test build #76103 has finished for PR 17743 at commit c83af37.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #76104 has finished for PR 17743 at commit 1e620ce.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[steveloughran]

Just reread this; still looks correct. Review comments welcome

[srowen]

It looks OK to me. I believe the content is accurate though I'm not as sure about the behavior of object stores. I take your word for it. Earlier I had wondered whether this level of detail is useful to the user but I think it is here.

[steveloughran]

People don't realise how much object stores aren't file systems until they discover all their assumptions are broken.

Once you know how they work, you can set up a workflow which is more efficient and reliable.

[SparkQA]

Test build #3934 has finished for PR 17743 at commit 1e620ce.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master