[SPARK-3916] [Streaming] discover new appended data for fileStream()

[davies]

In a case that new data will be appended to existed files continuously, then fileStream() should discovery the new appended data. This patch brings this ability to fileStream.

In order to get an RDD based on partial data of file, added a private partialHadoopRDD() API.

cc @tdas

[SparkQA]

QA tests have started for PR 2806 at commit 05ad755.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2806 at commit 05ad755.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class CustomPathFilter(maxModTime: Long)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21736/
Test FAILed.

[SparkQA]

QA tests have started for PR 2806 at commit 09561e8.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2806 at commit 09561e8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class CustomPathFilter(maxModTime: Long)

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21739/
Test FAILed.

[SparkQA]

QA tests have started for PR 2806 at commit 09561e8.

• This patch merges cleanly.

[SparkQA]

Tests timed out for PR 2806 at commit 09561e8 after a configured wait of 120m.

[SparkQA]

QA tests have started for PR 2806 at commit 09561e8.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2806 at commit 09561e8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class CustomPathFilter(maxModTime: Long)

[davies]

@tdas Could you help to review this? The failed tests run stable locally, I'm investigating it.

[SparkQA]

QA tests have started for PR 2806 at commit 09561e8.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2806 at commit 09561e8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class CustomPathFilter(maxModTime: Long)

[tdas]

@davies this is a significant PR. Lets talk about this PR after the 1.2 rush is over.

[tdas]

There has been significant refactoring done in the FileInputStream. Can you update the PR accordingly?

[tdas]

Also, I took a quick look at the PR. Its seems a little complicated to understand just by looking at the code, so could you write a short design doc (or update the PR description) on the high-level technique used to implement this. It does not have to be very detailed, just enough for any one understand the logic and then verify it in the code.

[tdas]

Since we are not working on this feature right now, mind closing this? We can open it again when we are want to work on it.