[SPARK-4826] Fix generation of temp file names in WAL tests

[JoshRosen]

This PR should fix SPARK-4826, an issue where a bug in how we generate temp. file names was causing spurious test failures in the write ahead log suites.

Closes #3695.
Closes #3701.

[SparkQA]

Test build #24464 has started for PR 3704 at commit 9362919.

• This patch merges cleanly.

[JoshRosen]

This might look race prone (deleting a file and hoping that someone else won't come along and write it in the meantime), but it should be safe because:

1. Different Jenkins builds will have different temp directories (dir).
2. Within a JVM, multiple calls to createTempFile will never return the same pathname (see Javadoc).

[harishreedharan]

This looks good to me, though the approach does not make it obvious why this approach was chosen (of course you can figure out in this context, but imagine reading this code a year later).

I think the other two are slightly simpler approaches by ensuring unique names on file creation.

[vanzin]

I buy your explanation, given the javadoc, so this LGTM. But I think a cleaner approach that doesn't require reasoning like that to convince people would be to just use a different temp dir per test (i.e. use BeforeAndAfter and create the dir in the before block).

[JoshRosen]

The root issue is that this code is trying to return a unique file system path that meets two conditions:

• The path does not correspond to an existing file.
• Files / directories created at that path will be cleaned up by the test suite.

I think this was the intent expressed by the original Random.nextString(10) code. One approach would be to have the directory be random and per-test and to just use a fixed filename within that directory. This precludes parallel execution of the tests within a single copy of the write-ahead log suites, but I don't think we're pursuing that type of parallelism right now so I don't think that will be a huge deal.

[vanzin]

This precludes parallel execution of the tests within a single copy of the write-ahead log suites

I assume that frameworks handle that automatically (e.g. by having multiple instances of the test class), otherwise you could never parallelize tests that use "before" initializers. I'm pretty sure that works as intented at least with JUnit, but not super familiar with scalatest internals.

[JoshRosen]

Alright, I've simplified things to move the temp. dir creation to beforeEach instead of beforeAll and to use a fixed filename within that directory.

[harishreedharan]

+1. The latest changes look good.

[vanzin]

This doesn't work if the dir is not empty. You could use Utils.createTempDir() and, optionally, Utils.deleteRecursively() (since createTempDir already takes care of that for you).

[JoshRosen]

Good catch; I've fixed this.

[SparkQA]

Test build #24466 has started for PR 3704 at commit a693ddb.

• This patch merges cleanly.

[SparkQA]

Test build #24467 has started for PR 3704 at commit f2307f5.

• This patch merges cleanly.

[SparkQA]

Test build #24464 has finished for PR 3704 at commit 9362919.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24464/
Test PASSed.

[SparkQA]

Test build #24466 has finished for PR 3704 at commit a693ddb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class WriteAheadLogBackedBlockRDDSuite extends FunSuite with BeforeAndAfterAll with BeforeAndAfterEach

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24466/
Test PASSed.

[SparkQA]

Test build #24467 has finished for PR 3704 at commit f2307f5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class WriteAheadLogBackedBlockRDDSuite extends FunSuite with BeforeAndAfterAll with BeforeAndAfterEach

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24467/
Test PASSed.

[srowen]

Looks like a robust approach to me.

[JoshRosen]

Alright, I'm going to merge this. Thanks for looking this over!