[SPARK-36965][PYTHON] Extend python test runner by logging out the temp output files

[attilapiros]

What changes were proposed in this pull request?

Extending the python test runner by logging out the temp output files.

Why are the changes needed?

I was running a python test which was extremely slow and I was surprised the unit-tests.log has not been even created. Looked into the code and as I got the tests can be executed in parallel and each one has its own temporary output file which is only added to the unit-tests.log when a test is finished with a failure (after acquiring a lock to avoid parallel write on unit-tests.log).

To avoid such a confusion it would make sense to log out the path of those temporary output files this way when a test got stuck we can peek into its log file.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I was running the python tests:

./python/run-tests
Running PySpark tests. Output is in /Users/attilazsoltpiros/git/attilapiros/spark/python/unit-tests.log
Will test against the following Python executables: ['/usr/local/bin/python3']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml', 'pyspark-mllib', 'pyspark-pandas', 'pyspark-pandas-slow', 'pyspark-resource', 'pyspark-sql', 'pyspark-streaming']
/usr/local/bin/python3 python_implementation is CPython
/usr/local/bin/python3 version is: Python 3.9.7
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_feature (temp output: /tmp/usr_local_bin_python3__pyspark.ml.tests.test_feature__yc5_5mjk.log)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_algorithms (temp output: /tmp/usr_local_bin_python3__pyspark.ml.tests.test_algorithms__icc6xxai.log)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_base (temp output: /tmp/usr_local_bin_python3__pyspark.ml.tests.test_base__4m6xyiv5.log)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_evaluation (temp output: /tmp/usr_local_bin_python3__pyspark.ml.tests.test_evaluation__fkzjlfmm.log)
Finished test(/usr/local/bin/python3): pyspark.ml.tests.test_base (16s)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_image (temp output: /tmp/usr_local_bin_python3__pyspark.ml.tests.test_image__iuckk_c0.log)
Finished test(/usr/local/bin/python3): pyspark.ml.tests.test_evaluation (20s)
Starting test(/usr/local/bin/python3): pyspark.ml.tests.test_linalg (temp output: /tmp/usr_local_bin_python3__pyspark.ml.tests.test_linalg__3tncana4.log)
...

[SparkQA]

Test build #144052 has finished for PR 34233 at commit 017844f.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48529/

[SparkQA]

Test build #144053 has finished for PR 34233 at commit b8bf885.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48529/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48530/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48530/

[attilapiros]

cc @HyukjinKwon

[HyukjinKwon]

Looks fine

[SparkQA]

Test build #144117 has finished for PR 34233 at commit 0bf05ea.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48594/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48594/

[JoshRosen]

LGTM