[SPARK-28776][ML] SparkML Writer gets hadoop conf from session state

[helenyugithub]

What changes were proposed in this pull request?

SparkML writer gets hadoop conf from session state, instead of the spark context.

Why are the changes needed?

Allow for multiple sessions in the same context that have different hadoop configurations.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Tested in pyspark.ml.tests.test_persistence.PersistenceTest test_default_read_write

[helenyugithub]

@jkbradley you seem to have reviewed previous prs to this file like #18742. Would you be able to review this pr, or suggest another person to do so?

[srowen]

Just for my reference, does this make it consistent with other similar code? just want to understand the argument why this needs to change - what problem does the current code cause?

[helenyugithub]

Hi @srowen
I believe the general convention is to use the session state's hadoop conf (see

spark/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala

Line 115 in 39577a2

val fs = FileSystem.get(spark.sessionState.newHadoopConf())

) . This also prevents correctness issues when multiple spark sessions are sharing the same context and they want to have their own hadoop conf (for example, that contains an authentication key that shouldn't be shared across users)

[srowen]

Sounds good to me. There are probably some more cases where a similar fix should be made then, but OK to address this isolated one.

[SparkQA]

Test build #4837 has finished for PR 25505 at commit 9c8f93c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[helenyugithub]

Thanks @srowen ! Now that the tests have passed, is this PR good to merge?

[srowen]

(I usually leave it open at least a day to catch any more comments)

[srowen]

Merged to master