[SPARK-40163][SQL] feat: SparkSession.config(Map)

[seunggabi]

https://issues.apache.org/jira/browse/SPARK-40163

What changes were proposed in this pull request?

• as-is

    private fun config(builder: SparkSession.Builder): SparkSession.Builder {
        val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)

        var b = builder
        map.keys.forEach {
            val k = it
            val v = map[k]

            b = when (v) {
                is Long -> b.config(k, v)
                is String -> b.config(k, v)
                is Double -> b.config(k, v)
                is Boolean -> b.config(k, v)
                else -> b
            }
        }

        return b
    }
}

• to-be

    private fun config(builder: SparkSession.Builder): SparkSession.Builder {
        val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)

        return b.config(map)
    }
}

Why are the changes needed?

• string, boolean, long, double -> toString
• so this is simple code!

Does this PR introduce any user-facing change?

How was this patch tested?

• added test code

[AmplabJenkins]

Can one of the admins verify this patch?

[seunggabi]

I will add .conf test code today.

[srowen]

This also lacks a JIRA, but, don't bother until we decide whether this can move forward even

[seunggabi]

This also lacks a JIRA, but, don't bother until we decide whether this can move forward even

yes I am ready for create JIRA.
If you say to me create JIRA, I create JIRA.

[srowen]

Same question as above - isn't this working by just binding to deprecated methods? I don't think this is worth deprecating all the calls apps make now.

[seunggabi]

Same question as above - isn't this working by just binding to deprecated methods? I don't think this is worth deprecating all the calls apps make now.

right, so I removed @deprecated. I add just new conf method. (vs config).
maybe I don't understand your last comment.
would you show me to-be ?

[srowen]

What would the new method bind to now? wouldn't it just bind to the existing methods in almost all cases? so what is the value of the new one?

[seunggabi]

@srowen
yes. you're right.
as-is works every case.

but, I suggested for flexible code.
worth is flexible code? <- it may be my own idea.

if this pr is closed, I'm OK!
please make a decision (destiny of this PR)

---

as-is

            b = when (v) {
                is Long -> b.config(k, v)
                is String -> b.config(k, v)
                is Double -> b.config(k, v)
                is Boolean -> b.config(k, v)
                else -> b
            }

to-be

            b = b.conf(it, map[it])

[srowen]

I see your point, OK. Hm, maybe a more useful overload is a "put all" method that takes a Map?

[seunggabi]

@srowen
Thx, I will make map conf method this week.

Please wait for me

[seunggabi]

@srowen
I am done.
please check.

have a nice day.

[seunggabi]

choose

1. type check -> config(String, type) method
2. foreach -> options += key -> value

---

I chose 2.
because it is simple code.

[srowen]

Last thing, I'd go ahead and make a JIRA for this. It's minor but non trivial

[seunggabi]

thx @srowen
have a nice weekend.!

[srowen]

Please edit the title to link to the JIRA

[seunggabi]

@srowen
You mean,
A: I make JIRA, and change title and attach link.(I dont have jira project auth)
B: you give to JIRA, and change title and attach link.

A or B?

I maybe think B.
I will change title and attach link.
B right?

[srowen]

You can file a JIRA, please

[seunggabi]

@srowen
thx for guide, I changed title & attach link!

[srowen]

It goes in the title, I added it. (https://spark.apache.org/contributing.html) LGTM

[seunggabi]

@srowen
I forget, to click title edit update buttion.
I edited code. (JIRA -> test code comment). caee621

Thank you for spending a lot of time for me as a beginner.
I learned a lot, and I think I can become a better contributor in the future. 🚀

[srowen]

Merged to master