[SPARK-28333][SQL] NULLS FIRST for DESC and NULLS LAST for ASC

[shivusondur]

What changes were proposed in this pull request?

changed the default null ordering for ACS and DESC, and updated the corresponding tests

How was this patch tested?

Ran sql/catalyst module UT and updated the test case

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

I think this is a breaking change and workaround looks pretty easy. Can you check what other DBMSs do? If it's consistent with some other DBMSs, I think we shouldn't fix it and resolve the ticket as Won't Do

[shivusondur]

@HyukjinKwon
From the below link, spark is consistent with mysql, sqlserver
But inconsitent with DB2, Oracle, Postgresql
https://docs.mendix.com/refguide/null-ordering-behavior#3-overview-of-default-nulls-sort-order

[HyukjinKwon]

I think the current behaviour already makes sense. I wouldn't fix it. cc @dongjoon-hyun, @wangyum, @gatorsmile WDYT?

[dongjoon-hyun]

+1 for @HyukjinKwon 's comment.

[dongjoon-hyun]

This is introduced at 2.1.0 via #14842 . And, we are followinging Hive's behavior. We cannot change this default behavior due to the reason @HyukjinKwon 's pointed out .

@shivusondur . We can make a configuration for this instead if needed.

Could you make a decision, @gatorsmile , whether closing this issue as Won't Do or redirecting this issue to the conf, please?

[shivusondur]

This is introduced at 2.1.0 via #14842 . And, we are followinging Hive's behavior. We cannot change this default behavior due to the reason @HyukjinKwon 's pointed out .

@shivusondur . We can make a configuration for this instead if needed.

Could you make a decision, @gatorsmile , whether closing this issue as Won't Do or redirecting this issue to the conf, please?

@gatorsmile
Let me know your comment for the same.

[dongjoon-hyun]

Hi, @shivusondur . This is a tough call. Although Spark 3.0.0 is a good chance for this, let's close this PR and the issue as Won't Do for now since there is no positive comments.

[HyukjinKwon]

Yup, let's just close - I don't particularly think this is worth, and no positive comments from others as pointed out.

[gatorsmile]

@shivusondur Can you submit a PR to document our current behavior?

[gatorsmile]

Create a subtask in https://issues.apache.org/jira/browse/SPARK-28588?