[SPARK-32667][SQL] Script transform 'default-serde' mode should pad null value to filling column

[AngersZhuuuu]

What changes were proposed in this pull request?

Hive no serde mode when column less then output specified column, it will pad null value to it, spark should do this also.

hive> SELECT TRANSFORM(a, b)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY '|'
    >   LINES TERMINATED BY '\n'
    >   NULL DEFINED AS 'NULL'
    > USING 'cat' as (a string, b string, c string, d string)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY '|'
    >   LINES TERMINATED BY '\n'
    >   NULL DEFINED AS 'NULL'
    > FROM (
    > select 1 as a, 2 as b
    > ) tmp ;
OK
1	2	NULL	NULL
Time taken: 24.626 seconds, Fetched: 1 row(s)

Why are the changes needed?

Keep save behavior with hive data.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added UT

[HyukjinKwon]

cc @cloud-fan

[SparkQA]

Test build #127712 has finished for PR 29500 at commit 4173cfb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

The patch LGTM, but I have a question about the terminology here. AFAIK no-serde in Hive means a default serde is picked. Shall we use "default-serde mode" instead of "no-serde mode"?

[AngersZhuuuu]

The patch LGTM, but I have a question about the terminology here. AFAIK no-serde in Hive means a default serde is picked. Shall we use "default-serde mode" instead of "no-serde mode"?

Yea

[cloud-fan]

thanks, merging to master!

[cloud-fan]

please update "no-serde" in the codebase in your other TRANSFORM PRs.

[AngersZhuuuu]

please update "no-serde" in the codebase in your other TRANSFORM PRs.

Will raise a pr to handle this together