[SPARK-40000][SQL] Update INSERTs without user-specified fields to not automatically add default values

[dtenedor]

What changes were proposed in this pull request?

Update INSERTs without user-specified fields to not automatically add default values.

For example, with the new behavior, this INSERT INTO command will fail with an error message reporting that the table has two columns but the command only inserted one:

CREATE TABLE t (a INT DEFAULT 1, b INT DEFAULT 2) USING PARQUET;
INSERT INTO t VALUES (42);

For INSERTs with user-specified fields, these commands may now specify fewer field/value pairs than the number of columns in the target table. The analyzer will assign the default value for each remaining column (either NULL, or else the explicit DEFAULT value assigned to the column from a previous command).

For example, with the new behavior, this INSERT INTO command will succeed, assigning the new row (42, 2) to the target table:

CREATE TABLE t (a INT DEFAULT 1, b INT DEFAULT 2) USING PARQUET;
INSERT INTO t (a) VALUES (42);

To implement this behavior, this PR creates the following config which is true by default:

spark.sql.defaultColumn.addMissingValuesForInsertsWithExplicitColumns

To switch back to the previous behavior of returning errors for INSERT INTO commands with fewer user-specified fields than the number of columns in the target table, switch this new config to false.

Why are the changes needed?

After looking at desired SQL semantics, it is preferred to be strict and require that the number of inserted columns exactly matches the target table to prevent against accidental mistakes.

Does this PR introduce any user-facing change?

Yes, see above.

How was this patch tested?

Updated unit test coverage.

[dtenedor]

Hi @gengliangwang this change will be helpful to let SQL engines toggle whether to return an error if INSERT commands without user-specified fields include fewer columns than present in the target table.

[AmplabJenkins]

Can one of the admins verify this patch?

[dtenedor]

Thanks for your review!

[dtenedor]

Hi @gengliangwang, thanks for your review, this is ready again :)

[gengliangwang]

Had an offline discussion with @dtenedor . Let's make it simple and strict for now: throw an exception if a table insert contains less columns than the table and there is not a insert column list.

[dtenedor]

Thanks for your review! The unit tests had to change a bit, but the behavior is simpler and more consistent now.

[dtenedor]

Hi @gengliangwang responded to comments, this is ready for another round when ready :)

[dtenedor]

Hi @gengliangwang per offline discussion, I have added the new flag spark.sql.defaultColumn.addMissingValuesForInsertsWithExplicitColumns to toggle whether commands like INSERT INTO t (a, b) VALUES ... will succeed or fail when the number of inserted columns is less than the number of columns in the target table.

[gengliangwang]

@dtenedor let's update the migration guide https://github.com/apache/spark/blob/master/docs/sql-migration-guide.md since this changes the default behavior.

Also, please update the PR description as per the latest code changes.

[gengliangwang]

@dtenedor FYI there is a conflict with the master branch now.

[dtenedor]

OK @gengliangwang all the conflicts should be resolved now. 👍

[dtenedor]

Thanks @gengliangwang for the reviews, applied suggestions and updated the PR description 👍

[gengliangwang]

Thanks, merging to master

[dongjoon-hyun]

This is reverted via 50c1635

[gengliangwang]

@dongjoon-hyun Thanks for the note!