[SPARK-46370][SQL] Fix bug when querying from table after changing column defaults

[dtenedor]

What changes were proposed in this pull request?

This PR fixes a bug when querying from table after changing defaults:

drop table if exists t;
create table t(i int, s string default 'def') using parquet;
insert into t select 1, default;
alter table t alter column s drop default;
insert into t select 2, default;
select * from t;  -- Removing this line changes the following results!
alter table t alter column s set default 'mno';
insert into t select 3, default;
select * from t;

The bug is related to the relation cache, and the fix involves adding a manual refresh to the cache to make sure we use the right table schema.

Why are the changes needed?

This PR fixes a correctness bug.

Does this PR introduce any user-facing change?

Yes, see above.

How was this patch tested?

This PR adds test coverage.

Was this patch authored or co-authored using generative AI tooling?

No.

[dtenedor]

cc @cloud-fan

[cloud-fan]

let's add some code comments to explain the reason: this command may change column default values and we need to refresh the table relation cache so that DML commands can resolve default values correctly.

[dtenedor]

This is done.

[beliefer]

Shall we call catalog.refreshTable(tableName) after check the default values changed?

[dtenedor]

Hi @beliefer thanks for your comment, I think we have to check it before calling catalog.getTableRawMetadata so that method performs a catalog lookup as expected rather than reading from the cache (which may be stale). There's only one call to catalog.getTableRawMetadata in this function so we should be OK here, but thanks for keeping this question in mind :)

[cloud-fan]

Can you retrigger the test? The failed test seems unrelated.

[beliefer]

LGTM except one comment.

[beliefer]

Shall we call catalog.refreshTable(tableName) after check the default values changed?

[dtenedor]

Can you retrigger the test? The failed test seems unrelated.

@cloud-fan I started this, it is in progress now:

[dtenedor]

@cloud-fan all tests are passing now:

[cloud-fan]

thanks, merging to master/3.5!