[SUPPORT] MERGE INTO with UPDATE */ INESRT * - new incoming columns dropped, automatic schema evolution feature

[kazdy]

Describe the problem you faced

I've tried using MERGE INTO with UPDATE * and INSERT * statement with full schema evolution enabled.
I've noticed that during insert new columns from incoming batch (that do not exist in target table yet) are dropped and target schema is applied. No warnings nor failed writes.

Therefore can we as users automatically evolve schema on MERGE INTO operations?
I guess this should only be supported when we use update set * and insert * in merge operation.

Expected behavior

When incoming data is missing columns that already declared in target table these should be injected with default/null values.
When incoming data has new columns that are not yet declared in the target table, these should be added to the target table.
Case when incoming data has both missing columns and new columns, missing coluns should be injected with null/ default values, new columns should be added to the target table.

New columns should be reflected in metastore table schema.

Would be great to support complex types, and nested schemas.

Thread from dev mailing list as a reference:
https://lists.apache.org/thread/kr59hh7yqr2c1y33kzfv3n97h6ydbz9b

Environment Description

• Hudi version : 0.11

• Spark version : 3.2.0-amzn

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :

• Running on Docker? (yes/no) :

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[codope]

@kazdy Isn't it same as HUDI-4276?
cc @xiarixiaoyao

[kazdy]

@codope it's related because it also calls for automatic schema evolution with similar logic.
With MERGE INTO new fields are being dropped, so once HUDI-4276 is done it can be used in MERGE INTO I guess?

Looking at MergeIntoHoodieTableCommand I think the target table schema is always applied on write, so whatever new column was added (in source table but not in target table), it's just being dropped from source DF.
Because of this line:
HoodieWriteConfig.WRITE_SCHEMA.key -> getTableSchema.toString
where getTableSchema returns target table schema.

[nsivabalan]

@xiarixiaoyao : Can you look into this issue. Looks like its related to schema evolution.

[xiarixiaoyao]

@kazdy no full scheam evolution cannot support automatic evolution, i am working for the feature , of course complex types, and nested schemas will supported.

[kazdy]

hi @xiarixiaoyao thanks for response :)
So full schema evolution is a separate feature to automatic evolution, but automatic evolution itself can be supported as a separate feature? Do I understand you correctly?

[nsivabalan]

yes, thats my understanding too.

[nsivabalan]

@xiarixiaoyao : if we have a tracking jira, we can close the github issue.

[kazdy]

@nsivabalan @xiarixiaoyao I created this jira - HUDI-4872, so I'm closing the ticket.