[ch140705] Sanitize column names on overwrite import

[Shylpx]

Resources

• Clubhouse story

Context

This corner case appears when a table is imported using the overwrite collision strategy, and the columns required sanitization. For example, when the column name starts with a number (1234_column), an underscore will be added (_1234_column). First import is successfully, but the second one compares the schemas. Since the sanitization is executed on table creation (almost as last step), the code is going to compare 1234_column with _1234_column that are indeed different. So an IncompatibleSchemas exception will be raised.

The idea is to apply the sanitization to the column names of the original schemas in order to compare the existing table's schema with the real names that the new table would have.

Changes

• Sanitize column names on comparison when the collision strategy is overwrite.
• Add column sanitization to the result table on overwrite_register

[shortcut-integration]

This pull request has been linked to Clubhouse Story #140705: [advocacy-institute] IncompatibleSchemas error with collision_strategy overwrite and column names normalization.

[amiedes]

👍