[#3136] fix(spark-connector): support replace column with same column name for spark connector #3461
Conversation
FANNG1
changed the title
|
@jerryshao @mchades @qqqttt123 please help to review this PR when you are free, thanks |
...ration-test/src/test/java/com/datastrato/gravitino/integration/test/spark/SparkCommonIT.java
Outdated
Show resolved
Hide resolved
...ration-test/src/test/java/com/datastrato/gravitino/integration/test/spark/SparkCommonIT.java
Outdated
Show resolved
Hide resolved
FANNG1
force-pushed
the
issue_3136
branch
2 times, most recently
from
63b4b47
to
133c7b3
Compare
| @@ -171,40 +171,7 @@ private void doUpdateColumnType( | |||
| icebergUpdateSchema.updateColumn(fieldName, (PrimitiveType) type); | |||
| } | |||
|
|
|||
| private ColumnPosition getAddColumnPosition(StructType parent, ColumnPosition columnPosition) { | |||
There was a problem hiding this comment.
Iceberg lib will handle this internally
|
@mchades @jerryshao please help to review when you are free, thanks. |
| @@ -143,7 +142,8 @@ private void doMoveColumn( | |||
| } else if (columnPosition instanceof TableChange.First) { | |||
| icebergUpdateSchema.moveFirst(DOT.join(fieldName)); | |||
| } else { | |||
| throw new NotSupportedException( | |||
| Preconditions.checkArgument( | |||
| columnPosition instanceof TableChange.Default, | |||
There was a problem hiding this comment.
Does this change mean that if the user does not specify the column position, nothing will happen?
There was a problem hiding this comment.
no, Iceberg will treat it as last position.
There was a problem hiding this comment.
But here is the end of the block, I don't see any action after the code
There was a problem hiding this comment.
Iceberg will add the column to the last position, so no need to move the column
There was a problem hiding this comment.
I noticed that this method is called in two places, one is doAddColumn and the other is doUpdateColumnPosition. If it's called by doUpdateColumnPosition, no changes will be made either. Are you sure this meets expectations?
There was a problem hiding this comment.
yes, this seems bugy, let me think about it
There was a problem hiding this comment.
@mchades , fix it and add test, please help to review again
| @@ -156,7 +160,9 @@ private void doUpdateColumnType( | |||
| icebergUpdateSchema.updateColumn(fieldName, (PrimitiveType) type); | |||
| } | |||
|
|
|||
| private ColumnPosition getAddColumnPosition(StructType parent, ColumnPosition columnPosition) { | |||
| // Iceberg doesn't support LAST | |||
There was a problem hiding this comment.
Iceberg doesn't support LAST, so what do you do in the method?
There was a problem hiding this comment.
this method transform Last Position to After, or first
|
@mchades all comments are addressed, please review again |
What changes were proposed in this pull request?
To support replace column:
same column namein valid column change logic in hive catalogWhy are the changes needed?
Fix: #3136
Does this PR introduce any user-facing change?
no
How was this patch tested?
add IT