[HUDI-8828] Test coverage of MIT partial insert and update#12583
[HUDI-8828] Test coverage of MIT partial insert and update#12583yihua merged 3 commits intoapache:masterfrom
Conversation
Davis-Zhang-Onehouse
force-pushed
the
HUDI-8828
branch
from
8670331 to
fff9de9
Compare
...-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestMergeIntoTable.scala
Show resolved
Hide resolved
| | primaryKey = 'id' | ||
| | type = '$tableType', | ||
| | primaryKey = 'id', | ||
| | precombineKey = 'ts' |
There was a problem hiding this comment.
Is precombineKey required in this test?
There was a problem hiding this comment.
not required, it is a workaround for https://issues.apache.org/jira/browse/HUDI-8835.
In general, for MIT to be able to operate independently from precombine field, it requires more work
There was a problem hiding this comment.
My point is that we should not add more config or constraint in the test that make it more specific, if not necessary. The test passes without setting precombineKey before. Could we try that? If some case fails, let's understand why.
There was a problem hiding this comment.
yes, I have tried, then I hit https://issues.apache.org/jira/browse/HUDI-8835. Precombine key is added as a workaround for this issue. Otherwise we can wait until 8835 is triaged. According to our last discussion, I'm waiting for you to take a look first. Happy to triage myself, just lmk.
There was a problem hiding this comment.
I see. Let me take a look first.
There was a problem hiding this comment.
follow ups:
- So even if I set precombine field it does not address 8835.
- To unblock this PR, I make the test a separate one and only keep test dimensions for COW as it does not hit 8835. Once that is fixed, we can uncomment the MOR dimensions
...-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestMergeIntoTable.scala
Outdated
Show resolved
Hide resolved
Davis-Zhang-Onehouse
force-pushed
the
HUDI-8828
branch
from
5912957 to
757290d
Compare
yihua
changed the title
3 participants
Change Logs
add MIT test coverage
Impact
better coverage
Risk level (write none, low medium or high below)
none
Documentation Update
none
Contributor's checklist