[CARBONDATA-3790]Fix SI table consistency with main table segments #3734
Conversation
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1188/ |
|
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2905/ |
akashrn5
changed the title
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2906/ |
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1189/ |
| .carbonLoadModel | ||
| .getDatabaseName | ||
| }.${ secondaryIndexModel.secondaryIndex.indexName } SET | ||
| |SERDEPROPERTIES ('isSITableEnabled' = 'true')""".stripMargin).collect() |
There was a problem hiding this comment.
suppose to disable here ? why setting to true ?
please add test case or verify the latest PR in cluster.
There was a problem hiding this comment.
that was by mistake during code change, corrected it.
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1211/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2929/ |
|
LGTM |
Why is this PR needed?
Consider a scenario when SI loading is happening after main tale load, then when taking segment load, we got an issue and we skipped assuming that the missed segments will be loaded in next load by
SILoadEventListenerForFailedSegments, but the status of SI is still enabled. ButSILoadEventListenerForFailedSegmentswill only load to skipped segments if the status is disabled which will lead to segment mismatch between main and SI table, which ay lead query failure or data mismatch.What changes were proposed in this PR?
If it fails to take segment lock, during Si load, add to a skip list, if that is not empty, make the SI disable, so that
SILoadEventListenerForFailedSegmentswill take care to load the missing ones in next load to the main table.Does this PR introduce any user interface change?
Is any new test case added?