Core: Ignore split offsets array when split offset is past file length

[amogh-jahagirdar]

Follow up from a miss in the fix done in #8860. There's an additional splitOffsetArray method which gets used here

iceberg/core/src/main/java/org/apache/iceberg/BaseContentScanTask.java

Line 134 in aa891ac

return ((BaseFile<?>) file).splitOffsetArray();

. We would want to surface null for this array in case we know the split offsets are corrupted and invalid (stemming from an issue in 1.4.0). The split offsets are invalid when the final offset is beyond the end of the file.

Alternatively, we can undo the optimization done here: #8336 (comment) and then the original fix in #8860 would be sufficient, and be simpler. Since the optimization seemed impactful and there was community interest in it, this fix retains it and currently just fixes both cases explicitly.

[rdblue]

@amogh-jahagirdar, maybe this time we should create a test to validate FileScanTask.split when there are bad split offsets? I think that would have caught this in the last PR.

[amogh-jahagirdar]

@amogh-jahagirdar, maybe this time we should create a test to validate FileScanTask.split when there are bad split offsets? I think that would have caught this in the last PR.

Done, added a new test which will exercise both splitOffsetArray and splitOffsets. This will also validate the number of tasks per file