CsvExternalSort.sortInBatch drops a row for each batch

[z9security]

In the case where if (currentBlock.get() < blocksize) is false, the CSVRecord e is lost. You'll notice this as a missing row for each block.

My solution was to remove the conditional logic for processing e, and add the conditional logic only in the part where the block is processed.

Here is my updated code:
` try (CSVParser parser = new CSVParser(fbr, sortOptions.getFormat())) {
parser.spliterator().forEachRemaining(e -> {
if (e.getRecordNumber() <= sortOptions.getNumHeader()) {
header[0] = e;
} else {
tmplist.add(e);
currentBlock.addAndGet(SizeEstimator.estimatedSizeOf(e));
}
if (currentBlock.get() >= blocksize) {
try {
files.add(sortAndSave(tmplist, tmpdirectory, sortOptions, header[0]));
} catch (IOException e1) {
logger.warn(String.format("Error during the sort in batch"),e1); //$NON-NLS-1$
}
tmplist.clear();
currentBlock.getAndSet(0);
}

		});
	}

`

[z9security]

CsvExternalSort.java.txt

[lemire]

I agree that it is a bug and I agree that your fix is correct. Will merge manually and issue a release.

[lemire]

Closing.

[z9security]

Thanks!