[Bug] CSV PERMISSIVE mode drops every field after the first malformed one

[LuciferYang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master, 2788fe596 (2.1-SNAPSHOT). release-1.4 carries it as well; 1.2 and 1.3 do not.

Compute Engine

Flink and Spark, on any table read through Paimon's own CSV reader ('file.format' = 'csv'). For a format table that is the default path, since format-table.implementation defaults to paimon.

Minimal reproduce step

Read a three-column CSV whose first field does not parse as INT, with the default csv.mode:

CREATE TABLE t (a INT, b STRING, c DOUBLE) WITH (
    'type' = 'format-table',
    'file.format' = 'csv'
);
-- one line in the table directory: x,Alice,1.5
SELECT * FROM t;

The row comes back as (null, null, null), although Alice and 1.5 are both well formed.

CsvParser.parse walks the projected fields in a for loop, and the PERMISSIVE branch breaks out of that loop:

for (int i = 0; i < projectMapping.length; i++) {
    ...
    if (parseResult != null && parseResult.getLeft()) {
        row.setField(i, parseResult.getValue());
    } else if (mode == PERMISSIVE
            && (parseResult == null || !parseResult.getLeft() || exception != null)) {
        break;
    }

Every field after the malformed one keeps the null that new GenericRow(...) started with.

What doesn't meet your expectations?

csv.mode defaults to PERMISSIVE, and the option is documented as "sets malformed fields to null", which means the field and not the rest of the row. Nothing throws and nothing is logged, so the query returns wrong data quietly. A row whose malformed field happens to be the last projected column reads correctly, which is why the current tests pass.

Anything else?

This regressed in #6856 (184b95aae). The mode used to be handled by a switch:

switch (mode) {
    case PERMISSIVE:
        break;

There the break left the switch and the loop went on to the next field. That commit turned the switch into an if-else chain, where the same break leaves the loop instead.

Are you willing to submit a PR?

• I'm willing to submit a PR!