[CALCITE-2769] CSV Adapter does not handle - Empty and malformed csv lines, space before or after comma

[chiradip]

CSV Adapter does not handle - Empty and malformed csv lines, space before or after comma

File: org.apache.calcite.adapter.csv.CsvEnumerator.java in example folder did not handle CSV with space between delimiter i.e."," and also did not handle empty lines or malformed lines in the middle of the file or to the end. The sqlline crashes is there is an empty line with java.lang.ArrayIndexOutOfBoundsException.

Also test cases fail. Refer jira: https://issues.apache.org/jira/projects/CALCITE/issues/CALCITE-2769

[kgyrtkirk]

I don't think this line should be accepted; the space seems to belong to a field which is the quoted - it becomes cloudy for me that this field is quoted or not

[kgyrtkirk]

I don't think we should skip records if the conversion fails; I think every row should contain a full record.
A more descriptive error message and some info about where is the line in the file which is incorrect would probably help the end user to correct the mistake.
https://tools.ietf.org/html/rfc4180

[kgyrtkirk]

this would probably mean that a space value is not anymore considered a value - it will be the same as an empty string
I think rfc4180 permits multiple spaces without quotes

[asereda-gs]

Proposed solution looks very specific.

What people think about the following alternative:

1. Add default method to Source interface:

  default Stream<String> lines() {
     // TODO add close() logic
     return new BufferedReader(reader()).lines();
   }

2. Add / Replace constructor of CsvSchema to accept a list (stream) of sources (instead of parent dir as File) :

public CsvSchema(Stream<Source> sources, CsvTable.Flavor flavor) {
}

If user wants to modify / filter lines based on particular logic they can wrap existing source and override lines() method:

return lines().map(String::trim()).filter(line -> !line.isEmpty());

It also has the advantage of accepting generic sources of data like HDFS.