readCSV deprecation, how do we specify charset now?

[corlaez]

I couldn't find a way to specify charset at all, my file is utf-8. The deprecated method has a charset parameter but the new method doesn't offer any way to change the charset, not even through parser options.

am i missing something? why not keep the charset param?

[Jolanrensen]

Hi! I've moved this issue to DataFrame, as that holds the readCsv() function.

The new readCsv() implementation is based on the blazing fast Deephaven CSV library. This library reads files at the byte level, so it handles charsets itself. As far as I know, Deephaven and its Double parser are based on UTF-8, so you should be good to go :)

If you need some other charset, I would recommend manually reading the file to String first and then calling DataFrame.readCsvStr().

[corlaez]

From what I see the new method is not using UTF-8

I have a UTF encoded csv file and I see the following:
deprecated:

new:

(Sure, I shouldn't be using the weird " for a json, but it serves to illustrate the wrong encoding)

[Jolanrensen]

@corlaez could you send a sample of your data? Since it's the " character breaking, it might be because of the quote = parameter, which is " by default :)

[corlaez]

Id,InsertedAt,RefTimestamp,mutableData
1,2000-01-01 01:01:01,2000-01-01 01:01:01,"{“mg”: 15, “socialOnly”: false}"

\u201c
and
\u201d

[corlaez]

"ȭ" in mutable data displays as

old:

new:

[Jolanrensen]

huh, strange, it doesn't reproduce on my machine. Which OS are you using?

Even "ȭ"

[corlaez]

readCsvStr is using utf-8 but readCsv isn't utf 8 here

Windows 10.

1.0.0-Beta3n

[Jolanrensen]

@zaleslaw could you try to reproduce this on Windows when you have the time? :)

I also wonder how this works with raw https://github.com/deephaven/deephaven-csv

@corlaez Could you try running

import io.deephaven.csv.CsvSpecs
import io.deephaven.csv.reading.CsvReader
import io.deephaven.csv.sinks.SinkFactory

CsvReader.read(
    CsvSpecs.builder().build(),
    File("path/to/your.csv").inputStream(),
    SinkFactory.arrays()
).columns()[5]
    .let { it.data() as Array<String> }
    .first()
    .let(::println)

[corlaez]

Using CsvReader, the utf8 encoding is not selected
{“mg�: 15, “socialOnly�: false}

[corlaez]

So, my default encoding is not utf8 and that library could be relying in default encoding (at least for readCsv, because readCsvStr reads well for me, which makes sense because that one isn't relying on default file encoding).

I think it is important to have parameters to control this and not be bound on the default file encoding. Infamously, default file encoding can't change at runtime, so that means that we would be forever tied to opening files in a single encoding unless the process is launched again with a different default file encoding.

Beyond that, I can't seem to make Kotlin Notebook (which is where I am running my dataframe) run with default encoding UTF-8, but even if I could I find this to be too restrictive.

---

If my theory is correct, I would be tempted to rephrase this ticket as a bug. Deprecating the ability to open and read different file encodings in a single process seems unjustifiable (especially if that is leading to a removal).

[Jolanrensen]

@corlaez Yes, I agree this should be configurable somehow, however, as you showed after running io.deephaven.csv.reading.CsvReader.read(), this is not a limit of DataFrame itself, but of Deephaven CSV. We moved away from Apache Commons CSV some time ago because it was very slow and memory-intensive. This allows us to finally be able to read 1M+ rows of CSV easily. One downside is that it's less configurable.

I will create an issue on their GitHub to ask them about adding the ability to change encodings programmatically. That way we can add it too :)

deephaven/deephaven-csv#278