Can qsv handle encodings other than UTF-8?

[jqnatividad]

qsv is written in Rust, which requires UTF-8 encoding for its string data type.

Therefore, for qsv commands that manipulate input data as strings, this means UTF-8 encoding is required (denoted by 🔣 - the Input Symbols emoji).

However, for commands that manipulate input data as bytes, qsv is encoding-agnostic and can work with any encoding.

To check if your CSV is UTF8-encoded, you can use the validate command without a JSON schema, i.e:

qsv validate mydata.csv

and it will emit an error if its NOT UTF-8 encoded.

You can also use the input command to handle non-UTF8 encoded CSV files with its --encoding-errors option. This option has three modes:

• replace - allows "lossy-transcoding" of non-UTF8 input data, replacing non-UTF8 sequences with "�".
• skip - fields with non-UTF8 sequences are replaced with the value "<SKIPPED>"
• strict - fail immediately on any encoding error.

However, both commands will not infer the current encoding.

To get the current encoding, you'll need the file command:

file -i mydata.csv
mydata.csv: application/csv; charset=iso-8859-1

To convert it to UTF-8 encoding, you'll need the iconv command:

iconv -f iso-8859-1 -t UTF8//TRANSLIT mydata.csv -o mydata-utf8.csv

[datatraveller1]

It would be great if it was possible to add a --input-encoding option to be able to use ANSI encodings like codepage 1252. This would then internally be converted to UTF-8 by Rust (maybe using https://crates.io/crates/encoding_rs)? Then, processing and output could be done as usual with UTF-8. I don't know Rust well, but I know that with Python an input encoding can be specified when opening a file. Do you think this may also be possible with Rust by using e.g. the crate above? If so, a --input-encoding option would be a great addition.

[jqnatividad]

In an older version of qsv, I attempted to incorporate the ability to do exactly what you're requesting with the input command, but I eventually gave up on it as it introduced complexity and added dependencies to qsv.

Also, iconv is purpose-built for transcoding and is widely available, so as @ondohotola suggested, I'd just transcode to UTF-8 before piping it into qsv.

[ondohotola]

And if you need a file, it’s only one additional step.

As a general observation, if something can be achieved by a widely available tool via a pipe, at reasonable speed, I don’t think one does not have to necessarily reinvent the wheel. Especially if it increases the size to an extent that the automatic (CI?) generation of “assets” by GitHub becomes an issue. Which the Homebrew version is prone to, if I recall correctly.

I for example don’t use qsv lens much, as | csview continues to work extremely well, especially in my bash workflows. In large samples however searching and stuff works very well in qsv lens which means I could dump the csvlens app itself.

[datatraveller1]

In an older version of qsv, I attempted to incorporate the ability to do exactly what you're requesting with the input command, but I eventually gave up on it as it introduced complexity and added dependencies to qsv.

ok, I fully understand. Thank you both!

[ondohotola]

Can’t you filter the data into UTF-8 before piping it into qsv?