CSV with encoding=iso-8859-1 fails

[indigoviolet]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

Even though the documentation suggests that encodings other than 'utf8' or 'utf8-lossy' are supported, this fails.

Reproducible example

echo -e "Column 1,Column 2\nVälüë 1,Vàlüë 2\nÄnöthër vãlüë,Ånd anøthér vãlüë" | iconv -t ISO-8859-1 > output.csv

gzip output.csv -c > output_gz.csv.gz

In [8]: pl.read_csv('output.csv', encoding='iso-8859-1')
Polars found a filename. Ensure you pass a path to the file instead of a python file object when possible for best performance.
---------------------------------------------------------------------------
ComputeError                              Traceback (most recent call last)
Cell In[8], line 1
----> 1 pl.read_csv('output.csv', encoding='iso-8859-1')
...

ComputeError: invalid utf-8 sequence in csv

In [9]: pl.read_csv('output_gz.csv.gz', encoding='iso-8859-1')
Polars found a filename. Ensure you pass a path to the file instead of a python file object when possible for best performance.
Out[9]: 
shape: (0, 1)
┌──────────────────────┐
│ s��)��S0�q�0���y
{… │
│ ---                  │
│ str                  │
╞══════════════════════╡
└──────────────────────┘

### Expected behavior

succeeds with `use_pyarrow`

```python
In [10]: pl.read_csv('output.csv', encoding='iso-8859-1', use_pyarrow=True)
Out[10]: 
shape: (2, 2)
┌───────────────┬───────────────────┐
│ Column 1      ┆ Column 2          │
│ ---           ┆ ---               │
│ str           ┆ str               │
╞═══════════════╪═══════════════════╡
│ Välüë 1       ┆ Vàlüë 2           │
│ Änöthër vãlüë ┆ Ånd anøthér vãlüë │
└───────────────┴───────────────────┘

In [11]: pl.read_csv('output_gz.csv.gz', encoding='iso-8859-1', use_pyarrow=True)
Out[11]: 
shape: (2, 2)
┌───────────────┬───────────────────┐
│ Column 1      ┆ Column 2          │
│ ---           ┆ ---               │
│ str           ┆ str               │
╞═══════════════╪═══════════════════╡
│ Välüë 1       ┆ Vàlüë 2           │
│ Änöthër vãlüë ┆ Ånd anøthér vãlüë │
└───────────────┴───────────────────┘

Installed versions

Details

---Version info---
Polars: 0.16.14
Index type: UInt32
Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
Python: 3.9.16 | packaged by conda-forge | (main, Feb  1 2023, 21:39:03) 
[GCC 11.3.0]
---Optional dependencies---
numpy: 1.23.4
pandas: 1.5.3
pyarrow: 11.0.0
connectorx: <not installed>
deltalake: <not installed>
fsspec: 2023.1.0
matplotlib: 3.7.0
xlsx2csv: <not installed>
xlsxwriter: <not installed>

[ritchie46]

You select an encoding we do not support, as you said yourself. This is not a bug, but a feature request IMO.

[alexander-beedie]

We should probably further-clarify the docs... 🤔

[indigoviolet]

I wasn't trying to find fault; the docs say this:

"Lossy means that invalid utf8 values are replaced with � characters. When using other encodings than utf8 or utf8-lossy, the input is first decoded im memory with python. Defaults to utf8."

[ghuls]

It is supported for non-gzip compressed files when given as a bytes or BytesIO object or Path (but not str) when fsspec is installed:

In [105]: from pathlib import Path

In [106]: pl.read_csv(Path("output.csv"), encoding='iso-8859-1')
Out[106]: 
shape: (2, 2)
┌───────────────┬───────────────────┐
│ Column 1      ┆ Column 2          │
│ ---           ┆ ---               │
│ str           ┆ str               │
╞═══════════════╪═══════════════════╡
│ Välüë 1       ┆ Vàlüë 2           │
│ Änöthër vãlüë ┆ Ånd anøthér vãlüë │
└───────────────┴───────────────────┘

If fsspec is not installed it works with a str:

In [6]: pl.read_csv(Path("output.csv"), encoding='iso-8859-1')
Out[6]: 
shape: (2, 2)
┌───────────────┬───────────────────┐
│ Column 1      ┆ Column 2          │
│ ---           ┆ ---               │
│ str           ┆ str               │
╞═══════════════╪═══════════════════╡
│ Välüë 1       ┆ Vàlüë 2           │
│ Änöthër vãlüë ┆ Ånd anøthér vãlüë │
└───────────────┴───────────────────┘

In [7]: pl.read_csv("output.csv", encoding='iso-8859-1')
Out[7]: 
shape: (2, 2)
┌───────────────┬───────────────────┐
│ Column 1      ┆ Column 2          │
│ ---           ┆ ---               │
│ str           ┆ str               │
╞═══════════════╪═══════════════════╡
│ Välüë 1       ┆ Vàlüë 2           │
│ Änöthër vãlüë ┆ Ånd anøthér vãlüë │
└───────────────┴───────────────────┘

[romiof]

It is supported for non-gzip compressed files when given as a bytes or BytesIO object or Path (but not str) when fsspec is installed:

I've tried this week, to read some csv's which are encoded with cp1252 and can't make it work.

But now testing with a Path object, it just worked. Good to know!

--

Is there any way to convert, inside the method, a str call to a Path call ?
For making the method to work the same way, having fsspec instaled or not.

[ghuls]

It can be fixed so it works also with a plain str and fsspec installed.

[indigoviolet]

Should this be converted back to a bug if this is expected to work (per documentation) and has unexpected interactions with fsspec?

[alejohz]

I am encountering the same issue with the same encoding. Using Path fixes it locally, but when i am reading from S3, the encoding doesn't work.

The same file with pl.read_csv(Path(local_path), encoding="iso-8859-1") works, but pl.read_csv(s3_path, encoding="iso-8859-1") does not. Any ideas on how to fix it?