`scan_csv` limited to fewer encoding options than `read_csv`

[mcrumiller]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Issue description

read_csv can read utf-16 but scan_csv cannot. Presumably this is true for more encodings as well.

I understand that some optimizations cannot be performed when using scan_csv with non-utf8 encodings, but the functionality should still exist.

from io import StringIO
import polars as pl

with open("tmpfile.csv", "w", encoding="utf-16") as f:
    f.write("a,b,c\n1,2,3\n")

# ok
pl.read_csv("tmpfile.csv", encoding="utf-16")

# ValueError: csv `encoding` must be one of {'utf8', 'utf8-lossy'}, got utf-16
pl.scan_csv("tmpfile.csv", encoding="utf-16").collect()

Installed versions

Details

--------Version info---------
Polars:               0.20.31
Index type:           UInt32
Platform:             Windows-10-10.0.19045-SP0
Python:               3.11.7 (tags/v3.11.7:fa7a6f2, Dec  4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           0.3.2
deltalake:            <not installed>
fastexcel:            0.10.4
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.8.3
nest_asyncio:         <not installed>
numpy:                1.26.4
openpyxl:             3.1.2
pandas:               2.1.4
pyarrow:              16.0.0
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           2.0.23
torch:                <not installed>
xlsx2csv:             0.8.2
xlsxwriter:           3.1.9

[ritchie46]

We don't support this. This might be a side effect of something else.

[mcrumiller]

When you say don't support, do you mean neither read_csv nor scan_csv should allow for other encodings?

[ritchie46]

Our reader only supports utf8, so there is something on the python side that decodes the buffer and gives us a utf8 buffer. Might be handled on purpose like this in Polars' python side, but this is not something we can do in the scan operations.

[mcrumiller]

I see. I suppose the API is explicit enough that I can close this.