read_ndjson read_json: Add encoding parameter

[JulianCologne]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

data.json (encoding=UTF-8-BOM)

{"col1": "a", "col2": 1}
{"col1": "b", "col2": 2}

import polars as pl

pl.read_ndjson("data.json")

Log output

RuntimeError: BindingsError: "InternalError(TapeError) at character 0 ('ï')"

Issue description

read_ndjson errors with different encodings and there is no option to speficy encoding

Many popular etl tools like azure DataFactory write json files with UTF-8-BOM encoding so this is happens quite a lot.

IMO every time you interact with a file the encoding is important to specify. Pandas also has this parameter.

Expected behavior

should allow encoding parameter and read data into dataframe

Installed versions

--------Version info---------
Polars:               0.20.16
Index type:           UInt32
Platform:             Windows-10-10.0.19045-SP0
Python:               3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          <not installed>
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               <not installed>
gevent:               <not installed>
hvplot:               0.9.1
matplotlib:           <not installed>
numpy:                1.26.3
openpyxl:             <not installed>
pandas:               2.2.0
pyarrow:              14.0.0
pydantic:             <not installed>
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           <not installed>
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>

[ritchie46]

We will not support different encodings than ascii or utf8, but we should ignore the BOM, if we encounter it.

[ritchie46]

p-high because it is low effort and fixes unreadable files.