Support compressed csv in scan_csv

[corneliusroemer]

Problem description

It would be nice if polars could load compressed csvs out of the box, e.g. a zstd compressed csv.

I'm not sure what the best workaround is. xopen doesn't seem to work:

import polars as pl
import xopen 

#%%
with xopen.xopen("metadata_germany.tsv.zst", "rt") as f:
    pl.scan_csv(f, has_header=True, sep="\t").head(10).collect()

raises:

TypeError: argument 'path': 'TextIOWrapper' object cannot be converted to 'PyString'

Related: #3166

[corneliusroemer]

Now I'd also like zstd support for ndjson - would be great to be able to read from compressed files.

[natir]

A rust xopen alternative is niffler.

Sorry for self promoting, but I'm also need this feature, maybe niffler could help polars.

[corneliusroemer]

It could be that reading in rb mode would work, at least in the case of read_csv I got managed to get it to read from a zst compressed file with xopen, see https://stackoverflow.com/questions/76417610/how-to-read-csv-a-zstd-compressed-file-using-python-polars

The downside is that the entire uncompressed file/stream is read into memory before parsing, so this doesn't work for cases where uncompressed file is of similar size as the machine's memory

[ghuls]

You can use parquet-fromcsv in the meantime to convert compressed CSV/TSV files to parquet and use pl.scan_parquet on them:
#9283 (comment)