Getting Value Error while loading a dataset..

[valmetisrinivas]

Describe the bug

I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported

Steps to reproduce the bug

!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)

Expected behavior

I expect an iterable object as the output 'lds' to be created.

Environment info

Windows laptop with Google Colab notebook

[mariosasko]

Hi! I can't reproduce this error locally (Mac) or in Colab. What version of datasets are you using?

[valmetisrinivas]

Hi mariosasko, the datasets version is '2.8.0'.

[albertvillanova]

@valmetisrinivas you get that error because you imported datasets (and thus fsspec) before installing zstandard.

Please, restart your Colab runtime and execute the install commands before importing datasets:

!pip install datasets
!pip install zstandard

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(ds))

[valmetisrinivas]

@valmetisrinivas you get that error because you imported datasets (and thus fsspec) before installing zstandard.

Please, restart your Colab runtime and execute the install commands before importing datasets:

!pip install datasets
!pip install zstandard

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(ds))

I guess that was the problem, importing datasets before the installation of zstandard. Thank you for the feedback.