Related to #180, but different failure so new ticket:
In the colab getting_started tutorial, when creating a dataset using locally downloaded data, I'm getting an error
# Load the dataset descriptions from local copy of the data
sdss = load_dataset_builder("data/MultimodalUniverse/v1/sdss", trust_remote_code=True)
hsc = load_dataset_builder("data/MultimodalUniverse/v1/hsc", trust_remote_code=True)
...
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
[/tmp/ipython-input-1984604797.py](https://pr3j46tlglf-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-external_20251113-060048_RC00_831773243#) in <cell line: 0>()
1 # Load the dataset descriptions from local copy of the data
----> 2 sdss = load_dataset_builder("data/MultimodalUniverse/v1/sdss", trust_remote_code=True)
3 hsc = load_dataset_builder("data/MultimodalUniverse/v1/hsc", trust_remote_code=True)
1 frames
[/usr/local/lib/python3.12/dist-packages/datasets/load.py](https://pr3j46tlglf-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-external_20251113-060048_RC00_831773243#) in dataset_module_factory(path, revision, download_config, download_mode, data_dir, data_files, cache_dir, **download_kwargs)
915 raise RuntimeError(f"Dataset scripts are no longer supported, but found {filename}")
916 elif os.path.isfile(combined_path):
--> 917 raise RuntimeError(f"Dataset scripts are no longer supported, but found {filename}")
918 elif os.path.isdir(path):
919 return LocalDatasetModuleFactory(
RuntimeError: Dataset scripts are no longer supported, but found sdss.py
This is with HuggingFace datasets version 4.0.0, which was the default installed at the start of the tutorial notebook.
huggingface/datasets#7592 and has some discussion of the security risks and why it was removed.
It appears that this works with datasets==3.6.0, but that isn't compatible with numpy/2.x which is also the default for the tutorial.
Workaround for the tutorial
In the end, it worked to change the installation lines to
# Installing Hugging Face datasets
!pip install datasets==3.6.0 --quiet
# downgrade numpy to be compatible with datasets 3.6.0
!pip install "numpy<2" --quiet
and to work around #180, get sdss.py with
### !wget -r -np -nH --cut-dirs=1 -R "index.html*" -q https://users.flatironinstitute.org/~polymathic/data/MultimodalUniverse/v1/sdss/sdss.py
!wget -r -np -nH --cut-dirs=1 -R "index.html*" -q https://raw.githubusercontent.com/MultimodalUniverse/MultimodalUniverse/refs/heads/main/scripts/sdss/sdss.py
!mv MultimodalUniverse/refs/heads/main/scripts/sdss/sdss.py data/MultimodalUniverse/v1/sdss/
Longer term
It is unclear to me whether datasets/4.x is compatible with a local copy of MMU v1. Guidance there would be appreciated, especially given that the version of datasets that does work (3.6.0) isn't compatible with numpy 2.x.
Related to #180, but different failure so new ticket:
In the colab getting_started tutorial, when creating a dataset using locally downloaded data, I'm getting an error
This is with HuggingFace datasets version 4.0.0, which was the default installed at the start of the tutorial notebook.
huggingface/datasets#7592 and has some discussion of the security risks and why it was removed.
It appears that this works with datasets==3.6.0, but that isn't compatible with numpy/2.x which is also the default for the tutorial.
Workaround for the tutorial
In the end, it worked to change the installation lines to
and to work around #180, get sdss.py with
Longer term
It is unclear to me whether datasets/4.x is compatible with a local copy of MMU v1. Guidance there would be appreciated, especially given that the version of datasets that does work (3.6.0) isn't compatible with numpy 2.x.