ValueError: The provided None isn't supported. #67
Description
Script:
import os
from litdata import optimize
from utils import compress
from typing import Tuple, Dict
if __name__ == "__main__":
# 1. List the files and associate the images and their masks
# split = "train"
# split = "val"
split = "test"
input_dir = f"/teamspace/studios/this_studio/data/datasetv10/split_all_bands/{split}"
output_dir = f"/teamspace/studios/this_studio/data/datasetv10/split_all_bands_stream_dict/{split}"
filepath_pairs = dict()
for f in sorted(os.listdir(input_dir)):
name = f.split('.')[0].replace("_merged", "")
if name not in filepath_pairs:
filepath_pairs[name] = [name]
filepath_pairs[name].append(os.path.join(input_dir, f))
assert all(len(v) == 3 for v in filepath_pairs.values())
# 3. Define the inputs
inputs = list([v for v in filepath_pairs.values()])
len(inputs)
# 4. Run the optimization
optimize(
fn=compress,
inputs=inputs,
output_dir=output_dir,
chunk_bytes="128MB",
num_workers=2,
)Error:
Storing the files under /teamspace/studios/this_studio/data/datasetv10/split_all_bands_stream_dict/test
Setup started with fast_dev_run=False.
Worker 0 gets 1201.2 MB (88 files)
Worker 1 gets 1201.2 MB (88 files)
Setup finished in 0.138 seconds. Found 176 items to process.
Starting 2 workers with 176 items.
Workers are ready ! Starting data processing...
Process Process-1:1: | 0/176 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litdata/processing/data_processor.py", line 163, in _download_data_target
raise ValueError(f"The provided {input_dir.url} isn't supported.")
ValueError: The provided None isn't supported.
Process Process-1:2:
Traceback (most recent call last):
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litdata/processing/data_processor.py", line 163, in _download_data_target
raise ValueError(f"The provided {input_dir.url} isn't supported.")
ValueError: The provided None isn't supported.
Process Process-2:1:
Traceback (most recent call last):
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litdata/processing/data_processor.py", line 163, in _download_data_target
raise ValueError(f"The provided {input_dir.url} isn't supported.")
ValueError: The provided None isn't supported.
Process Process-2:2:
Traceback (most recent call last):
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/litdata/processing/data_processor.py", line 163, in _download_data_target
raise ValueError(f"The provided {input_dir.url} isn't supported.")
ValueError: The provided None isn't supported.