-
Notifications
You must be signed in to change notification settings - Fork 377
Closed
Description
To allow manual download of a dataset, we can add a data_dir field in the builder. It will be helpful to know what the conf variable does.
def get_dataset_builder(path, conf=None):
"Get a dataset builder from name and conf."
module_path = datasets.load.prepare_module(path, dataset=True)
builder_cls = datasets.load.import_main_class(module_path[0], dataset=True)
if conf:
builder_instance = builder_cls(name=conf, cache_dir=None, hash=module_path[1],
data_dir = 'path/to/local/')
else:
builder_instance = builder_cls(cache_dir=None, hash=module_path[1])
return builder_instanceAlso, since Story Cloze is a special case we can add it to the dataset_list field i.e dataset_list += ["story_cloze"].
A couple of things that we need to figure out ?
- What is the directory for storing the dataset locally in the server?
- Since the dataset is not public we should only show a few samples to help in making the templates.
- If the PR is not merged by the weekend we should consider adding custom data loaders.
Metadata
Metadata
Assignees
Labels
No labels