Skip to content

remove custom datasets (story_cloze) #549

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 17, 2021
Merged

remove custom datasets (story_cloze) #549

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ec32ce1
remove custom datasets (story_cloze)
VictorSanh Oct 15, 2021
a92ccbc
remove pkg
VictorSanh Oct 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion promptsource/custom_datasets/story_cloze/dataset_infos.json
View file
Open in desktop

This file was deleted.

127 changes: 0 additions & 127 deletions promptsource/custom_datasets/story_cloze/story_cloze.py
View file
Open in desktop

This file was deleted.

8 changes: 0 additions & 8 deletions promptsource/seqio_tasks/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import re

import datasets
import pkg_resources
import tensorflow as tf

import promptsource.utils
Expand Down Expand Up @@ -64,13 +63,6 @@ def filter_fn(ex):


def get_dataset_splits(dataset_name, subset_name=None):
# `datasets.get_dataset_infos` pulls infos from hf/datasets's master.
# story_cloze hasn't been merged yet (https://github.com/huggingface/datasets/pull/2907)
# This is a temporary fix to be able to do `import promptsource.seqio_tasks`
# Once PR 2907 is merged, we can remove this if condition (along with the `custom_datasets` folder)
# Also see `promptsource.utils.get_dataset_builder`
if dataset_name == "story_cloze":
dataset_name = pkg_resources.resource_filename("promptsource", "custom_datasets/story_cloze")
info = datasets.get_dataset_infos(dataset_name)
subset_name = subset_name or list(info.keys())[0]
return info[subset_name].splits
Expand Down
8 changes: 0 additions & 8 deletions promptsource/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# coding=utf-8

import datasets
import pkg_resources
import requests

from promptsource.templates import INCLUDED_USERS
Expand Down Expand Up @@ -34,13 +33,6 @@ def renameDatasetColumn(dataset):

def get_dataset_builder(path, conf=None):
"Get a dataset builder from name and conf."
# `datasets.load.prepare_module` pulls infos from hf/datasets's master.
# story_cloze hasn't been merged yet (https://github.com/huggingface/datasets/pull/2907)
# This is a temporary fix for the tests (more specifically test_templates.py)
# Once PR 2907 is merged, we can remove this if condition (along with the `custom_datasets` folder)
# Also see `promptsource.seqio_tasks.utils.get_dataset_splits`
if path == "story_cloze":
path = pkg_resources.resource_filename("promptsource", "custom_datasets/story_cloze")
module_path = datasets.load.prepare_module(path, dataset=True)
builder_cls = datasets.load.import_main_class(module_path[0], dataset=True)
if conf:
Expand Down