Loading GLUE dataset loads CoLA by default

[zphang]

If I run:

dataset = nlp.load_dataset('glue')

The resultant dataset seems to be CoLA be default, without throwing any error. This is in contrast to calling:

metric = nlp.load_metric("glue")

which throws an error telling the user that they need to specify a task in GLUE. Should the same apply for loading datasets?

[zphang]

As a follow-up to this: It looks like the actual GLUE task name is supplied as the name argument. Is there a way to check what names/sub-datasets are available under a grouping like GLUE? That information doesn't seem to be readily available in info from nlp.list_datasets().

Edit: I found the info under Glue.BUILDER_CONFIGS

[thomwolf]

Yes so the first config is loaded by default when no name is supplied but for GLUE this should probably throw an error indeed.

We can probably just add an __init__ at the top of the class Glue(nlp.GeneratorBasedBuilder) in the glue.py script which does this check:

class Glue(nlp.GeneratorBasedBuilder):
    def __init__(self, *args, **kwargs):
        assert 'name' in kwargs and kwargs[name] is not None, "Glue has to be called with a configuration name"
        super(Glue, self).__init__(*args, **kwargs)

[lhoestq]

An error is raised if the sub-dataset is not specified :)

ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`