Add HuggingfaceDatasetReader for using Huggingface datasets

[ghost]

Added a new reader to allow for reading huggingface datasets as instance
Mapped limited datasets.features to allenlp.data.fields

Verified for selective dataset and/or dataset configurations for training split, mentioned in the documentation comments of the reader.

datasets==1.5.0

Joint work with @divijbajaj @annajung @prajaktakini-vmware @agururajvais

Signed-off-by: Abhishek P (VMware) pab@vmware.com

Fixes #4962

Changes proposed in this pull request:
Introduce a new reader that wraps huggingface datasets to provide instances for a split of the dataset with configuration if required

[ghost]

Looking at the lint and format issues. Will address them hopefully today.

[ghost]

Locally new tests pass -
tests/data/dataset_readers/huggingface_datasets_test.py ........ . [ 25%]

But since they are doing downloads it is quite possible they fail. Will try to see if I can pin point it. But should be okay for the draft.

[dirkgr]

It's OK for tests to do downloads, but it's better if the downloads are small. Is there a tiny dataset we can use? Maybe we upload one just for this purpose?

[dirkgr]

I'm really excited to have this. I've been wanting to do this since HF came out with the datasets project.

[albertvillanova]

Thanks for this PR, @pab-vmware, @divijbajaj, @annajung, @pkini-vmware and @agururajvais.

And thanks @dirkgr for your thorough review.

Just let me know if there is anything we can do on Hugging Face Datasets to help you with the integration. 🤗

[ghost]

It's OK for tests to do downloads, but it's better if the downloads are small. Is there a tiny dataset we can use? Maybe we upload one just for this purpose?

Nice Idea, we can formulate a custom dataset and test around it to validate the mapping of datasets.features to allenlp.data.fields.

In the meantime I noticed glue is small, will change UTs to use that only.

We can probably have sanity scripts to test w.r.t specific important datasets to validate support for them, independent of UT.

Let me make one such dataset as part of it. Will ask on where to place it (my repo or somewhere else) once I have it.

[epwalsh]

This looks really good so far!

[ghost]

Looks like I need to do more changes to get it working even for glue. Will inform once it is done.

[ghost]

Working for 3 datasets. With couple different types of features.

Pending addressing of type check failures

Samples from the 2 datasets, xnli is a little too big for a comment

glue-cola:
From:{'idx': 0, 'label': 1, 'sentence': "Our friends won't buy this analysis, let alone the next one we propose."}
To:
Instance with fields:
 	 sentence: TextField of length 1 with text: 
 		[Our friends won't buy this analysis, let alone the next one we propose.]
 
 	 label: LabelField with label: 1 in namespace: 'label'. 
 	 idx: LabelField with label: 0 in namespace: 'idx'. 


_universal_dependencies - en_lines:_
From:
{'deprel': ['mark', 'det', 'nsubj', 'advcl', 'case', 'det', 'compound', 'flat', 'obl', 'case', 'det', 'compound', 'flat', 'nmod', 'punct', 'det', 'nsubj:pass', 'aux:pass', 'root', 'case', 'det', 'compound', 'flat', 'compound', 'obl', 'punct'], 'deps': ['None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None'], 'feats': ["{'Case': 'Nom'}", "{'Definite': 'Ind', 'PronType': 'Art'}", "{'Number': 'Sing'}", "{'Mood': 'Ind', 'Number': 'Sing', 'Person': '3', 'Tense': 'Pres', 'VerbForm': 'Fin'}", 'None', "{'Definite': 'Def', 'PronType': 'Art'}", "{'Number': 'Sing'}", "{'Number': 'Sing'}", "{'Number': 'Sing'}", 'None', "{'Definite': 'Ind', 'PronType': 'Art'}", "{'Number': 'Sing'}", "{'Case': 'Nom'}", "{'Number': 'Sing'}", 'None', "{'Definite': 'Def', 'PronType': 'Art'}", "{'Number': 'Sing'}", "{'Mood': 'Ind', 'Number': 'Sing', 'Person': '3', 'Tense': 'Pres', 'VerbForm': 'Fin'}", "{'Tense': 'Past', 'VerbForm': 'Part', 'Voice': 'Pass'}", 'None', "{'Definite': 'Ind', 'PronType': 'Art'}", "{'Case': 'Nom'}", "{'Case': 'Nom'}", "{'Number': 'Sing'}", "{'Number': 'Sing'}", 'None'], 'head': ['4', '3', '4', '19', '9', '9', '9', '7', '4', '14', '14', '14', '12', '9', '4', '17', '19', '19', '0', '25', '25', '25', '22', '25', '19', '19'], 'idx': 'en_lines-ud-dev-doc1-3177', 'lemmas': ['when', 'a', 'user', 'connect', 'to', 'the', 'SQL', 'server', 'database', 'through', 'a', 'Microsoft', 'Access', 'project', ',', 'the', 'connection', 'be', 'enable', 'through', 'a', 'Windows', 'NT', 'user', 'account', '.'], 'misc': ['None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', "{'SpaceAfter': 'No'}", 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', "{'SpaceAfter': 'No'}", 'None'], 'text': 'When a user connects to the SQL Server database through a Microsoft Access project, the connection is enabled through a Windows NT user account.', 'tokens': ['When', 'a', 'user', 'connects', 'to', 'the', 'SQL', 'Server', 'database', 'through', 'a', 'Microsoft', 'Access', 'project', ',', 'the', 'connection', 'is', 'enabled', 'through', 'a', 'Windows', 'NT', 'user', 'account', '.'], 'upos': [5, 8, 0, 16, 2, 8, 10, 0, 0, 2, 8, 10, 10, 0, 1, 8, 0, 17, 16, 2, 8, 10, 10, 0, 0, 1], 'xpos': [None, 'IND-SG', 'SG-NOM', 'PRES', None, 'DEF', 'SG-NOM', 'SG-NOM', 'SG-NOM', None, 'IND-SG', 'SG-NOM', 'SG-NOM', 'SG-NOM', 'Comma', 'DEF', 'SG-NOM', 'PRES-AUX', 'PASS', None, 'IND-SG', 'SG-NOM', 'SG-NOM', 'SG-NOM', 'SG-NOM', 'Period']}

To:
{'idx': <allennlp.data.fields.text_field.TextField object at 0x147f4b0c0>, 'text': <allennlp.data.fields.text_field.TextField object at 0x147c7a740>, 'tokens': <allennlp.data.fields.list_field.ListField object at 0x145b8c040>, 'lemmas': <allennlp.data.fields.list_field.ListField object at 0x145b8c820>, 'upos': <allennlp.data.fields.list_field.ListField object at 0x145b8c1f0>, 'xpos': <allennlp.data.fields.list_field.ListField object at 0x145b8cd60>, 'feats': <allennlp.data.fields.list_field.ListField object at 0x147ed3a00>, 'head': <allennlp.data.fields.list_field.ListField object at 0x147ed30d0>, 'deprel': <allennlp.data.fields.list_field.ListField object at 0x147ed39d0>, 'deps': <allennlp.data.fields.list_field.ListField object at 0x147ed3a30>, 'misc': <allennlp.data.fields.list_field.ListField object at 0x147ed3af0>}

[ghost]

Have working implementation with limited , features-> field mapping. let me work on getting the code coverage up for the diff even if I have to use slightly heavier dataset at this point.
I will also look at adding a custom HF dataset so we can use it for testing purposes.

I will try to increase the features that we are mapping but is the current change (post coverage) sufficient to go in?

[epwalsh]

I will also look at adding a custom HF dataset so we can use it for testing purposes.

That would great!

[dirkgr]

What's the status of this? Is it close? Anything you need? I'd love to have this in, maybe even for the 2.3 release!

[ghost]

Hi Dirk, So now that the coverage is up to 90%, this MR IMO although slightly Raw in w.r.t implementation can go in.
Let me re-check the known supported dataset/config and update it.

There are things that need to be done, for most of which I have added TODOs and I will pick them up one by one in the coming weeks.

• Cleanup the code so we have mapper class/method for each of the features
• Support Nested Features (Linked to Above)
• Add Custom Light Dataset and configs and use them fo in UT
• Add a test where the new reader is compared to the customer conll2003 reader - This would validate the reader in a practical manner

[ghost]

So I have updated the datasets list, added validation tests that are now skipped. Squashed all changes into a single commit. Please take a look when time permits.

I will work on cleaning up the code and providing nested support.
If the dataset download during tests becomes problematic, I will prioritize the creation of custom dataset.

[dirkgr]

Squashed all changes into a single commit.

Don't worry about doing that for our sake. It'll get squashed when we merge anyways. In fact, now I have no way of seeing just the changes you made since the last time I reviewed this.

[wolhandlerdeb]

Hi,
I would be glad to know if you add this code to the allennlp library. It would be very helpful.
Thanks

[Abhishek-P]

Hi, I would be glad to know if you add this code to the allennlp library. It would be very helpful. Thanks

Hi @wolhandlerdeb,
I am a little strapped for bandwidth and hence this has been at the bottom of my personal queue.
It is pending some work, before it is up to standards for check-in as described in #5095 (comment)

Please feel free to let clone the changes and continue the work.
I plan to get back to it in some time, although I am not exactly sure when as of yet.