Problem deciphering Squad Entry definition

[Abhishek-P]

As part of allenai/allennlp#5095 I am working on using this(datasets)library in AllenNLP with better ease.
I am attempting use the feature definitions of the dataset to process them.

When doing the same with SQUAD, train split I am coming across a problem.

The definition for the 'answers' field looks as follows

        "answers": {
        "answer_start": [1],
        "text": ["This is a test text"]
         }

But the definition looks as below

Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)

I am having two problems with it.

1. It is indicated as Sequence but works a dict of list, and the indexes need to be matched to work with
2. The values are elements in a list rather than a single list.

Not sure if the data modeling should be changed or the features specification should be changed to match it.

cc: @albertvillanova

Starting this discussion for help with understanding the same.
Thanks in advance.

[albertvillanova]

Hi @Abhishek-P, thanks for your question.

Please, bear in mind that due to legacy compatibility with TensorFlow Datasets, the Sequence feature with an inner dictionary is automatically converted into a dictionary of lists.

You have the explanation about this in our docs, where indeed we explicitly show the SQuAD dataset as an example: Adding dataset metadata

One specific behavior here is the fact that the Sequence field in "answers" is given a dictionary of sub-fields. As mentioned in the above note, in this case, this feature is actually converted into a dictionary of lists (instead of the list of dictionary that we read in the feature here).