Loading datasets

[andreamad8]

While running this command as suggested in the repo:
nvidia-docker run -it --rm -v `pwd`:/decaNLP/ -u $(id -u):$(id -g) decanlp bash -c "python /decaNLP/train.py --train_tasks squad iwslt.en.de cnn_dailymail multinli.in.out sst srl zre woz.en wikisql schema --train_iterations 1 --gpu 1"
It can download and process all the following datasets: squad iwslt.en.de cnn_dailymail multinli.in.out sst srl zre.
Once it arrives to woz.en I have the following error:

process_main - zre has 840000 training examples
process_main - Loading woz.en
process_main - Adding woz.en to training datasets
downloading woz_train_en.json
downloading woz_test_de.json
downloading woz_test_en.json
downloading woz_train_de.json
downloading woz_validate_de.json
downloading woz_validate_en.json
Traceback (most recent call last):
File "/decaNLP/train.py", line 365, in
main()
File "/decaNLP/train.py", line 352, in main
field, train_sets, val_sets = prepare_data(args, field, logger)
File "/decaNLP/train.py", line 67, in prepare_data
split = get_splits(args, task, FIELD, **kwargs)[0]
File "/decaNLP/util.py", line 138, in get_splits
fields=FIELD, root=args.data, **kwargs)
File "/decaNLP/text/torchtext/datasets/generic.py", line 976, in splits
os.path.join(path, f'{train}.jsonl'), fields, **kwargs)
File "/decaNLP/text/torchtext/datasets/generic.py", line 897, in init
ex = data.Example.fromlist([context, question, answer, CONTEXT_SPECIAL, QUESTION_SPECIAL, context_question, woz_id], fields)
File "/decaNLP/text/torchtext/data/example.py", line 62, in fromlist
setattr(ex, name, [sys.intern(x) for x in field.preprocess(val)])
TypeError: 'int' object is not iterable

Thanks in advance

Andrea

[bmccann]

Looks like I broke this recently trying to save on memory. I’ll have this fixed today.

[bmccann]

847a9dd should fix this -- let me know if not

[andreamad8]

Issued solved. Thank you a lot for the quick response.