Question about running example prediction script

[p-null]

Hi thanks for this great work!

I was trying to run the example prediction script provided in the readme and I got the error max() arg is an empty sequence.

To reproduce the error, I created a notebook here

Any thoughts on how to sove this issue? Thanks!

Edit:
Is it caused by this line? Why do we want to return empty {} when there is a header.

MultiQA/models/multiqa_predictor.py

Line 9 in aba17d3

if 'header' in json_dict:

[alontalmor]

Hi , thanks for noticing!

This is a recent bug do to a format change in MultiQA, i've fixed it and pushed the changes.

Can you please try again?

[p-null]

Thanks.
I ran it again and I got the following error:

100% 405604964/405604964 [00:31<00:00, 12684570.80B/s]
100% 407873900/407873900 [00:36<00:00, 11329474.30B/s]
100% 231508/231508 [00:00<00:00, 421437.03B/s]
100% 3399499/3399499 [00:01<00:00, 2039919.62B/s]
  4% 76/2067 [00:19<07:35,  4.37it/s]Encountered the loss key in the model's return dictionary which couldn't be split by the batch size. Key will be ignored.
100% 2067/2067 [08:59<00:00,  3.75it/s]
{"EM": 79.90539262062441, "f1": 87.4580633712794}
{"exact_match": 79.90539262062441, "f1": 87.45806337127941}
Traceback (most recent call last):
  File "predict.py", line 95, in <module>
    all_predictions = factory.format_predictions(args.dataset_name, all_predictions)
  File "/content/MultiQA/datasets/multiqa_factory.py", line 22, in format_predictions
    dataset_class = getattr(mod, dastaset_name)()
TypeError: __init__() missing 8 required positional arguments: 'preprocessor', 'split', 'dataset_version', 'dataset_flavor', 'dataset_specific_props', 'sample_size', 'max_contexts_in_file', and 'custom_input_file'

Can you please check this error?

[alontalmor]

Thanks, i've pushed another fix.

Also added a test under pytests/predict_test.py

[p-null]

Thanks. I ran it again and gave the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/overrides/overrides.py", line 165, in _get_base_class
    obj = namespace[components[0]]
KeyError: 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "predict.py", line 2, in <module>
    import models
  File "/content/MultiQA/MultiQA/models/__init__.py", line 1, in <module>
    from models.multiqa_reader import *
  File "/content/MultiQA/MultiQA/models/multiqa_reader.py", line 22, in <module>
    from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/__init__.py", line 1, in <module>
    from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/dataset_readers/__init__.py", line 10, in <module>
    from allennlp.data.dataset_readers.ccgbank import CcgBankDatasetReader
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/dataset_readers/ccgbank.py", line 9, in <module>
    from allennlp.data.dataset_readers.dataset_reader import DatasetReader
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/dataset_readers/dataset_reader.py", line 8, in <module>
    from allennlp.data.instance import Instance
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/instance.py", line 3, in <module>
    from allennlp.data.fields.field import DataArray, Field
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/fields/__init__.py", line 10, in <module>
    from allennlp.data.fields.knowledge_graph_field import KnowledgeGraphField
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/fields/knowledge_graph_field.py", line 14, in <module>
    from allennlp.data.token_indexers.token_indexer import TokenIndexer, TokenType
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/token_indexers/__init__.py", line 5, in <module>
    from allennlp.data.token_indexers.dep_label_indexer import DepLabelIndexer
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/token_indexers/dep_label_indexer.py", line 9, in <module>
    from allennlp.data.tokenizers.token import Token
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/tokenizers/__init__.py", line 7, in <module>
    from allennlp.data.tokenizers.word_tokenizer import WordTokenizer
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/tokenizers/word_tokenizer.py", line 8, in <module>
    from allennlp.data.tokenizers.word_splitter import WordSplitter, SpacyWordSplitter
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/tokenizers/word_splitter.py", line 14, in <module>
    from allennlp.data.token_indexers.openai_transformer_byte_pair_indexer import text_standardize
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/token_indexers/openai_transformer_byte_pair_indexer.py", line 33, in <module>
   class OpenaiTransformerBytePairIndexer(TokenIndexer[int]):
  File "/usr/local/lib/python3.6/dist-packages/allennlp/data/token_indexers/openai_transformer_byte_pair_indexer.py", line 109, in OpenaiTransformerBytePairIndexer
    def count_vocab_items(self, token: Token, counter: Dict[str, Dict[str, int]]):
  File "/usr/local/lib/python3.6/dist-packages/overrides/overrides.py", line 58, in overrides
    for super_class in _get_base_classes(sys._getframe(2), method.__globals__):
  File "/usr/local/lib/python3.6/dist-packages/overrides/overrides.py", line 75, in _get_base_classes
    class_name_components in _get_base_class_names(frame)]
  File "/usr/local/lib/python3.6/dist-packages/overrides/overrides.py", line 75, in <listcomp>
    class_name_components in _get_base_class_names(frame)]
  File "/usr/local/lib/python3.6/dist-packages/overrides/overrides.py", line 167, in _get_base_class
    obj = getattr(namespace["__builtins__"], components[0])
AttributeError: 'dict' object has no attribute 'int'

I had thought there was anything wrong in my code. After I checked it again and refered to the example and test script, I didn't find my script was wrong. Can you please check this error?

[alontalmor]

I'm sorry but i ran the tests and the prediction on SQUAD1-1 :
python predict.py --model https://multiqa.s3.amazonaws.com/models/BERTBase/SQuAD1-1.tar.gz --dataset https://multiqa.s3.amazonaws.com/data/SQuAD1-1_dev.jsonl.gz --dataset_name SQuAD

And got a correct result:
4%|▎ | 76/2067 [01:32<35:40, 1.07s/it]Encountered the loss key in the model's return dictionary which couldn't be split by the batch size. Key will be ignored.
100%|█████████▉| 2058/2067 [39:02<00:07, 1.19it/s]{"EM": 79.90539262062441, "f1": 87.4580633712794}
100%|██████████| 2067/2067 [39:10<00:00, 1.15s/it]
{"exact_match": 79.90539262062441, "f1": 87.45806337127941}

And what command did you run it?

[p-null]

Thanks for replying.
Weird that I can't reproduce the error today despite I ran the exact the same notebook. You can find it here
Predicting with GPU works good now.

The only problem might be I used to run it on GPU by specifying --cuda_device 0 while it seems to have some issues when predicting using cpu:

100% 405604964/405604964 [00:15<00:00, 25790165.88B/s]
100% 407873900/407873900 [00:14<00:00, 27279251.33B/s]
100% 231508/231508 [00:00<00:00, 940919.62B/s]
100% 3399499/3399499 [00:00<00:00, 5238596.43B/s]
  0% 0/2067 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "predict.py", line 110, in <module>
    predict(args)
  File "predict.py", line 38, in predict
    curr_pred, full_predictions = predictor.predict_json(context)
  File "/content/MultiQA/models/multiqa_predictor.py", line 27, in predict_json
    min(offset+20, len(question_instances))])
  File "/usr/local/lib/python3.6/dist-packages/allennlp/predictors/predictor.py", line 213, in predict_batch_instance
    outputs = self._model.forward_on_instances(instances)
  File "/usr/local/lib/python3.6/dist-packages/allennlp/models/model.py", line 153, in forward_on_instances
    outputs = self.decode(self(**model_input))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/MultiQA/models/multiqa_bert.py", line 107, in forward
    if torch.cuda.is_available() else torch.LongTensor(span_starts_list)
RuntimeError: legacy constructor for device type: cuda was passed device type: cpu, but device type must be: cuda

which can also be found in the notebook.