hot_reader pipeline crashes

[guillaumecherel]

Problem description

The "hot_reader" pipeline crashes when run through the evaluation script src/evaluation/retriever_reader/retriever_reader_eval_squad.py.

Steps to reproduce:

• Checkout master branch
• Replace the content of src/evaluation/config/retriever_reader_eval_squad_config.py with:

parameters = {
    "k_retriever": [3],
    "k_title_retriever" : [1], # must be present, but only used when retriever_type == title_bm25
    "k_reader_per_candidate": [20],
    "k_reader_total": [5],
    "threshold_score": [1.00],# must be present, but only used when retriever_type == hot_reader
    "reader_model_version": ["053b085d851196110d7a83d8e0f077d0a18470be"],
    "retriever_model_version": ["1a01b38498875d45f69b2a6721bf6fe87425da39"],
    "dpr_model_version": ["v1.0"],
    "retriever_type": ["hot_reader"], # Can be bm25, sbert, dpr, title or title_bm25
    "squad_dataset": ["test/samples/squad/tiny.json"],
    "filter_level": [None],
    "preprocessing": [False],
    "boosting" : [1], #default to 1
    "split_by": ["word"],  # Can be "word", "sentence", or "passage"
    "split_length": [1000],
}
# rules:
# corpus and retriever type requires reloading ES indexing
# filtering requires >v10
#


parameter_tuning_options = {
    # "experiment_name": "DILA_fullspfV1",
    "experiment_name": "test",

    # Tuning method alternatives:
    # - "optimization": use bayesian optimisation
    # - "grid_search"
    "tuning_method": "grid_search",

    # Additionnal options for the grid search method
    "use_cache": False,

    # Additionnal options for the optimization method
    "optimization_ncalls": 10,
}

• run python3 -m src.evaluation.retriever_reader.retriever_reader_eval_squad

After a certain time, the script display the error:

2021-07-28 16:54:02,565 — root — ERROR — single_run:349 — Could not run this config: {'boosting': 1, 'dpr_model_version': 'v1.0', 'filter_level': None, 'k_reader_per_candidate': 20, 'k_reader_total': 5, 'k_retriever': 3, 'k_title_retriever': 1, 'preprocessing': False, 'reader_model_version': '053b085d851196110d7a83d8e0f077d0a18470be', 'retriever_model_version': '1a01b38498875d45f69b2a6721bf6fe87425da39', 'retriever_type': 'hot_reader', 'split_by': 'word', 'split_length': 1000, 'squad_dataset': 'test/samples/squad/tiny.json', 'threshold_score': 1.0, 'date': '2021-07-28_16-53-35', 'hostname': 'guillaume-ThinkPad-T495', 'experiment_id': '9a7c'}. Error run() got an unexpected keyword argument 'labels'.

Solution description

Additional context / screeshot