MS MARCO Passage Retrieval - Subset Replication

[Dahlia-Chehata]

Re-Ranking with monoBERT

I've got the same results

precision@1 0.27619
recall@3    0.42698
recall@50   0.81746
recall@1000 0.84762
mrr         0.4109
mrr@10      0.40268

Re-Ranking with monoT5

I've got approximately the same results

precision@1 0.27619
recall@3    0.4746
recall@50   0.80635
recall@1000 0.84762
mrr         0.39734
mrr@10      0.38983

Issues

The current commands in the doc give the following error (caused by #127):

  File "pygaggle/pygaggle/run/evaluate_passage_ranker.py", line 208, in <module>
    main()
  File "pygaggle/pygaggle/run/evaluate_passage_ranker.py", line 178, in main
    options = PassageRankingEvaluationOptions(**vars(args))
  File ".../.../pydantic/main.py", line 338, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 1 validation error for PassageRankingEvaluationOptions
duo_model
  none is not an allowed value (type=type_error.none.not_allowed)

I worked around this by specifying the duo_model as empty string

[ronakice]

Can you check if the error pops up now without the duo_model specified?

[Dahlia-Chehata]

It's all fixed now.

[ronakice]

Awesome, merging!