Update MS MARCO (V1) documentation (#1754)

castorini · Feb 5, 2022 · 5cdf9ec · 5cdf9ec
1 parent 4c33f13
commit 5cdf9ec
Show file tree

Hide file tree

Showing 36 changed files with 610 additions and 100 deletions.
diff --git a/docs/regressions-dl19-doc-docTTTTTquery.md b/docs/regressions-dl19-doc-docTTTTTquery.md
@@ -112,3 +112,4 @@ Settings tuned on the MS MARCO document sparse judgments _may not_ work well on
 Note that retrieval metrics are computed to depth 100 hits per query (as opposed to 1000 hits per query for DL19 passage ranking).
 Also, remember that we keep qrels of _all_ relevance grades, unlike the case for DL19 passage ranking, where relevance grade 1 needs to be discarded when computing certain metrics.
 
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl19-doc-segmented-docTTTTTquery.md b/docs/regressions-dl19-doc-segmented-docTTTTTquery.md
@@ -113,3 +113,4 @@ Settings tuned on the MS MARCO document sparse judgments _may not_ work well on
 Note that retrieval metrics are computed to depth 100 hits per query (as opposed to 1000 hits per query for DL19 passage ranking).
 Also, remember that we keep qrels of _all_ relevance grades, unlike the case for DL19 passage ranking, where relevance grade 1 needs to be discarded when computing certain metrics.
 
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl19-doc-segmented.md b/docs/regressions-dl19-doc-segmented.md
@@ -144,3 +144,5 @@ Settings tuned on the MS MARCO document sparse judgments _may not_ work well on
 
 Note that retrieval metrics are computed to depth 100 hits per query (as opposed to 1000 hits per query for DL19 passage ranking).
 Also, remember that we keep qrels of _all_ relevance grades, unlike the case for DL19 passage ranking, where relevance grade 1 needs to be discarded when computing certain metrics.
+
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl19-doc.md b/docs/regressions-dl19-doc.md
@@ -155,3 +155,5 @@ These regressions correspond to official TREC 2019 Deep Learning Track submissio
 + `bm25tuned_rm3` = BM25 (tuned) + RM3, `k1=3.44`, `b=0.87`
 + `bm25tuned_ax` = BM25 (tuned) + Ax, `k1=3.44`, `b=0.87`
 + `bm25tuned_prf` = BM25 (tuned) + PRF, `k1=3.44`, `b=0.87`
+
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl19-passage-docTTTTTquery.md b/docs/regressions-dl19-passage-docTTTTTquery.md
@@ -130,3 +130,5 @@ Settings tuned on the MS MARCO passage sparse judgments _may not_ work well on t
 
 Note that retrieval metrics are computed to depth 1000 hits per query (as opposed to 100 hits per query for DL19 doc ranking).
 Also, for computing nDCG, remember that we keep qrels of _all_ relevance grades, whereas for other metrics (e.g., MAP), relevance grade 1 is considered not relevant (i.e., use the `-l 2` option in `trec_eval`).
+
+Note this regression was revamped as part of [#1730](https://github.com/castorini/anserini/issues/1730), but the results did not change.
diff --git a/docs/regressions-dl19-passage.md b/docs/regressions-dl19-passage.md
@@ -160,3 +160,5 @@ These regressions correspond to official TREC 2019 Deep Learning Track submissio
 + `bm25tuned_rm3_p` = BM25 (tuned) + RM3, `k1=0.82`, `b=0.68`
 + `bm25tuned_ax_p` = BM25 (tuned) + Ax, `k1=0.82`, `b=0.68`
 + `bm25tuned_prf_p` = BM25 (tuned) + PRF, `k1=0.82`, `b=0.68`
+
+Note this regression was revamped as part of [#1730](https://github.com/castorini/anserini/issues/1730), but the results did not change.
diff --git a/docs/regressions-dl20-doc-docTTTTTquery.md b/docs/regressions-dl20-doc-docTTTTTquery.md
@@ -121,3 +121,5 @@ Some of these regressions correspond to official TREC 2020 Deep Learning Track s
 
 + `d_d2q_bm25` = BM25 (default), `k1=0.9`, `b=0.4`
 + `d_d2q_bm25rm3` = BM25 (default) + RM3, `k1=0.9`, `b=0.4`
+
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl20-doc-segmented-docTTTTTquery.md b/docs/regressions-dl20-doc-segmented-docTTTTTquery.md
@@ -118,3 +118,4 @@ Settings tuned on the MS MARCO document sparse judgments _may not_ work well on
 Note that retrieval metrics are computed to depth 100 hits per query (as opposed to 1000 hits per query for DL20 passage ranking).
 Also, remember that we keep qrels of _all_ relevance grades, unlike the case for DL20 passage ranking, where relevance grade 1 needs to be discarded when computing certain metrics.
 
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl20-doc-segmented.md b/docs/regressions-dl20-doc-segmented.md
@@ -149,3 +149,5 @@ Settings tuned on the MS MARCO document sparse judgments _may not_ work well on
 
 Note that retrieval metrics are computed to depth 100 hits per query (as opposed to 1000 hits per query for DL20 passage ranking).
 Also, remember that we keep qrels of _all_ relevance grades, unlike the case for DL20 passage ranking, where relevance grade 1 needs to be discarded when computing certain metrics.
+
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl20-doc.md b/docs/regressions-dl20-doc.md
@@ -138,3 +138,5 @@ Some of these regressions correspond to official TREC 2020 Deep Learning Track s
 
 + `d_bm25` = BM25 (default), `k1=0.9`, `b=0.4`
 + `d_bm25rm3` = BM25 (default) + RM3, `k1=0.9`, `b=0.4`
+
+Note that [#1721](https://github.com/castorini/anserini/issues/1721) slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-dl20-passage-docTTTTTquery.md b/docs/regressions-dl20-passage-docTTTTTquery.md
@@ -157,3 +157,5 @@ Some of these regressions correspond to official TREC 2020 Deep Learning Track s
 
 + `p_d2q_bm25` = BM25 (default), `k1=0.9`, `b=0.4`
 + `p_d2q_bm25rm3` = BM25 (default) + RM3, `k1=0.9`, `b=0.4`
+
+Note this regression was revamped as part of [#1730](https://github.com/castorini/anserini/issues/1730), but the results did not change.
diff --git a/docs/regressions-dl20-passage.md b/docs/regressions-dl20-passage.md
@@ -179,3 +179,5 @@ Some of these regressions correspond to official TREC 2020 Deep Learning Track s
 
 + `p_bm25` = BM25 (default), `k1=0.9`, `b=0.4`
 + `p_bm25rm3` = BM25 (default) + RM3, `k1=0.9`, `b=0.4`
+
+Note this regression was revamped as part of [#1730](https://github.com/castorini/anserini/issues/1730), but the results did not change.
diff --git a/docs/regressions-msmarco-doc-docTTTTTquery.md b/docs/regressions-msmarco-doc-docTTTTTquery.md
@@ -103,13 +103,13 @@ To generate an MS MARCO submission with the BM25 tuned parameters, corresponding
 
 ```bash
 $ sh target/appassembler/bin/SearchMsmarco -hits 100 -k1 4.68 -b 0.87 -threads 9 \
-   -index indexes/lucene-index.msmarco-doc-docTTTTTquery-per-doc.pos+docvectors+raw \
-   -queries src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
-   -output runs/run.msmarco-doc-docTTTTTquery-per-doc.bm25-tuned.txt
+    -index indexes/lucene-index.msmarco-doc-docTTTTTquery \
+    -queries src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -output runs/run.msmarco-doc-docTTTTTquery.bm25-tuned.txt
 
 $ python tools/scripts/msmarco/msmarco_doc_eval.py \
-   --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
-   --run runs/run.msmarco-doc-docTTTTTquery-per-doc.bm25-tuned.txt
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc-docTTTTTquery.bm25-tuned.txt
 
 #####################
 MRR @100: 0.3265190296491929
@@ -118,3 +118,48 @@ QueriesRanked: 5193
 ```
 
 This run corresponds to the MS MARCO document ranking leaderboard entry "Anserini's BM25 + doc2query-T5 expansion (per document), parameters tuned for recall@100 (k1=4.68, b=0.87)" dated 2020/12/11, and is reported in the Lin et al. (SIGIR 2021) Pyserini paper.
+
+As of February 2022, following resolution of [#1721](https://github.com/castorini/anserini/issues/1721), BM25 runs for the MS MARCO leaderboard can be generated with the commands below.
+For default parameters (`k1=0.9`, `b=0.4`):
+
+```
+$ sh target/appassembler/bin/SearchCollection \
+    -index indexes/lucene-index.msmarco-doc-docTTTTTquery/ \
+    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -topicreader TsvInt \
+    -output runs/run.msmarco-doc-docTTTTTquery.bm25-default.txt \
+    -format msmarco \
+    -bm25 -hits 100
+
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc-docTTTTTquery.bm25-default.txt
+
+#####################
+MRR @100: 0.2880115937357742
+QueriesRanked: 5193
+#####################
+```
+
+For tuned parameters (`k1=4.68`, `b=0.87`):
+
+```
+$ sh target/appassembler/bin/SearchCollection \
+    -index indexes/lucene-index.msmarco-doc-docTTTTTquery/ \
+    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -topicreader TsvInt \
+    -output runs/run.msmarco-doc-docTTTTTquery.bm25-tuned.txt \
+    -format msmarco \
+    -bm25 -bm25.k1 4.68 -bm25.b 0.87 -hits 100
+
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc-docTTTTTquery.bm25-tuned.txt
+
+#####################
+MRR @100: 0.3268656233100833
+QueriesRanked: 5193
+#####################
+```
+
+Note that the resolution of [#1721](https://github.com/castorini/anserini/issues/1721) _did_ slightly change the results, since we corrected underlying issues with data preparation.
diff --git a/docs/regressions-msmarco-doc-segmented-docTTTTTquery.md b/docs/regressions-msmarco-doc-segmented-docTTTTTquery.md
@@ -97,16 +97,16 @@ The MaxP passage retrieval functionality is available in `SearchCollection`.
 To generate an MS MARCO submission with the BM25 default parameters, corresponding to "BM25 (default)" above:
 
 ```bash
-$ target/appassembler/bin/SearchCollection -topicreader TsvString \
-   -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
-   -index indexes/lucene-index.msmarco-doc-docTTTTTquery-per-passage.pos+docvectors+raw \
-   -output runs/run.msmarco-doc-docTTTTTquery-per-passage.bm25-default.txt -format msmarco \
-   -bm25 -bm25.k1 0.9 -bm25.b 0.4 -hits 1000 \
-   -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 100
+$ sh target/appassembler/bin/SearchCollection -topicreader TsvString \
+    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -index indexes/lucene-index.msmarco-doc-segmented-docTTTTTquery/ \
+    -output runs/run.msmarco-doc-segmented-docTTTTTquery.bm25-default.txt -format msmarco \
+    -bm25 -bm25.k1 0.9 -bm25.b 0.4 -hits 1000 \
+    -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 100
 
 $ python tools/scripts/msmarco/msmarco_doc_eval.py \
-   --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
-   --run runs/run.msmarco-doc-docTTTTTquery-per-passage.bm25-default.txt
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc-segmented-docTTTTTquery.bm25-default.txt
 
 #####################
 MRR @100: 0.31779258157039536
@@ -119,16 +119,16 @@ Note that the above command uses `-format msmarco` to directly generate a run in
 To generate an MS MARCO submission with the BM25 tuned parameters, corresponding to "BM25 (tuned)" above:
 
 ```bash
-$ target/appassembler/bin/SearchCollection -topicreader TsvString \
-   -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
-   -index indexes/lucene-index.msmarco-doc-docTTTTTquery-per-passage.pos+docvectors+raw \
-   -output runs/run.msmarco-doc-docTTTTTquery-per-passage.bm25-tuned.txt -format msmarco \
-   -bm25 -bm25.k1 2.56 -bm25.b 0.59 -hits 1000 \
-   -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 100
+$ sh target/appassembler/bin/SearchCollection -topicreader TsvString \
+    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -index indexes/lucene-index.msmarco-doc-segmented-docTTTTTquery/ \
+    -output runs/run.msmarco-doc-segmented-docTTTTTquery.bm25-tuned.txt -format msmarco \
+    -bm25 -bm25.k1 2.56 -bm25.b 0.59 -hits 1000 \
+    -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 100
 
 $ python tools/scripts/msmarco/msmarco_doc_eval.py \
-   --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
-   --run runs/run.msmarco-doc-docTTTTTquery-per-passage.bm25-tuned.txt
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc-segmented-docTTTTTquery.bm25-tuned.txt
 
 #####################
 MRR @100: 0.32081861579183746
@@ -137,4 +137,33 @@ QueriesRanked: 5193
 ```
 
 This run corresponds to the MS MARCO document ranking leaderboard entry "Anserini's BM25 + doc2query-T5 expansion (per passage), parameters tuned for recall@100 (k1=2.56, b=0.59)" dated 2020/12/11, and is reported in the Lin et al. (SIGIR 2021) Pyserini paper.
-Again, note that the above command uses `-format msmarco` to directly generate a run in the MS MARCO output format.
+Again, note that the above command uses `-format msmarco` to directly generate a run in the MS MARCO output format.
+
+As of February 2022, following resolution of [#1721](https://github.com/castorini/anserini/issues/1721), BM25 runs for the MS MARCO leaderboard can be generated with the same commands as above.
+However, the effectiveness has changed slightly, since we corrected underlying issues with data preparation.
+
+For default parameters (`k1=0.9`, `b=0.4`):
+
+```
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc-segmented-docTTTTTquery.bm25-default.txt
+
+#####################
+MRR @100: 0.317905445196054
+QueriesRanked: 5193
+#####################
+```
+
+For tuned parameters (`k1=2.56`, `b=0.59`):
+
+```
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc-segmented-docTTTTTquery.bm25-tuned.txt
+
+#####################
+MRR @100: 0.3209184381409182
+QueriesRanked: 5193
+#####################
+```
diff --git a/docs/regressions-msmarco-doc-segmented.md b/docs/regressions-msmarco-doc-segmented.md
@@ -147,14 +147,14 @@ To generate an MS MARCO submission with the BM25 default parameters, correspondi
 ```bash
 $ target/appassembler/bin/SearchCollection -topicreader TsvString \
    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
-   -index indexes/lucene-index.msmarco-doc-per-passage.pos+docvectors+raw \
-   -output runs/run.msmarco-doc-per-passage.bm25-default.txt -format msmarco \
+   -index indexes/lucene-index.msmarco-doc-segmented/ \
+   -output runs/run.msmarco-doc-segmented.bm25-default.txt -format msmarco \
    -bm25 -bm25.k1 0.9 -bm25.b 0.4 -hits 1000 \
    -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 100
 
 $ python tools/scripts/msmarco/msmarco_doc_eval.py \
    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
-   --run runs/run.msmarco-doc-per-passage.bm25-default.txt
+   --run runs/run.msmarco-doc-segmented.bm25-default.txt
 
 #####################
 MRR @100: 0.2682349308946578
@@ -170,14 +170,14 @@ To generate an MS MARCO submission with the BM25 tuned parameters, corresponding
 ```bash
 $ target/appassembler/bin/SearchCollection -topicreader TsvString \
    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
-   -index indexes/lucene-index.msmarco-doc-per-passage.pos+docvectors+raw \
-   -output runs/run.msmarco-doc-per-passage.bm25-tuned.txt -format msmarco \
+   -index indexes/lucene-index.msmarco-doc-segmented/ \
+   -output runs/run.msmarco-doc-segmented.bm25-tuned.txt -format msmarco \
    -bm25 -bm25.k1 2.16 -bm25.b 0.61 -hits 1000 \
    -selectMaxPassage -selectMaxPassage.delimiter "#" -selectMaxPassage.hits 100
 
 $ python tools/scripts/msmarco/msmarco_doc_eval.py \
    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
-   --run runs/run.msmarco-doc-per-passage.bm25-tuned.txt
+   --run runs/run.msmarco-doc-segmented.bm25-tuned.txt
 
 #####################
 MRR @100: 0.2751202109946902
@@ -187,3 +187,32 @@ QueriesRanked: 5193
 
 This run corresponds to the MS MARCO document ranking leaderboard entry "Anserini's BM25 (per passage), parameters tuned for recall@100 (k1=2.16, b=0.61)" dated 2021/01/20, and is reported in the Lin et al. (SIGIR 2021) Pyserini paper.
 Again, note that the above command uses `-format msmarco` to directly generate a run in the MS MARCO output format.
+
+As of February 2022, following resolution of [#1721](https://github.com/castorini/anserini/issues/1721), BM25 runs for the MS MARCO leaderboard can be generated with the same commands as above.
+However, the effectiveness has changed slightly, since we corrected underlying issues with data preparation.
+
+For default parameters (`k1=0.9`, `b=0.4`):
+
+```
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+   --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+   --run runs/run.msmarco-doc-segmented.bm25-default.txt
+
+#####################
+MRR @100: 0.26851990908986706
+QueriesRanked: 5193
+#####################
+```
+
+For tuned parameters (`k1=2.16`, `b=0.61`):
+
+```
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+   --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+   --run runs/run.msmarco-doc-segmented.bm25-tuned.txt
+
+#####################
+MRR @100: 0.27551963417683756
+QueriesRanked: 5193
+#####################
+```
diff --git a/docs/regressions-msmarco-doc.md b/docs/regressions-msmarco-doc.md
@@ -138,13 +138,13 @@ To generate an MS MARCO submission with the BM25 default parameters, correspondi
 
 ```bash
 $ sh target/appassembler/bin/SearchMsmarco -hits 100 -k1 0.9 -b 0.4 -threads 9 \
-   -index indexes/lucene-index.msmarco-doc.pos+docvectors+raw \
-   -queries src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
-   -output runs/run.msmarco-doc.bm25-default.txt
+    -index indexes/lucene-index.msmarco-doc/ \
+    -queries src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -output runs/run.msmarco-doc.bm25-default.txt
 
 $ python tools/scripts/msmarco/msmarco_doc_eval.py \
-   --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
-   --run runs/run.msmarco-doc.bm25-default.txt
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc.bm25-default.txt
 
 #####################
 MRR @100: 0.23005723505603573
@@ -158,7 +158,7 @@ To generate an MS MARCO submission with the BM25 tuned parameters, corresponding
 
 ```bash
 $ sh target/appassembler/bin/SearchMsmarco -hits 100 -k1 4.46 -b 0.82 -threads 9 \
-   -index indexes/lucene-index.msmarco-doc.pos+docvectors+raw \
+   -index indexes/lucene-index.msmarco-doc/ \
    -queries src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
    -output runs/run.msmarco-doc.bm25-tuned.txt
 
@@ -173,3 +173,48 @@ QueriesRanked: 5193
 ```
 
 This run was _not_ submitted to the MS MARCO document ranking leaderboard, but is reported in the Lin et al. (SIGIR 2021) Pyserini paper.
+
+As of February 2022, following resolution of [#1721](https://github.com/castorini/anserini/issues/1721), BM25 runs for the MS MARCO leaderboard can be generated with the commands below.
+For default parameters (`k1=0.9`, `b=0.4`):
+
+```
+$ sh target/appassembler/bin/SearchCollection \
+    -index indexes/lucene-index.msmarco-doc/ \
+    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -topicreader TsvInt \
+    -output runs/run.msmarco-doc.bm25-default.txt \
+    -format msmarco \
+    -bm25 -hits 100
+
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc.bm25-default.txt
+
+#####################
+MRR @100: 0.22994387925437856
+QueriesRanked: 5193
+#####################
+```
+
+For tuned parameters (`k1=4.46`, `b=0.82`):
+
+```
+$ sh target/appassembler/bin/SearchCollection \
+    -index indexes/lucene-index.msmarco-doc/ \
+    -topics src/main/resources/topics-and-qrels/topics.msmarco-doc.dev.txt \
+    -topicreader TsvInt \
+    -output runs/run.msmarco-doc.bm25-tuned.txt \
+    -format msmarco \
+    -bm25 -bm25.k1 4.46 -bm25.b 0.82 -hits 100
+
+$ python tools/scripts/msmarco/msmarco_doc_eval.py \
+    --judgments src/main/resources/topics-and-qrels/qrels.msmarco-doc.dev.txt \
+    --run runs/run.msmarco-doc.bm25-tuned.txt
+
+#####################
+MRR @100: 0.2766351807440808
+QueriesRanked: 5193
+#####################
+```
+
+Note that the resolution of [#1721](https://github.com/castorini/anserini/issues/1721) _did_ slightly change the results, since we corrected underlying issues with data preparation.