Skip to content

Commit

Permalink
Changed RM3 defaults to match settings in Indri (#539)
Browse files Browse the repository at this point in the history
  • Loading branch information
@lintool
lintool committed Dec 24, 2018
1 parent 0a293b6 commit 1aa3970
Show file tree
Hide file tree
Showing 28 changed files with 202 additions and 188 deletions.
4 changes: 2 additions & 2 deletions docs/experiments-car17.md
View file
Expand Up @@ -63,11 +63,11 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
All Topics | 0.1689 | 0.1386 | 0.1355 | 0.1516 | 0.1198 | 0.1082 |
All Topics | 0.1689 | 0.1287 | 0.1355 | 0.1516 | 0.1173 | 0.1082 |


RECIP_RANK | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
All Topics | 0.2321 | 0.1907 | 0.1857 | 0.2085 | 0.1653 | 0.1501 |
All Topics | 0.2321 | 0.1788 | 0.1857 | 0.2085 | 0.1573 | 0.1501 |


4 changes: 2 additions & 2 deletions docs/experiments-core17.md
View file
Expand Up @@ -64,11 +64,11 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
All Topics | 0.1977 | 0.2596 | 0.2700 | 0.1913 | 0.2405 | 0.2514 |
All Topics | 0.1977 | 0.2682 | 0.2700 | 0.1913 | 0.2485 | 0.2514 |


P30 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
All Topics | 0.4160 | 0.4820 | 0.4927 | 0.4373 | 0.4580 | 0.4827 |
All Topics | 0.4160 | 0.4940 | 0.4927 | 0.4373 | 0.4740 | 0.4827 |


4 changes: 2 additions & 2 deletions docs/experiments-core18.md
View file
Expand Up @@ -63,12 +63,12 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
All Topics | 0.2491 | 0.2952 | 0.2921 | 0.2522 | 0.2759 | 0.2975 |
All Topics | 0.2491 | 0.3147 | 0.2921 | 0.2522 | 0.3064 | 0.2975 |


P30 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
All Topics | 0.3580 | 0.4200 | 0.4007 | 0.3627 | 0.3753 | 0.4073 |
All Topics | 0.3580 | 0.4193 | 0.4007 | 0.3627 | 0.4007 | 0.4073 |



Expand Down
24 changes: 12 additions & 12 deletions docs/experiments-cw09b.md
View file
Expand Up @@ -110,29 +110,29 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
TREC 2010 Web Track: Topics 51-100 | 0.1126 | 0.1171 | 0.0928 | 0.1060 | 0.1117 | 0.1086 |
TREC 2011 Web Track: Topics 101-150 | 0.1094 | 0.1142 | 0.0974 | 0.0958 | 0.0964 | 0.0879 |
TREC 2012 Web Track: Topics 151-200 | 0.1106 | 0.1382 | 0.1315 | 0.1069 | 0.1167 | 0.1212 |
TREC 2010 Web Track: Topics 51-100 | 0.1126 | 0.0933 | 0.0928 | 0.1060 | 0.1019 | 0.1086 |
TREC 2011 Web Track: Topics 101-150 | 0.1094 | 0.1081 | 0.0974 | 0.0958 | 0.0837 | 0.0879 |
TREC 2012 Web Track: Topics 151-200 | 0.1106 | 0.1107 | 0.1315 | 0.1069 | 0.1059 | 0.1212 |


P30 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
TREC 2010 Web Track: Topics 51-100 | 0.2681 | 0.2819 | 0.2354 | 0.2431 | 0.2611 | 0.2618 |
TREC 2011 Web Track: Topics 101-150 | 0.2513 | 0.2700 | 0.2393 | 0.2147 | 0.2147 | 0.2167 |
TREC 2012 Web Track: Topics 151-200 | 0.2167 | 0.2473 | 0.2553 | 0.2080 | 0.2053 | 0.2140 |
TREC 2010 Web Track: Topics 51-100 | 0.2681 | 0.2389 | 0.2354 | 0.2431 | 0.2312 | 0.2618 |
TREC 2011 Web Track: Topics 101-150 | 0.2513 | 0.2467 | 0.2393 | 0.2147 | 0.2067 | 0.2167 |
TREC 2012 Web Track: Topics 151-200 | 0.2167 | 0.1920 | 0.2553 | 0.2080 | 0.1980 | 0.2140 |


NDCG20 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
TREC 2010 Web Track: Topics 51-100 | 0.1354 | 0.1545 | 0.1637 | 0.1143 | 0.1362 | 0.1454 |
TREC 2011 Web Track: Topics 101-150 | 0.1890 | 0.1823 | 0.1833 | 0.1619 | 0.1520 | 0.1509 |
TREC 2012 Web Track: Topics 151-200 | 0.1014 | 0.1329 | 0.1441 | 0.0868 | 0.1059 | 0.1030 |
TREC 2010 Web Track: Topics 51-100 | 0.1354 | 0.1369 | 0.1637 | 0.1143 | 0.1185 | 0.1454 |
TREC 2011 Web Track: Topics 101-150 | 0.1890 | 0.1916 | 0.1833 | 0.1619 | 0.1447 | 0.1509 |
TREC 2012 Web Track: Topics 151-200 | 0.1014 | 0.0917 | 0.1441 | 0.0868 | 0.0896 | 0.1030 |


ERR20 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
TREC 2010 Web Track: Topics 51-100 | 0.0733 | 0.0865 | 0.0981 | 0.0599 | 0.0649 | 0.0742 |
TREC 2011 Web Track: Topics 101-150 | 0.0959 | 0.1042 | 0.1091 | 0.0849 | 0.0865 | 0.0820 |
TREC 2012 Web Track: Topics 151-200 | 0.1304 | 0.2224 | 0.2355 | 0.1305 | 0.1475 | 0.1558 |
TREC 2010 Web Track: Topics 51-100 | 0.0733 | 0.0747 | 0.0981 | 0.0599 | 0.0592 | 0.0742 |
TREC 2011 Web Track: Topics 101-150 | 0.0959 | 0.0960 | 0.1091 | 0.0849 | 0.0786 | 0.0820 |
TREC 2012 Web Track: Topics 151-200 | 0.1304 | 0.1493 | 0.2355 | 0.1305 | 0.1334 | 0.1558 |


16 changes: 8 additions & 8 deletions docs/experiments-cw12.md
View file
Expand Up @@ -73,25 +73,25 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | QL | QL+RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1695 | 0.1498 | 0.1493 | 0.1280 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2469 | 0.2496 | 0.2467 | 0.2383 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1695 | 0.1464 | 0.1493 | 0.1291 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2469 | 0.2325 | 0.2467 | 0.2168 |


P30 | BM25 | BM25+RM3 | QL | QL+RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2767 | 0.2407 | 0.2613 | 0.2207 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.4533 | 0.4180 | 0.4380 | 0.4107 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2767 | 0.2387 | 0.2613 | 0.2347 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.4533 | 0.4073 | 0.4380 | 0.3793 |


NDCG20 | BM25 | BM25+RM3 | QL | QL+RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2086 | 0.1836 | 0.1993 | 0.1611 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2578 | 0.2430 | 0.2228 | 0.2258 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2086 | 0.2033 | 0.1993 | 0.1725 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.2578 | 0.2530 | 0.2228 | 0.2066 |


ERR20 | BM25 | BM25+RM3 | QL | QL+RM3 |
:---------------------------------------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1284 | 0.0974 | 0.1232 | 0.0913 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1630 | 0.1713 | 0.1321 | 0.1407 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1284 | 0.1264 | 0.1232 | 0.1008 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1630 | 0.1655 | 0.1321 | 0.1218 |


16 changes: 8 additions & 8 deletions docs/experiments-cw12b13.md
View file
Expand Up @@ -88,25 +88,25 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0468 | 0.0450 | 0.0435 | 0.0397 | 0.0319 | 0.0359 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.0224 | 0.0189 | 0.0180 | 0.0235 | 0.0205 | 0.0186 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0468 | 0.0412 | 0.0435 | 0.0397 | 0.0322 | 0.0359 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.0224 | 0.0210 | 0.0180 | 0.0235 | 0.0203 | 0.0186 |


P30 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2113 | 0.1787 | 0.1840 | 0.1767 | 0.1373 | 0.1513 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1273 | 0.1133 | 0.1107 | 0.1373 | 0.1173 | 0.1167 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.2113 | 0.1713 | 0.1840 | 0.1767 | 0.1507 | 0.1513 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1273 | 0.1207 | 0.1107 | 0.1373 | 0.1173 | 0.1167 |


NDCG20 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1286 | 0.1228 | 0.1287 | 0.1107 | 0.0880 | 0.1143 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1185 | 0.1012 | 0.0964 | 0.1177 | 0.1024 | 0.1001 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.1286 | 0.1129 | 0.1287 | 0.1107 | 0.0920 | 0.1143 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1185 | 0.1080 | 0.0964 | 0.1177 | 0.1003 | 0.1001 |


ERR20 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0838 | 0.0879 | 0.0943 | 0.0769 | 0.0568 | 0.0780 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1201 | 0.1039 | 0.0929 | 0.1091 | 0.1036 | 0.0896 |
[TREC 2013 Web Track: Topics 201-250](http://trec.nist.gov/data/web2013.html)| 0.0838 | 0.0763 | 0.0943 | 0.0769 | 0.0553 | 0.0780 |
[TREC 2014 Web Track: Topics 251-300](http://trec.nist.gov/data/web2014.html)| 0.1201 | 0.1065 | 0.0929 | 0.1091 | 0.0929 | 0.0896 |


12 changes: 6 additions & 6 deletions docs/experiments-disk12.md
View file
Expand Up @@ -90,15 +90,15 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
TREC-1 Ad Hoc Track: Topics 51-100 | 0.2273 | 0.2617 | 0.2640 | 0.2189 | 0.2478 | 0.2501 |
TREC-2 Ad Hoc Track: Topics 101-150 | 0.2010 | 0.2600 | 0.2722 | 0.2015 | 0.2485 | 0.2593 |
TREC-3 Ad Hoc Track: Topics 151-200 | 0.2580 | 0.3227 | 0.3318 | 0.2518 | 0.2996 | 0.3103 |
TREC-1 Ad Hoc Track: Topics 51-100 | 0.2273 | 0.2634 | 0.2640 | 0.2189 | 0.2435 | 0.2501 |
TREC-2 Ad Hoc Track: Topics 101-150 | 0.2010 | 0.2587 | 0.2722 | 0.2015 | 0.2442 | 0.2593 |
TREC-3 Ad Hoc Track: Topics 151-200 | 0.2580 | 0.3390 | 0.3318 | 0.2518 | 0.3042 | 0.3103 |


P30 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
TREC-1 Ad Hoc Track: Topics 51-100 | 0.4533 | 0.4867 | 0.5067 | 0.4520 | 0.4653 | 0.4953 |
TREC-2 Ad Hoc Track: Topics 101-150 | 0.4280 | 0.4580 | 0.4753 | 0.4207 | 0.4453 | 0.4740 |
TREC-3 Ad Hoc Track: Topics 151-200 | 0.4740 | 0.5040 | 0.5100 | 0.4580 | 0.4933 | 0.5167 |
TREC-1 Ad Hoc Track: Topics 51-100 | 0.4533 | 0.4800 | 0.5067 | 0.4520 | 0.4627 | 0.4953 |
TREC-2 Ad Hoc Track: Topics 101-150 | 0.4280 | 0.4593 | 0.4753 | 0.4207 | 0.4420 | 0.4740 |
TREC-3 Ad Hoc Track: Topics 151-200 | 0.4740 | 0.5273 | 0.5100 | 0.4580 | 0.4913 | 0.5167 |


12 changes: 6 additions & 6 deletions docs/experiments-gov2.md
View file
Expand Up @@ -90,15 +90,15 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.2689 | 0.2943 | 0.2665 | 0.2681 | 0.2806 | 0.2666 |
[TREC 2005 Terabyte Track: Topics 751-800](http://trec.nist.gov/data/terabyte05.html)| 0.3390 | 0.3800 | 0.3664 | 0.3303 | 0.3628 | 0.3646 |
[TREC 2006 Terabyte Track: Topics 801-850](http://trec.nist.gov/data/terabyte06.html)| 0.3080 | 0.3356 | 0.3069 | 0.2996 | 0.3173 | 0.3084 |
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.2689 | 0.2844 | 0.2665 | 0.2681 | 0.2708 | 0.2666 |
[TREC 2005 Terabyte Track: Topics 751-800](http://trec.nist.gov/data/terabyte05.html)| 0.3390 | 0.3820 | 0.3664 | 0.3303 | 0.3559 | 0.3646 |
[TREC 2006 Terabyte Track: Topics 801-850](http://trec.nist.gov/data/terabyte06.html)| 0.3080 | 0.3377 | 0.3069 | 0.2996 | 0.3154 | 0.3084 |


P30 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.4864 | 0.5313 | 0.4986 | 0.4755 | 0.4952 | 0.4932 |
[TREC 2005 Terabyte Track: Topics 751-800](http://trec.nist.gov/data/terabyte05.html)| 0.5540 | 0.5873 | 0.5933 | 0.5347 | 0.5720 | 0.5840 |
[TREC 2006 Terabyte Track: Topics 801-850](http://trec.nist.gov/data/terabyte06.html)| 0.4907 | 0.5160 | 0.5033 | 0.4720 | 0.4773 | 0.4920 |
[TREC 2004 Terabyte Track: Topics 701-750](http://trec.nist.gov/data/terabyte04.html)| 0.4864 | 0.5190 | 0.4986 | 0.4755 | 0.4925 | 0.4932 |
[TREC 2005 Terabyte Track: Topics 751-800](http://trec.nist.gov/data/terabyte05.html)| 0.5540 | 0.5920 | 0.5933 | 0.5347 | 0.5620 | 0.5840 |
[TREC 2006 Terabyte Track: Topics 801-850](http://trec.nist.gov/data/terabyte06.html)| 0.4907 | 0.5160 | 0.5033 | 0.4720 | 0.4847 | 0.4920 |


8 changes: 4 additions & 4 deletions docs/experiments-mb11.md
View file
Expand Up @@ -88,13 +88,13 @@ With the above commands, you should be able to replicate the following results:

MAP | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2011 Microblog Track](http://trec.nist.gov/data/microblog2011.html)| 0.3384 | 0.3621 | 0.4008 | 0.3584 | 0.4097 | 0.4201 |
[TREC 2012 Microblog Track](http://trec.nist.gov/data/microblog2012.html)| 0.1948 | 0.2124 | 0.2309 | 0.2102 | 0.2397 | 0.2474 |
[TREC 2011 Microblog Track](http://trec.nist.gov/data/microblog2011.html)| 0.3384 | 0.3650 | 0.4008 | 0.3584 | 0.3923 | 0.4201 |
[TREC 2012 Microblog Track](http://trec.nist.gov/data/microblog2012.html)| 0.1948 | 0.2193 | 0.2309 | 0.2102 | 0.2389 | 0.2474 |


P30 | BM25 | BM25+RM3 | BM25+AX | QL | QL+RM3 | QL+AX |
:---------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|
[TREC 2011 Microblog Track](http://trec.nist.gov/data/microblog2011.html)| 0.3959 | 0.4088 | 0.4612 | 0.4061 | 0.4483 | 0.4408 |
[TREC 2012 Microblog Track](http://trec.nist.gov/data/microblog2012.html)| 0.3316 | 0.3463 | 0.3554 | 0.3333 | 0.3571 | 0.3842 |
[TREC 2011 Microblog Track](http://trec.nist.gov/data/microblog2011.html)| 0.3959 | 0.4170 | 0.4612 | 0.4061 | 0.4435 | 0.4408 |
[TREC 2012 Microblog Track](http://trec.nist.gov/data/microblog2012.html)| 0.3316 | 0.3463 | 0.3554 | 0.3333 | 0.3520 | 0.3842 |


0 comments on commit 1aa3970

Please sign in to comment.