Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve reporting of BEIR results #1888

Merged
merged 2 commits into from
May 12, 2024
Merged

Improve reporting of BEIR results #1888

merged 2 commits into from
May 12, 2024

Conversation

lintool
Copy link
Member

@lintool lintool commented May 12, 2024

  • separate cqa
  • report to three decimal places

Run this:

python -m pyserini.2cr.beir --all --display-commands --dry-run

And you'll get this:

                              BM25-flat          BM25-mf             SPLADE           Contriever     Contriever-msmarco  BGE-base-en-v1.5   cohere (en-v3.0)
                          nDCG@10   R@100    nDCG@10   R@100    nDCG@10   R@100    nDCG@10   R@100    nDCG@10   R@100    nDCG@10   R@100    nDCG@10   R@100    
                           --------------     --------------     --------------     --------------     --------------     --------------     --------------
trec-covid                  0.595   0.109      0.656   0.114      0.727   0.128      0.273   0.037      0.596   0.091      0.781   0.141      0.818   0.159
bioasq                      0.522   0.769      0.465   0.715      0.498   0.739      0.302   0.541      0.383   0.607      0.415   0.632      0.457   0.679
nfcorpus                    0.322   0.246      0.325   0.250      0.347   0.284      0.317   0.294      0.328   0.301      0.373   0.337      0.386   0.351
nq                          0.305   0.751      0.329   0.760      0.538   0.930      0.254   0.771      0.498   0.925      0.541   0.942      0.616   0.956
hotpotqa                    0.633   0.796      0.603   0.740      0.687   0.818      0.481   0.705      0.638   0.777      0.726   0.873      0.707   0.823
fiqa                        0.236   0.539      0.236   0.539      0.347   0.631      0.245   0.562      0.329   0.656      0.406   0.742      0.421   0.736
signal1m                    0.330   0.370      0.330   0.370      0.301   0.340      0.234   0.257      0.278   0.322      0.289   0.311      0.263   0.283
trec-news                   0.395   0.447      0.398   0.422      0.415   0.441      0.348   0.423      0.428   0.492      0.442   0.499      0.504   0.543
robust04                    0.407   0.375      0.407   0.375      0.468   0.385      0.316   0.276      0.473   0.392      0.444   0.351      0.541   0.417
arguana                     0.397   0.932      0.414   0.943      0.520   0.974      0.379   0.901      0.446   0.977      0.636   0.992      0.540   0.982
webis-touche2020            0.442   0.582      0.367   0.538      0.247   0.471      0.167   0.374      0.204   0.442      0.257   0.487      0.326   0.516
cqa                         0.302   0.580      0.299   0.606      0.334   0.650      0.284   0.614      0.345   0.663      0.424   0.762      0.415   0.745
quora                       0.789   0.973      0.789   0.973      0.834   0.986      0.835   0.987      0.865   0.994      0.889   0.997      0.887   0.996
dbpedia-entity              0.318   0.468      0.313   0.398      0.437   0.562      0.292   0.453      0.413   0.541      0.407   0.530      0.434   0.536
scidocs                     0.149   0.348      0.158   0.356      0.159   0.373      0.149   0.360      0.165   0.378      0.217   0.496      0.203   0.451
fever                       0.651   0.918      0.753   0.931      0.788   0.946      0.682   0.936      0.758   0.949      0.863   0.972      0.890   0.965
climate-fever               0.165   0.425      0.213   0.436      0.230   0.521      0.155   0.442      0.237   0.575      0.312   0.636      0.259   0.581
scifact                     0.679   0.925      0.665   0.908      0.704   0.935      0.649   0.926      0.677   0.947      0.741   0.967      0.718   0.963
                           --------------     --------------     --------------     --------------     --------------     --------------     --------------
avg                         0.424   0.586      0.429   0.576      0.477   0.618      0.353   0.548      0.448   0.613      0.509   0.648      0.521   0.649


cqadupstack-android         0.380   0.683      0.371   0.689      0.390   0.740      0.377   0.744      0.425   0.750      0.508   0.845      0.500   0.832
cqadupstack-english         0.345   0.576      0.332   0.584      0.408   0.695      0.357   0.644      0.433   0.694      0.486   0.759      0.491   0.757
cqadupstack-gaming          0.482   0.765      0.442   0.757      0.496   0.813      0.460   0.809      0.528   0.848      0.597   0.904      0.605   0.900
cqadupstack-gis             0.290   0.612      0.290   0.646      0.315   0.632      0.241   0.579      0.302   0.627      0.413   0.768      0.392   0.744
cqadupstack-mathematica     0.202   0.488      0.205   0.521      0.238   0.580      0.184   0.513      0.235   0.573      0.316   0.692      0.304   0.667
cqadupstack-physics         0.321   0.633      0.325   0.649      0.360   0.720      0.343   0.701      0.416   0.762      0.472   0.808      0.438   0.784
cqadupstack-programmers     0.280   0.559      0.296   0.619      0.340   0.658      0.303   0.640      0.357   0.719      0.424   0.786      0.437   0.789
cqadupstack-stats           0.271   0.534      0.279   0.572      0.299   0.589      0.248   0.527      0.309   0.586      0.373   0.673      0.352   0.643
cqadupstack-tex             0.224   0.469      0.209   0.495      0.253   0.516      0.154   0.433      0.221   0.498      0.311   0.649      0.308   0.624
cqadupstack-unix            0.275   0.542      0.279   0.572      0.317   0.621      0.264   0.588      0.326   0.616      0.422   0.780      0.406   0.754
cqadupstack-webmasters      0.306   0.582      0.301   0.610      0.317   0.636      0.288   0.648      0.339   0.703      0.407   0.777      0.407   0.749
cqadupstack-wordpress       0.248   0.515      0.256   0.553      0.273   0.595      0.191   0.536      0.253   0.577      0.355   0.705      0.343   0.694
                           --------------     --------------     --------------     --------------     --------------     --------------     --------------
avg                         0.302   0.580      0.299   0.606      0.334   0.650      0.284   0.614      0.345   0.663      0.424   0.762      0.415   0.745

@thakur-nandan
Copy link
Member

LGTM.

Copy link
Member

@ehsk ehsk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@lintool lintool merged commit bf68fc5 into master May 12, 2024
@lintool lintool deleted the beir-tweaks branch May 12, 2024 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants