Add MAIR datasets by sunnweiwei · Pull Request #2363 · embeddings-benchmark/mteb

sunnweiwei · 2025-03-14T21:54:19Z

Add all 126 datasets introduced in https://arxiv.org/abs/2410.10127.

PS This is an updated pull request, as the previous one is outdated.

Code Quality

Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
x ] Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

Adding datasets checklist

Reason for dataset addition: ...

I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
- sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
- intfloat/multilingual-e5-small
I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
I have filled out the metadata object in the dataset file (find documentation on it here).
Run tests locally to make sure nothing is broken using make test.
Run the formatter to format the code using make lint.

Adding a model checklist

I have filled out the ModelMeta object to the extent possible
I have ensured that my model can be loaded using
- mteb.get_model(model_name, revision) and
- mteb.get_model_meta(model_name, revision)
I have tested the implementation works on a representative set of tasks.

mteb/tasks/MAIR/eng/MAIR.py

sunnweiwei · 2025-03-14T23:44:55Z

Also, add benchmarks to include these datasets

mteb/benchmarks/benchmarks.py

Samoed · 2025-03-15T10:10:12Z

mteb/tasks/Retrieval/eng/MAIR.py

+__all__ = []
+
+
+TASK2SPLIT = {'Competition-Math': ['queries'], 'ProofWiki_Proof': ['queries'], 'ProofWiki_Reference': ['queries'], 'Stacks_Proof': ['queries'], 'Stacks_Reference': ['queries'], 'Stein_Proof': ['queries'], 'Stein_Reference': ['queries'], 'Trench_Proof': ['queries'], 'Trench_Reference': ['queries'], 'TAD': ['queries'], 'TAS2': ['queries'], 'StackMathQA': ['queries'], 'APPS': ['queries'], 'CodeEditSearch': ['queries'], 'CodeSearchNet': ['queries'], 'Conala': ['queries'], 'HumanEval-X': ['queries'], 'LeetCode': ['queries'], 'MBPP': ['queries'], 'RepoBench': ['queries'], 'TLDR': ['queries'], 'SWE-Bench-Lite': ['astropy__astropy_12544_queries', 'astropy__astropy_13158_queries', 'astropy__astropy_13162_queries', 'astropy__astropy_13398_queries', 'astropy__astropy_13438_queries', 'astropy__astropy_14439_queries', 'astropy__astropy_14701_queries', 'astropy__astropy_14966_queries', 'astropy__astropy_7441_queries', 'astropy__astropy_8707_queries', 'django__django_11501_queries', 'django__django_12091_queries', 'django__django_13192_queries', 'django__django_13218_queries', 'django__django_13884_queries', 'django__django_14441_queries', 'django__django_15481_queries', 'django__django_15869_queries', 'django__django_16901_queries', 'django__django_17065_queries', 'matplotlib__matplotlib_20518_queries', 'matplotlib__matplotlib_23314_queries', 'matplotlib__matplotlib_23913_queries', 'matplotlib__matplotlib_24627_queries', 'matplotlib__matplotlib_24849_queries', 'matplotlib__matplotlib_25027_queries', 'matplotlib__matplotlib_25238_queries', 'matplotlib__matplotlib_25404_queries', 'matplotlib__matplotlib_25430_queries', 'matplotlib__matplotlib_25746_queries', 'mwaskom__seaborn_2389_queries', 'mwaskom__seaborn_2576_queries', 'mwaskom__seaborn_2766_queries', 'mwaskom__seaborn_2813_queries', 'mwaskom__seaborn_2853_queries', 'mwaskom__seaborn_2946_queries', 'mwaskom__seaborn_2979_queries', 'mwaskom__seaborn_2996_queries', 'mwaskom__seaborn_3202_queries', 'mwaskom__seaborn_3407_queries', 'pallets__flask_4045_queries', 'pallets__flask_4074_queries', 'pallets__flask_4160_queries', 'pallets__flask_4169_queries', 'pallets__flask_4544_queries', 'pallets__flask_4575_queries', 'pallets__flask_4642_queries', 'pallets__flask_4992_queries', 'pallets__flask_5014_queries', 'pallets__flask_5063_queries', 'psf__requests_1537_queries', 'psf__requests_1713_queries', 'psf__requests_1733_queries', 'psf__requests_1766_queries', 'psf__requests_2193_queries', 'psf__requests_2466_queries', 'psf__requests_2821_queries', 'psf__requests_3362_queries', 'psf__requests_5414_queries', 'psf__requests_863_queries', 'pydata__xarray_4339_queries', 'pydata__xarray_4767_queries', 'pydata__xarray_4827_queries', 'pydata__xarray_4911_queries', 'pydata__xarray_4966_queries', 'pydata__xarray_5033_queries', 'pydata__xarray_5682_queries', 'pydata__xarray_6135_queries', 'pydata__xarray_6461_queries', 'pydata__xarray_7391_queries', 'pylint_dev__pylint_4398_queries', 'pylint_dev__pylint_4604_queries', 'pylint_dev__pylint_5175_queries', 'pylint_dev__pylint_5446_queries', 'pylint_dev__pylint_5613_queries', 'pylint_dev__pylint_6358_queries', 'pylint_dev__pylint_6412_queries', 'pylint_dev__pylint_6556_queries', 'pylint_dev__pylint_8281_queries', 'pylint_dev__pylint_8757_queries', 'pytest_dev__pytest_10371_queries', 'pytest_dev__pytest_11047_queries', 'pytest_dev__pytest_11148_queries', 'pytest_dev__pytest_5356_queries', 'pytest_dev__pytest_6680_queries', 'pytest_dev__pytest_7158_queries', 'pytest_dev__pytest_7352_queries', 'pytest_dev__pytest_9064_queries', 'pytest_dev__pytest_9279_queries', 'scikit_learn__scikit_learn_10198_queries', 'scikit_learn__scikit_learn_10803_queries', 'scikit_learn__scikit_learn_10949_queries', 'scikit_learn__scikit_learn_11333_queries', 'scikit_learn__scikit_learn_11635_queries', 'scikit_learn__scikit_learn_12827_queries', 'scikit_learn__scikit_learn_12834_queries', 'scikit_learn__scikit_learn_13302_queries', 'scikit_learn__scikit_learn_13392_queries', 'scikit_learn__scikit_learn_13779_queries', 'sphinx_doc__sphinx_11312_queries', 'sphinx_doc__sphinx_11502_queries', 'sphinx_doc__sphinx_7356_queries', 'sphinx_doc__sphinx_7590_queries', 'sphinx_doc__sphinx_7757_queries', 'sphinx_doc__sphinx_7831_queries', 'sphinx_doc__sphinx_8125_queries', 'sphinx_doc__sphinx_8863_queries', 'sphinx_doc__sphinx_9309_queries', 'sphinx_doc__sphinx_9828_queries', 'sympy__sympy_13091_queries', 'sympy__sympy_14817_queries', 'sympy__sympy_14821_queries', 'sympy__sympy_15151_queries', 'sympy__sympy_15933_queries', 'sympy__sympy_16493_queries', 'sympy__sympy_16858_queries', 'sympy__sympy_17251_queries', 'sympy__sympy_18532_queries', 'sympy__sympy_20212_queries'], 'Apple': ['queries'], 'ConvFinQA': ['queries'], 'FinQA': ['queries'], 'FinanceBench': ['queries'], 'HC3Finance': ['queries'], 'TAT-DQA': ['queries'], 'Trade-the-event': ['queries'], 'AY2': ['queries'], 'ELI5': ['queries'], 'Fever': ['queries'], 'TREx': ['queries'], 'WnCw': ['queries'], 'WnWi': ['queries'], 'WoW': ['queries'], 'zsRE': ['queries'], 'AILA2019-Case': ['queries'], 'AILA2019-Statutes': ['queries'], 'BSARD': ['queries'], 'BillSum': ['queries'], 'CUAD': ['GOOSEHEADINSURANCE_queries', 'GRANTIERRAENERGY_queries', 'HarpoonTherapeutics_queries', 'Monsanto_Company_queries'], 'GerDaLIR': ['queries'], 'LeCaRDv2': ['queries'], 'LegalQuAD': ['queries'], 'REGIR-EU2UK': ['queries'], 'REGIR-UK2EU': ['queries'], 'ArguAna': ['queries'], 'CQADupStack': ['CQADupStack_Android_queries', 'CQADupStack_English_queries', 'CQADupStack_Gaming_queries', 'CQADupStack_Gis_queries', 'CQADupStack_Math_queries', 'CQADupStack_Physics_queries', 'CQADupStack_Programmers_queries', 'CQADupStack_Stats_queries', 'CQADupStack_Tex_queries', 'CQADupStack_Unix_queries', 'CQADupStack_WebMasters_queries', 'CQADupStack_Wordpress_queries'], 'FiQA': ['queries'], 'NFCorpus': ['queries'], 'Quora': ['queries'], 'SciDocs': ['queries'], 'SciFact': ['queries'], 'TopiOCQA': ['queries'], 'Touche': ['queries'], 'Trec-Covid': ['queries'], 'ACORDAR': ['queries'], 'CPCD': ['queries'], 'ChroniclingAmericaQA': ['queries'], 'Monant': ['queries'], 'NTCIR': ['queries'], 'PointRec': ['queries'], 'ProCIS-Dialog': ['queries'], 'ProCIS-Turn': ['queries'], 'QuanTemp': ['queries'], 'WebTableSearch': ['queries'], 'CARE': ['queries'], 'MISeD': ['Bmr006_queries', 'Bro027_queries', 'covid4_queries', 'covid9_queries', 'education4_queries'], 'SParC': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'SParC-SQL': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'Spider': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'Spider-SQL': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'LitSearch': ['queries'], 'CAsT_2019': ['queries'], 'CAsT_2020': ['queries'], 'CAsT_2021': ['queries'], 'CAsT_2022': ['queries'], 'Core_2017': ['queries'], 'Microblog_2011': ['queries'], 'Microblog_2012': ['queries'], 'Microblog_2013': ['queries'], 'Microblog_2014': ['queries'], 'PrecisionMedicine_2017': ['queries'], 'PrecisionMedicine_2018': ['queries'], 'PrecisionMedicine_2019': ['queries'], 'PrecisionMedicine-Article_2019': ['queries'], 'PrecisionMedicine-Article_2020': ['queries'], 'CliniDS_2014': ['queries'], 'CliniDS_2015': ['queries'], 'CliniDS_2016': ['queries'], 'ClinicalTrials_2021': ['queries'], 'ClinicalTrials_2022': ['queries'], 'ClinicalTrials_2023': ['queries'], 'DD_2015': ['queries'], 'DD_2016': ['queries'], 'DD_2017': ['queries'], 'FairRanking_2020': ['queries'], 'FairRanking_2021': ['queries'], 'FairRanking_2022': ['queries'], 'Genomics-AdHoc_2004': ['queries'], 'Genomics-AdHoc_2005': ['queries'], 'Genomics-AdHoc_2006': ['queries'], 'Genomics-AdHoc_2007': ['queries'], 'TREC-Legal_2011': ['queries'], 'NeuCLIR-Tech_2023': ['queries'], 'NeuCLIR_2022': ['queries'], 'NeuCLIR_2023': ['queries'], 'ProductSearch_2023': ['queries'], 'ToT_2023': ['queries'], 'ToT_2024': ['queries'], 'FoodAPI': ['queries'], 'HuggingfaceAPI': ['queries'], 'PytorchAPI': ['queries'], 'SpotifyAPI': ['queries'], 'TMDB': ['queries'], 'TensorAPI': ['queries'], 'ToolBench': ['queries'], 'WeatherAPI': ['queries'], 'ExcluIR': ['queries'], 'Core17': ['queries'], 'News21': ['queries'], 'Robust04': ['queries'], 'InstructIR': ['queries'], 'NevIR': ['queries'], 'IFEval': ['detectable_format__number_bullet_lists_2078_queries', 'detectable_format__number_bullet_lists_102_queries', 'detectable_format__number_bullet_lists_2195_queries', 'detectable_format__number_bullet_lists_2314_queries', 'detectable_format__number_bullet_lists_1934_queries', 'detectable_format__number_bullet_lists_2667_queries', 'detectable_format__number_bullet_lists_1634_queries', 'detectable_format__number_bullet_lists_2100_queries', 'detectable_format__number_bullet_lists_1286_queries', 'detectable_format__number_bullet_lists_2457_queries', 'keywords__letter_frequency_1130_queries', 'keywords__letter_frequency_2107_queries', 'keywords__letter_frequency_1964_queries', 'keywords__letter_frequency_2265_queries', 'detectable_format__constrained_response_3752_queries', 'detectable_format__constrained_response_3755_queries', 'detectable_format__constrained_response_3754_queries', 'detectable_format__constrained_response_3753_queries', 'detectable_format__constrained_response_227_queries', 'detectable_format__constrained_response_3749_queries', 'detectable_format__constrained_response_3756_queries', 'detectable_format__constrained_response_3751_queries', 'detectable_format__constrained_response_3750_queries', 'detectable_format__constrained_response_3757_queries', 'punctuation__no_comma_2245_queries', 'punctuation__no_comma_1107_queries', 'punctuation__no_comma_1162_queries', 'punctuation__no_comma_1418_queries', 'punctuation__no_comma_1001_queries', 'punctuation__no_comma_1187_queries', 'punctuation__no_comma_1738_queries', 'punctuation__no_comma_1300_queries', 'punctuation__no_comma_2069_queries', 'punctuation__no_comma_1643_queries', 'keywords__existence_3156_queries', 'keywords__existence_2485_queries', 'keywords__existence_1531_queries', 'keywords__existence_3732_queries', 'keywords__existence_2662_queries', 'change_case__english_capital_2341_queries', 'change_case__english_capital_3186_queries', 'change_case__english_capital_2563_queries', 'change_case__english_capital_1999_queries', 'change_case__english_capital_24_queries', 'change_case__english_capital_1645_queries', 'change_case__english_lowercase_1122_queries', 'change_case__english_lowercase_1361_queries', 'change_case__english_lowercase_1019_queries', 'change_case__english_lowercase_1087_queries', 'change_case__english_lowercase_1667_queries', 'change_case__english_lowercase_1516_queries', 'change_case__english_lowercase_1535_queries', 'change_case__english_lowercase_1593_queries', 'change_case__english_lowercase_1843_queries', 'keywords__frequency_1393_queries', 'keywords__frequency_1733_queries', 'keywords__frequency_2142_queries', 'keywords__frequency_2292_queries', 'keywords__frequency_1498_queries', 'keywords__frequency_1203_queries', 'keywords__frequency_1857_queries', 'length_constraints__number_sentences_1837_queries', 'length_constraints__number_sentences_2674_queries', 'length_constraints__number_sentences_2617_queries', 'length_constraints__number_sentences_1381_queries', 'length_constraints__number_sentences_2266_queries', 'length_constraints__number_sentences_1268_queries', 'length_constraints__number_sentences_179_queries', 'length_constraints__number_paragraphs_1236_queries', 'length_constraints__number_paragraphs_2941_queries', 'length_constraints__number_paragraphs_1248_queries', 'length_constraints__number_paragraphs_1858_queries', 'length_constraints__number_paragraphs_1377_queries', 'length_constraints__number_paragraphs_2357_queries', 'length_constraints__number_paragraphs_2921_queries', 'length_constraints__number_paragraphs_1082_queries', 'length_constraints__number_paragraphs_2467_queries', 'combination__two_responses_1591_queries', 'combination__two_responses_1793_queries', 'combination__two_responses_2912_queries', 'combination__two_responses_1332_queries', 'combination__two_responses_2383_queries', 'combination__two_responses_136_queries', 'combination__two_responses_1098_queries', 'combination__two_responses_1746_queries', 'combination__two_responses_247_queries', 'combination__two_responses_2918_queries', 'detectable_content__postscript_2273_queries', 'detectable_content__postscript_2070_queries', 'detectable_content__postscript_1800_queries', 'detectable_content__postscript_1305_queries', 'detectable_content__postscript_1759_queries', 'detectable_content__postscript_1367_queries', 'detectable_content__postscript_1537_queries', 'detectable_content__postscript_1879_queries', 'detectable_content__postscript_1246_queries', 'detectable_content__postscript_1620_queries', 'startend__end_checker_2398_queries', 'startend__end_checker_1902_queries', 'startend__end_checker_2268_queries', 'startend__end_checker_1659_queries', 'startend__end_checker_1893_queries', 'startend__end_checker_2475_queries', 'startend__end_checker_1128_queries', 'startend__end_checker_1939_queries', 'startend__end_checker_1446_queries', 'startend__end_checker_1220_queries', 'detectable_content__number_placeholders_3280_queries', 'detectable_content__number_placeholders_1372_queries', 'detectable_content__number_placeholders_3221_queries', 'detectable_content__number_placeholders_1927_queries', 'detectable_content__number_placeholders_3126_queries', 'detectable_content__number_placeholders_2164_queries', 'detectable_content__number_placeholders_2136_queries', 'detectable_content__number_placeholders_2304_queries', 'detectable_content__number_placeholders_3743_queries', 'length_constraints__number_words_2323_queries', 'length_constraints__number_words_1072_queries', 'length_constraints__number_words_1258_queries', 'length_constraints__number_words_1251_queries', 'length_constraints__number_words_164_queries', 'detectable_format__number_highlighted_sections_168_queries', 'detectable_format__number_highlighted_sections_1237_queries', 'detectable_format__number_highlighted_sections_1601_queries', 'detectable_format__number_highlighted_sections_167_queries', 'detectable_format__number_highlighted_sections_1773_queries', 'detectable_format__number_highlighted_sections_1646_queries', 'detectable_format__number_highlighted_sections_1379_queries', 'detectable_format__number_highlighted_sections_1307_queries', 'detectable_format__number_highlighted_sections_1886_queries', 'detectable_format__number_highlighted_sections_1644_queries', 'detectable_format__json_format_1094_queries', 'detectable_format__json_format_1148_queries', 'detectable_format__json_format_1137_queries', 'detectable_format__json_format_1075_queries', 'detectable_format__json_format_2857_queries', 'detectable_format__json_format_3223_queries', 'detectable_format__json_format_2404_queries', 'detectable_format__json_format_321_queries', 'detectable_format__json_format_13_queries', 'change_case__capital_word_frequency_2820_queries', 'change_case__capital_word_frequency_2849_queries', 'change_case__capital_word_frequency_2870_queries', 'change_case__capital_word_frequency_1592_queries', 'detectable_format__multiple_sections_2023_queries', 'detectable_format__multiple_sections_1548_queries', 'detectable_format__multiple_sections_2925_queries', 'detectable_format__multiple_sections_1131_queries', 'detectable_format__multiple_sections_357_queries', 'startend__quotation_2015_queries', 'startend__quotation_219_queries', 'startend__quotation_2010_queries', 'startend__quotation_1658_queries', 'startend__quotation_1325_queries', 'startend__quotation_1776_queries', 'startend__quotation_2239_queries', 'startend__quotation_1845_queries', 'startend__quotation_2209_queries', 'length_constraints__nth_paragraph_first_word_2880_queries', 'length_constraints__nth_paragraph_first_word_181_queries', 'length_constraints__nth_paragraph_first_word_2250_queries', 'length_constraints__nth_paragraph_first_word_2215_queries', 'length_constraints__nth_paragraph_first_word_3073_queries', 'length_constraints__nth_paragraph_first_word_2590_queries', 'length_constraints__nth_paragraph_first_word_3624_queries', 'length_constraints__nth_paragraph_first_word_1954_queries', 'detectable_format__title_1262_queries', 'detectable_format__title_2229_queries', 'detectable_format__title_295_queries', 'detectable_format__title_2097_queries', 'detectable_format__title_1802_queries', 'detectable_format__title_1322_queries', 'detectable_format__title_2969_queries', 'detectable_format__title_3057_queries', 'detectable_format__title_1551_queries', 'detectable_format__title_2807_queries']}


In mteb we've already have apps and CodeEditSearch, NFcorpus. Can you compare your tasks with existing? Also, I think these dayasets have different citations and you should include them

KennethEnevoldsen · 2025-06-15T18:58:58Z

@sunnweiwei, it seems like this pr has gotten stale. I will close it for now, but do re-open it if you have the time to finish it

Muennighoff · 2025-08-19T15:19:06Z

Would be great to still get this in @sunnweiwei if interested! I think the community would love to see a MAIR leaderboard tab!

sunnweiwei added 4 commits March 14, 2025 17:25

Create __init__.py

5fe16c3

Create MAIR.py

3c6f04f

Create __init__.py

990f12e

Update __init__.py

8bb7943

Samoed reviewed Mar 14, 2025

View reviewed changes

mteb/tasks/MAIR/eng/MAIR.py Show resolved Hide resolved

sunnweiwei added 7 commits March 14, 2025 19:35

Delete mteb/tasks/MAIR directory

8dd7de6

Add files via upload

7250db8

Update __init__.py

0807fd8

Update __init__.py

27f6282

Delete mteb/tasks/Retrieval/MAIR directory

55c85fd

Add files via upload

6945019

Update benchmarks.py

9552a63

Update adding_a_dataset.md

d40bc5f

Samoed requested changes Mar 15, 2025

View reviewed changes

Update benchmarks.py

9fb02c6

KennethEnevoldsen closed this Jun 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MAIR datasets#2363

Add MAIR datasets#2363
sunnweiwei wants to merge 13 commits intoembeddings-benchmark:mainfrom
sunnweiwei:main

sunnweiwei commented Mar 14, 2025

Uh oh!

Uh oh!

sunnweiwei commented Mar 14, 2025

Uh oh!

Uh oh!

Samoed Mar 15, 2025

Uh oh!

KennethEnevoldsen commented Jun 15, 2025

Uh oh!

Muennighoff commented Aug 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		__all__ = []


		TASK2SPLIT = {'Competition-Math': ['queries'], 'ProofWiki_Proof': ['queries'], 'ProofWiki_Reference': ['queries'], 'Stacks_Proof': ['queries'], 'Stacks_Reference': ['queries'], 'Stein_Proof': ['queries'], 'Stein_Reference': ['queries'], 'Trench_Proof': ['queries'], 'Trench_Reference': ['queries'], 'TAD': ['queries'], 'TAS2': ['queries'], 'StackMathQA': ['queries'], 'APPS': ['queries'], 'CodeEditSearch': ['queries'], 'CodeSearchNet': ['queries'], 'Conala': ['queries'], 'HumanEval-X': ['queries'], 'LeetCode': ['queries'], 'MBPP': ['queries'], 'RepoBench': ['queries'], 'TLDR': ['queries'], 'SWE-Bench-Lite': ['astropy__astropy_12544_queries', 'astropy__astropy_13158_queries', 'astropy__astropy_13162_queries', 'astropy__astropy_13398_queries', 'astropy__astropy_13438_queries', 'astropy__astropy_14439_queries', 'astropy__astropy_14701_queries', 'astropy__astropy_14966_queries', 'astropy__astropy_7441_queries', 'astropy__astropy_8707_queries', 'django__django_11501_queries', 'django__django_12091_queries', 'django__django_13192_queries', 'django__django_13218_queries', 'django__django_13884_queries', 'django__django_14441_queries', 'django__django_15481_queries', 'django__django_15869_queries', 'django__django_16901_queries', 'django__django_17065_queries', 'matplotlib__matplotlib_20518_queries', 'matplotlib__matplotlib_23314_queries', 'matplotlib__matplotlib_23913_queries', 'matplotlib__matplotlib_24627_queries', 'matplotlib__matplotlib_24849_queries', 'matplotlib__matplotlib_25027_queries', 'matplotlib__matplotlib_25238_queries', 'matplotlib__matplotlib_25404_queries', 'matplotlib__matplotlib_25430_queries', 'matplotlib__matplotlib_25746_queries', 'mwaskom__seaborn_2389_queries', 'mwaskom__seaborn_2576_queries', 'mwaskom__seaborn_2766_queries', 'mwaskom__seaborn_2813_queries', 'mwaskom__seaborn_2853_queries', 'mwaskom__seaborn_2946_queries', 'mwaskom__seaborn_2979_queries', 'mwaskom__seaborn_2996_queries', 'mwaskom__seaborn_3202_queries', 'mwaskom__seaborn_3407_queries', 'pallets__flask_4045_queries', 'pallets__flask_4074_queries', 'pallets__flask_4160_queries', 'pallets__flask_4169_queries', 'pallets__flask_4544_queries', 'pallets__flask_4575_queries', 'pallets__flask_4642_queries', 'pallets__flask_4992_queries', 'pallets__flask_5014_queries', 'pallets__flask_5063_queries', 'psf__requests_1537_queries', 'psf__requests_1713_queries', 'psf__requests_1733_queries', 'psf__requests_1766_queries', 'psf__requests_2193_queries', 'psf__requests_2466_queries', 'psf__requests_2821_queries', 'psf__requests_3362_queries', 'psf__requests_5414_queries', 'psf__requests_863_queries', 'pydata__xarray_4339_queries', 'pydata__xarray_4767_queries', 'pydata__xarray_4827_queries', 'pydata__xarray_4911_queries', 'pydata__xarray_4966_queries', 'pydata__xarray_5033_queries', 'pydata__xarray_5682_queries', 'pydata__xarray_6135_queries', 'pydata__xarray_6461_queries', 'pydata__xarray_7391_queries', 'pylint_dev__pylint_4398_queries', 'pylint_dev__pylint_4604_queries', 'pylint_dev__pylint_5175_queries', 'pylint_dev__pylint_5446_queries', 'pylint_dev__pylint_5613_queries', 'pylint_dev__pylint_6358_queries', 'pylint_dev__pylint_6412_queries', 'pylint_dev__pylint_6556_queries', 'pylint_dev__pylint_8281_queries', 'pylint_dev__pylint_8757_queries', 'pytest_dev__pytest_10371_queries', 'pytest_dev__pytest_11047_queries', 'pytest_dev__pytest_11148_queries', 'pytest_dev__pytest_5356_queries', 'pytest_dev__pytest_6680_queries', 'pytest_dev__pytest_7158_queries', 'pytest_dev__pytest_7352_queries', 'pytest_dev__pytest_9064_queries', 'pytest_dev__pytest_9279_queries', 'scikit_learn__scikit_learn_10198_queries', 'scikit_learn__scikit_learn_10803_queries', 'scikit_learn__scikit_learn_10949_queries', 'scikit_learn__scikit_learn_11333_queries', 'scikit_learn__scikit_learn_11635_queries', 'scikit_learn__scikit_learn_12827_queries', 'scikit_learn__scikit_learn_12834_queries', 'scikit_learn__scikit_learn_13302_queries', 'scikit_learn__scikit_learn_13392_queries', 'scikit_learn__scikit_learn_13779_queries', 'sphinx_doc__sphinx_11312_queries', 'sphinx_doc__sphinx_11502_queries', 'sphinx_doc__sphinx_7356_queries', 'sphinx_doc__sphinx_7590_queries', 'sphinx_doc__sphinx_7757_queries', 'sphinx_doc__sphinx_7831_queries', 'sphinx_doc__sphinx_8125_queries', 'sphinx_doc__sphinx_8863_queries', 'sphinx_doc__sphinx_9309_queries', 'sphinx_doc__sphinx_9828_queries', 'sympy__sympy_13091_queries', 'sympy__sympy_14817_queries', 'sympy__sympy_14821_queries', 'sympy__sympy_15151_queries', 'sympy__sympy_15933_queries', 'sympy__sympy_16493_queries', 'sympy__sympy_16858_queries', 'sympy__sympy_17251_queries', 'sympy__sympy_18532_queries', 'sympy__sympy_20212_queries'], 'Apple': ['queries'], 'ConvFinQA': ['queries'], 'FinQA': ['queries'], 'FinanceBench': ['queries'], 'HC3Finance': ['queries'], 'TAT-DQA': ['queries'], 'Trade-the-event': ['queries'], 'AY2': ['queries'], 'ELI5': ['queries'], 'Fever': ['queries'], 'TREx': ['queries'], 'WnCw': ['queries'], 'WnWi': ['queries'], 'WoW': ['queries'], 'zsRE': ['queries'], 'AILA2019-Case': ['queries'], 'AILA2019-Statutes': ['queries'], 'BSARD': ['queries'], 'BillSum': ['queries'], 'CUAD': ['GOOSEHEADINSURANCE_queries', 'GRANTIERRAENERGY_queries', 'HarpoonTherapeutics_queries', 'Monsanto_Company_queries'], 'GerDaLIR': ['queries'], 'LeCaRDv2': ['queries'], 'LegalQuAD': ['queries'], 'REGIR-EU2UK': ['queries'], 'REGIR-UK2EU': ['queries'], 'ArguAna': ['queries'], 'CQADupStack': ['CQADupStack_Android_queries', 'CQADupStack_English_queries', 'CQADupStack_Gaming_queries', 'CQADupStack_Gis_queries', 'CQADupStack_Math_queries', 'CQADupStack_Physics_queries', 'CQADupStack_Programmers_queries', 'CQADupStack_Stats_queries', 'CQADupStack_Tex_queries', 'CQADupStack_Unix_queries', 'CQADupStack_WebMasters_queries', 'CQADupStack_Wordpress_queries'], 'FiQA': ['queries'], 'NFCorpus': ['queries'], 'Quora': ['queries'], 'SciDocs': ['queries'], 'SciFact': ['queries'], 'TopiOCQA': ['queries'], 'Touche': ['queries'], 'Trec-Covid': ['queries'], 'ACORDAR': ['queries'], 'CPCD': ['queries'], 'ChroniclingAmericaQA': ['queries'], 'Monant': ['queries'], 'NTCIR': ['queries'], 'PointRec': ['queries'], 'ProCIS-Dialog': ['queries'], 'ProCIS-Turn': ['queries'], 'QuanTemp': ['queries'], 'WebTableSearch': ['queries'], 'CARE': ['queries'], 'MISeD': ['Bmr006_queries', 'Bro027_queries', 'covid4_queries', 'covid9_queries', 'education4_queries'], 'SParC': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'SParC-SQL': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'Spider': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'Spider-SQL': ['chinook_1_queries', 'college_2_queries', 'store_1_queries'], 'LitSearch': ['queries'], 'CAsT_2019': ['queries'], 'CAsT_2020': ['queries'], 'CAsT_2021': ['queries'], 'CAsT_2022': ['queries'], 'Core_2017': ['queries'], 'Microblog_2011': ['queries'], 'Microblog_2012': ['queries'], 'Microblog_2013': ['queries'], 'Microblog_2014': ['queries'], 'PrecisionMedicine_2017': ['queries'], 'PrecisionMedicine_2018': ['queries'], 'PrecisionMedicine_2019': ['queries'], 'PrecisionMedicine-Article_2019': ['queries'], 'PrecisionMedicine-Article_2020': ['queries'], 'CliniDS_2014': ['queries'], 'CliniDS_2015': ['queries'], 'CliniDS_2016': ['queries'], 'ClinicalTrials_2021': ['queries'], 'ClinicalTrials_2022': ['queries'], 'ClinicalTrials_2023': ['queries'], 'DD_2015': ['queries'], 'DD_2016': ['queries'], 'DD_2017': ['queries'], 'FairRanking_2020': ['queries'], 'FairRanking_2021': ['queries'], 'FairRanking_2022': ['queries'], 'Genomics-AdHoc_2004': ['queries'], 'Genomics-AdHoc_2005': ['queries'], 'Genomics-AdHoc_2006': ['queries'], 'Genomics-AdHoc_2007': ['queries'], 'TREC-Legal_2011': ['queries'], 'NeuCLIR-Tech_2023': ['queries'], 'NeuCLIR_2022': ['queries'], 'NeuCLIR_2023': ['queries'], 'ProductSearch_2023': ['queries'], 'ToT_2023': ['queries'], 'ToT_2024': ['queries'], 'FoodAPI': ['queries'], 'HuggingfaceAPI': ['queries'], 'PytorchAPI': ['queries'], 'SpotifyAPI': ['queries'], 'TMDB': ['queries'], 'TensorAPI': ['queries'], 'ToolBench': ['queries'], 'WeatherAPI': ['queries'], 'ExcluIR': ['queries'], 'Core17': ['queries'], 'News21': ['queries'], 'Robust04': ['queries'], 'InstructIR': ['queries'], 'NevIR': ['queries'], 'IFEval': ['detectable_format__number_bullet_lists_2078_queries', 'detectable_format__number_bullet_lists_102_queries', 'detectable_format__number_bullet_lists_2195_queries', 'detectable_format__number_bullet_lists_2314_queries', 'detectable_format__number_bullet_lists_1934_queries', 'detectable_format__number_bullet_lists_2667_queries', 'detectable_format__number_bullet_lists_1634_queries', 'detectable_format__number_bullet_lists_2100_queries', 'detectable_format__number_bullet_lists_1286_queries', 'detectable_format__number_bullet_lists_2457_queries', 'keywords__letter_frequency_1130_queries', 'keywords__letter_frequency_2107_queries', 'keywords__letter_frequency_1964_queries', 'keywords__letter_frequency_2265_queries', 'detectable_format__constrained_response_3752_queries', 'detectable_format__constrained_response_3755_queries', 'detectable_format__constrained_response_3754_queries', 'detectable_format__constrained_response_3753_queries', 'detectable_format__constrained_response_227_queries', 'detectable_format__constrained_response_3749_queries', 'detectable_format__constrained_response_3756_queries', 'detectable_format__constrained_response_3751_queries', 'detectable_format__constrained_response_3750_queries', 'detectable_format__constrained_response_3757_queries', 'punctuation__no_comma_2245_queries', 'punctuation__no_comma_1107_queries', 'punctuation__no_comma_1162_queries', 'punctuation__no_comma_1418_queries', 'punctuation__no_comma_1001_queries', 'punctuation__no_comma_1187_queries', 'punctuation__no_comma_1738_queries', 'punctuation__no_comma_1300_queries', 'punctuation__no_comma_2069_queries', 'punctuation__no_comma_1643_queries', 'keywords__existence_3156_queries', 'keywords__existence_2485_queries', 'keywords__existence_1531_queries', 'keywords__existence_3732_queries', 'keywords__existence_2662_queries', 'change_case__english_capital_2341_queries', 'change_case__english_capital_3186_queries', 'change_case__english_capital_2563_queries', 'change_case__english_capital_1999_queries', 'change_case__english_capital_24_queries', 'change_case__english_capital_1645_queries', 'change_case__english_lowercase_1122_queries', 'change_case__english_lowercase_1361_queries', 'change_case__english_lowercase_1019_queries', 'change_case__english_lowercase_1087_queries', 'change_case__english_lowercase_1667_queries', 'change_case__english_lowercase_1516_queries', 'change_case__english_lowercase_1535_queries', 'change_case__english_lowercase_1593_queries', 'change_case__english_lowercase_1843_queries', 'keywords__frequency_1393_queries', 'keywords__frequency_1733_queries', 'keywords__frequency_2142_queries', 'keywords__frequency_2292_queries', 'keywords__frequency_1498_queries', 'keywords__frequency_1203_queries', 'keywords__frequency_1857_queries', 'length_constraints__number_sentences_1837_queries', 'length_constraints__number_sentences_2674_queries', 'length_constraints__number_sentences_2617_queries', 'length_constraints__number_sentences_1381_queries', 'length_constraints__number_sentences_2266_queries', 'length_constraints__number_sentences_1268_queries', 'length_constraints__number_sentences_179_queries', 'length_constraints__number_paragraphs_1236_queries', 'length_constraints__number_paragraphs_2941_queries', 'length_constraints__number_paragraphs_1248_queries', 'length_constraints__number_paragraphs_1858_queries', 'length_constraints__number_paragraphs_1377_queries', 'length_constraints__number_paragraphs_2357_queries', 'length_constraints__number_paragraphs_2921_queries', 'length_constraints__number_paragraphs_1082_queries', 'length_constraints__number_paragraphs_2467_queries', 'combination__two_responses_1591_queries', 'combination__two_responses_1793_queries', 'combination__two_responses_2912_queries', 'combination__two_responses_1332_queries', 'combination__two_responses_2383_queries', 'combination__two_responses_136_queries', 'combination__two_responses_1098_queries', 'combination__two_responses_1746_queries', 'combination__two_responses_247_queries', 'combination__two_responses_2918_queries', 'detectable_content__postscript_2273_queries', 'detectable_content__postscript_2070_queries', 'detectable_content__postscript_1800_queries', 'detectable_content__postscript_1305_queries', 'detectable_content__postscript_1759_queries', 'detectable_content__postscript_1367_queries', 'detectable_content__postscript_1537_queries', 'detectable_content__postscript_1879_queries', 'detectable_content__postscript_1246_queries', 'detectable_content__postscript_1620_queries', 'startend__end_checker_2398_queries', 'startend__end_checker_1902_queries', 'startend__end_checker_2268_queries', 'startend__end_checker_1659_queries', 'startend__end_checker_1893_queries', 'startend__end_checker_2475_queries', 'startend__end_checker_1128_queries', 'startend__end_checker_1939_queries', 'startend__end_checker_1446_queries', 'startend__end_checker_1220_queries', 'detectable_content__number_placeholders_3280_queries', 'detectable_content__number_placeholders_1372_queries', 'detectable_content__number_placeholders_3221_queries', 'detectable_content__number_placeholders_1927_queries', 'detectable_content__number_placeholders_3126_queries', 'detectable_content__number_placeholders_2164_queries', 'detectable_content__number_placeholders_2136_queries', 'detectable_content__number_placeholders_2304_queries', 'detectable_content__number_placeholders_3743_queries', 'length_constraints__number_words_2323_queries', 'length_constraints__number_words_1072_queries', 'length_constraints__number_words_1258_queries', 'length_constraints__number_words_1251_queries', 'length_constraints__number_words_164_queries', 'detectable_format__number_highlighted_sections_168_queries', 'detectable_format__number_highlighted_sections_1237_queries', 'detectable_format__number_highlighted_sections_1601_queries', 'detectable_format__number_highlighted_sections_167_queries', 'detectable_format__number_highlighted_sections_1773_queries', 'detectable_format__number_highlighted_sections_1646_queries', 'detectable_format__number_highlighted_sections_1379_queries', 'detectable_format__number_highlighted_sections_1307_queries', 'detectable_format__number_highlighted_sections_1886_queries', 'detectable_format__number_highlighted_sections_1644_queries', 'detectable_format__json_format_1094_queries', 'detectable_format__json_format_1148_queries', 'detectable_format__json_format_1137_queries', 'detectable_format__json_format_1075_queries', 'detectable_format__json_format_2857_queries', 'detectable_format__json_format_3223_queries', 'detectable_format__json_format_2404_queries', 'detectable_format__json_format_321_queries', 'detectable_format__json_format_13_queries', 'change_case__capital_word_frequency_2820_queries', 'change_case__capital_word_frequency_2849_queries', 'change_case__capital_word_frequency_2870_queries', 'change_case__capital_word_frequency_1592_queries', 'detectable_format__multiple_sections_2023_queries', 'detectable_format__multiple_sections_1548_queries', 'detectable_format__multiple_sections_2925_queries', 'detectable_format__multiple_sections_1131_queries', 'detectable_format__multiple_sections_357_queries', 'startend__quotation_2015_queries', 'startend__quotation_219_queries', 'startend__quotation_2010_queries', 'startend__quotation_1658_queries', 'startend__quotation_1325_queries', 'startend__quotation_1776_queries', 'startend__quotation_2239_queries', 'startend__quotation_1845_queries', 'startend__quotation_2209_queries', 'length_constraints__nth_paragraph_first_word_2880_queries', 'length_constraints__nth_paragraph_first_word_181_queries', 'length_constraints__nth_paragraph_first_word_2250_queries', 'length_constraints__nth_paragraph_first_word_2215_queries', 'length_constraints__nth_paragraph_first_word_3073_queries', 'length_constraints__nth_paragraph_first_word_2590_queries', 'length_constraints__nth_paragraph_first_word_3624_queries', 'length_constraints__nth_paragraph_first_word_1954_queries', 'detectable_format__title_1262_queries', 'detectable_format__title_2229_queries', 'detectable_format__title_295_queries', 'detectable_format__title_2097_queries', 'detectable_format__title_1802_queries', 'detectable_format__title_1322_queries', 'detectable_format__title_2969_queries', 'detectable_format__title_3057_queries', 'detectable_format__title_1551_queries', 'detectable_format__title_2807_queries']}

Conversation

sunnweiwei commented Mar 14, 2025

Code Quality

Documentation

Testing

Adding datasets checklist

Adding a model checklist

Uh oh!

Uh oh!

sunnweiwei commented Mar 14, 2025

Uh oh!

Uh oh!

Samoed Mar 15, 2025

Choose a reason for hiding this comment

Uh oh!

KennethEnevoldsen commented Jun 15, 2025

Uh oh!

Muennighoff commented Aug 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants