Paper writing

[Muennighoff]

Should we already create an Overleaf & get started on writing? I have Overleaf premium so created one here: https://www.overleaf.com/8693731561tpntkddrhngs#41bcff (but if someone else wants to own it, feel free to recreate)
It may make sense to ask every contributor to introduce their dataset in a paragraph in the Appendix?

NeurIPS D&B would be great I think but DDL is in 1 month - do you think we can make it @KennethEnevoldsen @imenelydiaker?

[imenelydiaker]

Thanks for creating the Overleaf, we should definitely start writing!

For the conferences, there is also the next ARR cycle for *CL conferences, this one would give us access to EMNLP, the deadline is June 15th.

The deadline for end of contributions is May 15th, I think we're still missing some languages. We can post about the missing languages and change the deadline for end of contributions, either way I think we need to start writing now else we'll be too short for both deadlines.

[KennethEnevoldsen]

Sounds like a good idea.

We should probably create the outline for the structure.

Here is my proposed structure:

Abstract

• primary contributions:
  • notable speedups of MTEB
  • coverage of more than X languages while maintaining evaluation speed
  • broader coverage of domains, languages, code and tasks

Introduction

• Limitations of MTEB
• Contributions

Introduction/related work

• speeding up benchmarks
• large collaborative effort in NLP

Methods

Methods/ensuring dataset quality

• metadata (non-translated datasets), non-trivial performance

Methods/constructing benchmarks

• Introduce benchmarks
  • We define a set of benchmarks, and introduce the notation of these:
    • Monolingual benchmarks: MMTEB(eng), MMTEB(fra), etc.
    • Code benchmarks: MMTEB(code)
    • Multilingual benchmarks: MMTEB(multilingual), MMTEB(indo-euro), etc.
• We might also discuss how we select tasks + aggregate scores

Methods/Speeding up the benchmark:

• to make the benchmark more accessible to low resource communities
• to encourage broader evaluation across languages

Results

• We select N relevant benchmarks (mteb(eng), mteb(multilingual), maybe some domains?)

Discussion:

• brief analysis of performance

Discussion/limitations of existing models

• Is there places where we see existing models fail? Language, domains, tasks?

Limitations:

• For large collaborative benchmarks: Continually updating the benchmark invalidated previous model results

Let me know what you guys think?

[imenelydiaker]

I took some liberties and organized the overleaf to facilitate the writing. I copied the outline you suggested here 🙂

[KennethEnevoldsen]

I have also created an issue #896 to get an overview of all the segment that needs to be written

[KennethEnevoldsen]

@Muennighoff, @imenelydiaker I have sent an email to you guys to set up a meeting to discuss this and the next stages of mmteb

[imenelydiaker]

EMNLP 2024 deadline (abstract + paper): June 15, 2024 11:59pm AOT (Wesbiste)
Template: https://github.com/acl-org/acl-style-files

[Muennighoff]

After discussion we realized EMNLP probably makes most sense so swapped to the template: https://www.overleaf.com/8693731561tpntkddrhngs#41bcff Also turned on track changes :)

[imenelydiaker]

TACL template in case we don't make it for EMNLP: https://www.overleaf.com/project/61c0608457aa0222e6ced44a

[isaac-chung]

Was talking to @imenelydiaker about the TACL submission guideline and realized that:

The submission is not a version of a paper that was reviewed by the ACL, AACL, EACL, NAACL, or EMNLP conferences within the last 9 months. Note that papers that were withdrawn before receiving a decision but nonetheless received reviews are thus not TACL-eligible during the 9-month window.

So if we submit to EMNLP/ACL in June, we won't be able to submit to TACL for 9 months from the review date.

[imenelydiaker]

I would appreciate some help on the related work part, I'm missing some references for these points:

1. Speeding up benchmarks: Any papers you know of can help on this?
2. Open-source large effort: Do you have any papers related to this? I've seen SeaCrowd, although I haven't found any paper linked to it. I know that OpenDevin is also a community effort project, although there's no paper for the moment, should we talk about it?

Should we include any other works in this part? References to some included datasets?

[KennethEnevoldsen]

Hi @dokato, we would love a helping hand. You might start out taking a look at the related works section as well as the introduction.

Also if there is segment that you feel should be added which is not currently considered in the paper.

[KranthiGV]

Happy to help with writing as well. Any sections you want me to start contributing to?

[KennethEnevoldsen]

@KranthiGV if you can create a script to add all the citations to a .bib file and create a table overview of all the tasks with the references (you might take inspiration from https://openreview.net/pdf/f5f1953a9c798ec61bb050e62bc7a94037fd4fab.pdf, potentially making it more dense, e.g. exclude description).

This is to ensure that we sight all the relevant dataset authors

[isaac-chung]

The task overview table already exists as tasks.md. Adding all bibtex from tasks weeks be great. Backfilling missing bibtex would be even better!

[KennethEnevoldsen]

Ahh right yes adding the old bibtex would also be great. Though, there is still some work in converting it to latex such that the dataset authors are given a citation.

[KennethEnevoldsen]

I have added the dataset construction section (an outline) as well in the appendix. Please refer relevant authors to fill out this section. Tagging a few people I know have something to add:

• @x-tabdeveloping multilevel clustering and multi clf
• @imenelydiaker cross-lingual tasks
• @Rysias wiki clustering construction (as an example)
• @orionw: instruction retrieval and reranking
• @Muennighoff general
• @awinml would love if you could write a bit about the conversion of legalbench as well

[x-tabdeveloping]

@KennethEnevoldsen Do we want a section explaining the methodology involved in the new tasks? Like how we did multilabel classification and hierarchical clustering?

[KennethEnevoldsen]

I have already written a general introduction to some of the speed-ups (please take a look at it), this goes into the hierarchical clustering as an example. However I agree that we should add a section on new task types

[isaac-chung]

Would you prefer if ClusteringFast to be part of this appendix as well? (B)

[KennethEnevoldsen]

@isaac-chung we have a section where we describe all the tasks, I would describe clustering there (probably just clusteringfast)

[isaac-chung]

Got it, thanks. From the structure now, I'll add that to B.2

[y12uc231]

hi all, I would love to contribute with the writing, is there any section that require some more hands?

[Sakshamrzt]

@KennethEnevoldsen If this is still open I can take this up.

[KennethEnevoldsen]

I believe it is

[KennethEnevoldsen]

@Sakshamrzt I have added you to an issue here: #897

[Sakshamrzt]

@KennethEnevoldsen can you tell how was the current custom.bib file built?

[KennethEnevoldsen]

I believe it is copy and paste (each contributor just copied their required bibtex in)

[mariyahendriksen]

Hi everyone,

I plan to spend some time working on the benchmark this weekend, adding additional datasets and I would also like to contribute to the paper writing. Is there anything specific you would like me to focus on for the paper draft? What would be most helpful?

Thanks!

[mariyahendriksen]

@KennethEnevoldsen sure, I will look into it tonight, will keep you posted!

[mariyahendriksen]

i split the appendix into two parts as we discussed.
Anything else I should focus on?

[KennethEnevoldsen]

Thanks @mariyahendriksen. I have made a few changes:

1. Converted C to B.3 for new datasets (and C is now an overview table)
2. started differentiating B1 and B2 such that B1 explains the task and B2 explains task construction. I'd like to ask you to finalize this if you have the time (I have done this for some of the tasks already). Feel free to leave the subsection empty if you are not sure how to fill it out.

[mariyahendriksen]

@KennethEnevoldsen I have gone over Appendix B one more time, some parts are missing but I see that they were already assigned to someone else so I will leave them as they are for now. If no one picks them up in the upcoming days, I am happy to fill-in the gaps myself.
Regardless, I will keep an eye on the Appendix to make sure all the parts are aligned.

What should focus on now?

[KennethEnevoldsen]

@mariyahendriksen I don't have any section atm. will take a look through the paper today - will let you know if something comes up

[asparius]

There seems to be a typo, it is M3TEB in the paper but M2TEB in abstract. Anything else I can help with paper writing or any other issue?

[KennethEnevoldsen]

@asparius I believe we have agreed on using MMTEB going forward (thought M2TEB and M3TEB were discussed)

[jaygala24]

@KennethEnevoldsen I would also like to help with the paper writing. Let me know where can I be of help. Thank you!

[KennethEnevoldsen]

@jaygala24 I have created an overview issue here #896. You could take a look at the related works section or limitations

[mariyahendriksen]

Did we miss the deadline? What’s our current target date? The next ACL ARR submission window is in August; should we aim for that? Alternatively, we could also consider TMLR with a self-imposed deadline.

[KennethEnevoldsen]

@mariyahendriksen I believe the next deadline is a preprint on arxiv this summer (which leaves us open to quite a few venues afterwards).

[mariyahendriksen]

ok, sounds good! do you happen to know the exact date of the arxiv deadline? knowing this would really help planning

[KennethEnevoldsen]

I do not - the current plan is to get us ready to run models as soon as possible (my hope is to get that started this month), then it would take about ~14 days to run the models. Then, about ~14 days for writing after that, which leaves us at approximately the start of August.

I will take it up with the steering group though to see if people agree.

[imenelydiaker]

I believe TACL is more adapted to our work: https://transacl.org/index.php/tacl, it's like TMLR but for NLP.