Skip to content

1.12.4
Compare
Choose a tag to compare
@KennethEnevoldsen KennethEnevoldsen released this 28 May 07:41
· 297 commits to main since this release
1.12.4
dfc5941

1.12.4 (2024-05-28)

Fix

  • fix: add model meta to create reproducible workflow (#807)

  • replace get_tasks as default filtering.

The intention here is to:

  1. move complexity away from the MTEB object
  2. ensure that the filters are applied in the same way across the benchmark (currently MTEB filters slightly differently due to not handling the new language codes)
  3. deprecate filtering in MTEB going forward (only with a warning atm.)
  4. doing it in a two step fashion ensure that users are able to inspect the tasks before they are run (also allow for much more custom filtering on the user end)
  • add model meta to create reproducible workflow
  • Add outline for model meta object
  • Added a single model as a an example
  • Added test for the reproducible workflow

The intention is that a reproducible workflow should then look like:

# assuming the same mteb and sent. trf. version

model_meta = mteb.get_model(model_name)
task = mteb.get_task(task_name)

model = model_meta.load_model() # load model either using custom loader or sentence transformer (with revision)

eval = MTEB(tasks=[task])
eval.run(model, output_folder="tests/results", overwrite_results=True)

For running models we can the simply have tasks like:

  1. implement model
  2. ensures that it runs on all tasks types

Running the models then become simple:

eval = MTEB(tasks=mteb.get_tasks())
for mdl_name in models:
   model_meta = mteb.get_model(mdl_name)
   mdl = model_meta.load_model()
   eval.run(mteb.get_model(mdl)

We can start with this already now e.g. on classification tasks.

  • import ISO_LANGUAGE from languages

  • fix import

  • Apply suggestions from code review

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

  • format

  • Apply suggestions from code review

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

  • Updated based on suggestions from review

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (0319105)

  • fix: Updated CLI to use new task filter (#826)

  • replace get_tasks as default filtering.

The intention here is to:

  1. move complexity away from the MTEB object
  2. ensure that the filters are applied in the same way across the benchmark (currently MTEB filters slightly differently due to not handling the new language codes)
  3. deprecate filtering in MTEB going forward (only with a warning atm.)
  4. doing it in a two step fashion ensure that users are able to inspect the tasks before they are run (also allow for much more custom filtering on the user end)

  • tests passing

  • Added corrections from review

  • Updated CLI

  • docs: Added points

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (fb5fec8)

Unknown

  • Update points table (f926216)

  • Update tasks table (d560c31)

  • Update points table (84e6856)

  • Add MLQuestions dataset (#799)

  • mlquestions load script

  • more metadata

  • add to init

  • baseline model results

  • add points

  • complete metadata

  • lint

  • Update points and metadata

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • clarification of period in comments

  • minor fix

  • linting

  • Fix validation error

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (3a14885)

Assets 4
Loading