Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Use model revision in results folder #842

Merged
merged 13 commits into from
Jun 1, 2024

Conversation

isaac-chung
Copy link
Collaborator

@isaac-chung isaac-chung commented May 29, 2024

Reason for dataset addition: Addresses #618

Changes

New results structure would incorporate model revisions like so:

results
|-- model name
  |-- model revision
    |-- datasetA.json

If the model revision cannot be found in the model (e.g. not loaded from huggingface), then it needs to be supplied to the run method or via the command line arg --model_revision.

Test

Added a test run result with:

mteb -m intfloat/multilingual-e5-base -t BlurbsClusteringS2S.v2

where the latest revision of the model is d13f1b27baf31030b7fd040960d60d909913633f.

@isaac-chung isaac-chung changed the title use model revision in results folder [WIP] Use model revision in results folder May 29, 2024
@isaac-chung isaac-chung marked this pull request as draft May 29, 2024 12:54
Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to integrate it with #807 as well. I believe we can now leave get_model_meta() to extract metadata for an arbitrary sentence trf model.

mteb/evaluation/MTEB.py Outdated Show resolved Hide resolved
mteb/cmd.py Show resolved Hide resolved
mteb/cmd.py Show resolved Hide resolved
@isaac-chung isaac-chung marked this pull request as ready for review May 30, 2024 07:47
mteb/evaluation/MTEB.py Outdated Show resolved Hide resolved
@isaac-chung isaac-chung changed the title [WIP] Use model revision in results folder fix: Use model revision in results folder Jun 1, 2024
@isaac-chung
Copy link
Collaborator Author

Added points as well.

@isaac-chung isaac-chung merged commit 2c6065b into main Jun 1, 2024
7 checks passed
@isaac-chung isaac-chung deleted the use-model-revision-in-results-structure branch June 1, 2024 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants