Refresh after submitting model issues

[raghavlite]

I uploaded my model here. Added the mteb_metadata.md to the top of readme.md

Here is my model: https://huggingface.co/raghavlight/SE_v1/

I don't see the model on leaderboard after I hit refresh. Am I doing something wrong here?

Here is what I see after I hit refresh.

[KennethEnevoldsen]

@Muennighoff do you have time to have a look at what is going wrong here?

[Muennighoff]

It looks like the Refresh button is broken. It completely changes the ranking and the composition of the Average (56 -> 27). I'm not sure what is happening but maybe @tomaarsen knows? I think we were discussing the Refresh button at some point 🤔

[KennethEnevoldsen]

@Muennighoff or @tomaarsen this problem seems to be preventing people from submitting to the leaderboard (#857) - what is the best approach to take a look at that?

[tomaarsen]

@Muennighoff We were indeed discussing the refresh button a while ago (back then the issue was that it caused everyone's leaderboard to display error messages in the top right), but I haven't made any changes to it. I suspect that the caching update perhaps broke it.
That said, models can still be submitted, correct? It's just hard/impossible to verify whether they actually work until an admin restarts the leaderboard?

Edit: For me the Refresh button just loads for 10+ minutes (so far)

• Tom Aarsen

[Muennighoff]

That said, models can still be submitted, correct? It's just hard/impossible to verify whether they actually work until an admin restarts the leaderboard?

Yes exactly!

[tomaarsen]

I was able to dedicate a bit of time this morning and was able to fix it: https://huggingface.co/spaces/mteb/leaderboard/discussions/122
It was caused by a common but very devious bug.

• Tom Aarsen

[KennethEnevoldsen]

Wonderful @tomaarsen - @raghavlite I assume this resolved the issue?

[tomaarsen]

Once the PR is merged, then it should be resolved. I haven't merged it (nor my 2nd PR regarding dim/seq/size of gated models) just in case one of the maintainers would like to have a quick look at the PRs on HF first.

• Tom Aarsen

[KennethEnevoldsen]

As it is merged, I will close this issue but feel free to re-open it if issues arise.