Default competition docker image updated, some competitions may be affected

[Didayolo]

Hello,

We have recently updated the default competition docker image from codalab/codalab-legacy:py2 to codalab/codalab-legacy:py39 (see https://groups.google.com/g/codalab-competitions/c/ePhMJKpcDFM). This is the environment in which the submissions run if nothing is set by the organizers.

I merge here all issues related to this problem: #3262, #3260, #3261, #3255, #3257 and more.

Please send me the URLs of affected competitions so I can solve the issue. Thank you.

The fix

Problems of retro-compatibility can be solved by manually setting the competition docker image to codalab/codalab-legacy:py2. This can be done by the organizers or by an administrator of the platform, using the editor. See the screenshot below:

[Didayolo]

Change done for:

• https://codalab.lisn.upsaclay.fr/competitions/6066
• https://codalab.lisn.upsaclay.fr/competitions/4318
• https://codalab.lisn.upsaclay.fr/competitions/3282
• https://codalab.lisn.upsaclay.fr/competitions/7384
• https://codalab.lisn.upsaclay.fr/competitions/7383
• https://codalab.lisn.upsaclay.fr/competitions/7404
• https://codalab.lisn.upsaclay.fr/competitions/7308
• and more...

[TXH-mercury]

https://codalab.lisn.upsaclay.fr/competitions/7404#learn_the_details
mscoco image caption challenge
thanks

@Didayolo hi, could you help with this bug or should I contact coco team to fix this? thank you

[qianrenmao]

#3255
and the URL is：
https://codalab.lisn.upsaclay.fr/competitions/7308#learn_the_details
Fact Extraction and VERification (FEVER) Challenge
Thx!

[heartStrive]

HO3D_v2 website is having problems again：https://codalab.lisn.upsaclay.fr/competitions/4318

[Didayolo]

We have currently a power outage in the whole district. The service will be back tomorrow and I'll do my best to solve this issue.

[Didayolo]

HO3D_v2 website is having problems again：https://codalab.lisn.upsaclay.fr/competitions/4318

Thanks for the feedback. OK so I guess I need to investigate to find which tag is the old default image.

UPDATE: this is weird. The image I've set up was indeed the old default image.

[ady-yu]

HO3D_v2 website is having problems again：https://codalab.lisn.upsaclay.fr/competitions/4318

Can you open the website? I can't open it.

[Didayolo]

It was due to a power outage. It is back now.

[hxwork]

Hi,

FYI, The HO3D_v2 still has the above-mentioned problem, as shown below.

[Didayolo]

Hi,

Indeed. I try to understand what is going on: the Docker image set up is the right one (codalab/codalab-legacy:py2), but the compute worker run the new one (codalab/codalab-legacy:latest). I have never seen this behavior before. I am currently investigating it.

Bad theory above but I'm working on it and will keep you informed.

[hxwork]

Got it. Many thanks! Looking forward to your update.

[Didayolo]

Wow I think I understand what happened.

The problem was indeed solved by reverting to the old docker image, but on your competition the organizers had upgraded their scoring program to match the new version of the docker image. So when I "fixed" it, I actually broke it because it was already solved in the meantime.

Hopefully it should work now, please give me feedback @hxwork

[Didayolo]

Just a remark, still about HO3D_v2.

With the new scoring program and new competition docker image, it seems that the F1 score announced in the scoring logs are always 0.

I don't want to touch the choice of scoring program for the organizers. If the organizers want to come back to last week behavior, they just need to select back the old scoring program and to put codalab/codalab-legacy:py2 in the field "competition docker image" using the editor.

If the new behavior if fine, then everything is OK.

[hxwork]

Hi @Didayolo,

I have tested the online evaluation and it works now. However, as you said that the F scores are always 0. Hope the organizers can fix this later.

[qianrenmao]

#3255

Hi @Didayolo

I submitted it again, and the results can be given normally. But it seems that the test took a long time.

Thx a lot!

[heartStrive]

Yes, I can open the website normally.

[Didayolo]

@hxwork

I have tested the online evaluation and it works now. However, as you said that the F scores are always 0. Hope the organizers can fix this later.

Yeah... I don't know which scoring program was used before so I can't make that change by myself... I think the best thing to do is to contact the organizers. They can simply put the old scoring program and codalab-legacy:py2 docker and everything should work as earlier.

[danlou]

Hi @Didayolo,

Looks like our competition may have been impacted too, could you please confirm and fix if needed?
https://codalab.lisn.upsaclay.fr/competitions/5360

Thanks

[Didayolo]

@danlou

Indeed, as the "competition Docker image" field is blank in the editor, your competition is affected. The fix is simply to fill it with codalab/codalab-legacy:py2.

I've done it already, everything should be good.

[danlou]

Thanks @Didayolo. I tried again this morning and it was still throwing the same error, so I tried uploading a new version of the scoring script and printing sys.version I see "3.9.12 (main, Apr 5 2022, 06:56:58)", not Python 2.

In any case I also pushed a change to the way I was writing to the output file, now using mode 'w' instead of 'wb' and that appears to have fixed the original error, so probably best to leave as is.

Submissions runs appear to be taking much longer than usual though, hopefully that's just temporary.

[Didayolo]

I see "3.9.12 (main, Apr 5 2022, 06:56:58)", not Python 2.

This is not normal.

@danlou Quick question, for testing this morning did you re-upload submissions or re-run them using the "submissions" interface?

When I try to re-upload one of your submission (not click on re-run), I can see this in the scoring logs:

2.7.13 |Anaconda 2.5.0 (64-bit)| (default, Dec 20 2016, 23:09:15) 

Fixing your scoring program to use Python 3 is not a bad idea indeed as Python 2 is deprecated. If you want to run on Python 3, leave blank the field "competition docker image" in the editor.

[danlou]

Ah, that's it then. I was re-running submissions from the Submissions page.
I've now tried uploading predictions again and see "2.7.13 |Anaconda 2.5.0 (64-bit)| (default, Dec 20 2016, 23:09:15) ".

Thanks for clarifying!

[ruoyueqi]

Hi @Didayolo,

Looks like the competition may have been impacted too, could you please confirm and fix it?
https://codalab.lisn.upsaclay.fr/competitions/3783

Thanks

problem as shown below
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
File "/tmp/codalab/tmp1ynlmU/run/program/score.py", line 494
global args
^
SyntaxError: name 'args' is used prior to global declaration

[Foltt]

Hi @Didayolo ,
It seems that the competition may have also been impacted, could you please confirm and fix it?
https://codalab.lisn.upsaclay.fr/competitions/395
Thank you

[Didayolo]

@ruoyueqi and @Foltt

It is done.

[EVEBios]

Hi @Didayolo, this competition seems to have some problems, too. Could you please take a look? Thanks.

https://codalab.lisn.upsaclay.fr/competitions/7403#participate-submit_results

[Foltt]

@ruoyueqi and @Foltt

It is done.

It works! Thanks!

[Didayolo]

@EVEBios

Done!

[Didayolo]

I close this issue for now. As mentioned in the first message, the fix (if needed) is easy to perform as a challenge organizer.

[cghAndy]

Hi @Didayolo , this competition seems to have the same problems. Could you please take a look? Thanks.
https://codalab.lisn.upsaclay.fr/competitions/7107

[Didayolo]

This competition is indeed using the default image.

If you are the competition organizer and you get similar errors, then you should change the Docker image:

1. Click on "EDIT"
2. Scroll to the field "Competition Docker image"
3. Put codalab/codalab-legacy:py2 in the field
4. Save the changes (at the bottom of the editor)

Then you can re-submit to see if the problem is solved. Do not use the "re-run" option from the submissions panel, because it re-runs submissions using their original Docker image.

I hope this helps.