Submissions stuck on different statuses - Hebrew Semantic Retrieval National Challenge

[team701]

Hi,
We are noticing an increasing number of submissions getting stuck on certain statuses (preparing, running, scoring) in our challenge:
https://www.codabench.org/competitions/9950/

Sometimes resubmitting the same zip file results in the new submission going through (while the old one remains stuck).
When we looked at a submission that was stuck in the "Scoring" status we noticed its predictions json file was valid, and as such it's expected to get a score, but in the output directory we also found the submitted model files (not just the predictions.json as our code dictates).

This means that every submission going through the pipeline is undergoing packaging again before being written to the output directory.
As our challenge involves relatively large file submissions (up to 7GB), we suspect this takes up too much of the submission runtime and causes unnecessary traffic issues.

Is there a way to control the contents of the output directory besides the ingestion code? We'd like to only get the "predictions.json" and the metadata files produced by the competitors, without their entire submitted model being there as well.

Thanks!

[Didayolo]

Hello,

The problem you are refering to is reported here:

• Compute worker - Avoid duplicating submission files #1874

Unfortunately there is no direct way to control this behavior.

If your competition involves code submissions, you could solve the problem with the following work-around, slightly tricky to setup but not that much (alternative solution at the end of this comment):

Step 1: Updating compute worker code

Clone Codabench repository and update the following file:
https://github.com/codalab/codabench/blob/develop/compute_worker/compute_worker.py

Comment out line 627:

#                shutil.copytree(program_dir, self.output_dir)

Step 2: Re-compile the compute worker and push it

Build the compute worker image to include the change (see wiki):

docker build -t your-username/competitions-v2-compute-worker:latest -f Dockerfile.compute_worker .

Then you can push this new version to your DockerHub account (cf DockerHub documentation if needed).

Step 3: Update your compute workers

Go inside each worker, stop it, and restart it using your new image.

You can do this by following the documentation: https://github.com/codalab/codabench/wiki/Compute-Worker-Management---Setup

While replacing the usual Docker image by your own Docker image.

---

This way, your own workers will avoid the duplication of the submission files.

Alternatively, you can also:

1. Enter inside each worker, and inside the compute_worker container
2. Update directly the compute_worker.py file to remove line 627

[team701]

Hi Didayolo,

Thank you so much for the detailed answer, we’ll try this workaround.

Regarding the traffic issues - we noticed we get low download speeds when downloading from the Codabench site (around 1MB sometimes even less than that).
Also, we are experiencing long download times in the compute_worker, evident in this log message: “... Getting bundle https://miniodis-rproxy.lisn.upsaclay.fr/.... “

Could this be the reason why large submissions get stuck in the preparing status?
Tell us if there’s something that we can do about this on our end.

Thanks again!

[Didayolo]

Hi,

That could be the reason indeed, if the downloading from our storage is slow and the files are big.
The servers are based in France, it may be distant from your compute workers.

About the work around mentionned, did you have success with it?

[team701]

We'll be happy to share with you our implementation to your suggested work-around. Using a simple Dockerfile to rebuild the image, while commnting out the line containing shutil.copytree in /compute_worker.py :

FROM codalab/competitions-v2-compute-worker:gpu1.3

USER root

# Comment out the line containing shutil.copytree in /compute_worker.py
RUN sed -i 's/shutil\.copytree(program_dir, self\.output_dir)/#&/' /compute_worker.py

# (Optional) print the modified line for verification
RUN sed -n '627p' /compute_worker.py

Thanks again for the work-around suggestion