[Bug]: `pullImage error` when the image is actually pulled successfully

[jvstme]

Steps to reproduce

Run a job on a VM-based backend that will require pulling a Docker image. For example, use a custom Docker image that is not yet pulled on the instance.

The issue happens randomly, you may need to try several times to reproduce.

Actual behaviour

The job fails:

 NAME     BACKEND              INSTANCE    RESOURCES                          RESERVATION  PRICE  STATUS  SUBMITTED  ERROR                      
 httpbin  datacrunch (FIN-01)  CPU.4V.16G  4xCPU, 16GB, 100.0GB (disk), SPOT               $0.01  failed  10:15      JOB_FAILED                 
                                                                                                                     (CREATING_CONTAINER_ERROR) 

httpbin provisioning completed (failed)
Run failed with error code CREATING_CONTAINER_ERROR.
Error: pullImage error: failed pulling kennethreitz/httpbin:latest: downloaded 211188154/214167220 bytes (21.26MB/s), errors: []
Check CLI, server, and run logs for more details.

shim.log:

time=2025-04-11T08:15:30.303305Z level=debug msg=Pulling image
time=2025-04-11T08:15:39.773158Z level=debug msg=Status: Downloaded newer image for kennethreitz/httpbin:latest
time=2025-04-11T08:15:39.779208Z level=error msg=pullImage error: failed pulling kennethreitz/httpbin:latest: downloaded 211188154/214167220 bytes (21.26MB/s), errors: []
time=2025-04-11T08:15:39.780045Z level=debug msg=released GPU(s) gpus=[] task=f14ca241-2bfa-4c0d-af09-10b2af0d3912
time=2025-04-11T08:15:39.780085Z level=error msg=failed to run task=f14ca241-2bfa-4c0d-af09-10b2af0d3912 err=failed pulling kennethreitz/httpbin:latest: downloaded 211188154/214167220 bytes (21.26MB/s), errors: []

The image is actually pulled successfully to the runner instance

# docker image ls
REPOSITORY             TAG       IMAGE ID       CREATED       SIZE
kennethreitz/httpbin   latest    b138b9264903   6 years ago   534MB

Expected behaviour

If the image is pulled successfully, shim does not fail with pullImage error and proceeds to start the runner.

dstack version

0.19.2

Server logs

Additional information

No response

[r4victor]

Is it because of downloaded 211188154/214167220 bytes? Any idea why the number of bytes does not match?

[jvstme]

Is it because of downloaded 211188154/214167220 bytes?

Yes.

Any idea why the number of bytes does not match?

No, I haven't investigated yet.

[ASmedberg-woolpert]

I'm hitting this too. Looks like the same error:
{"timestamp": "2025-04-24 18:00:35,903", "logger": "dstack._internal.server.background.tasks.process_running_jobs", "level": "WARNING", "message": "shim failed to execute job p1-mn10-0-0: creating_container_error (pullImage error: failed pulling ********.dkr.ecr.us-east-2.amazonaws.com/********: downloaded 5266659108/5268734644 bytes (48.24MB/s), errors: [])"}

I'm not super familiar with the intricacies of docker or dstack, so not sure what other information I could provide that would help. I did have dstack in debug print mode so I have the logs leading up to this if those would help.

Edit adding context:
Also running Dstack 0.19.2

This was on a newly created instance so a fresh download of the image.

[jvstme]

@ASmedberg-woolpert, thanks for sharing, this looks like the same problem. I think we have enough details to investigate this, just need to find the time to finally get to it. Until the bug is fixed, restarting the run should help

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale. Please reopen the issue if it is still relevant.

[jvstme]

Reproduced again on AWS g4dn.xlarge in eu-west-3:

pullImage error: failed pulling dstackai/base:0.10-base-ubuntu22.04: downloaded 981212828/988627893 bytes (42.35MB/s), errors: []

[r4victor]

Easier to reproduce when running many runs: I often get 1-2 runs fail with this error when provisioning 100 runs (aws, t3.large).