Pulling images not always works as expected

[GitHK]

I have a new machine and just published s4l. Pulling the service failed 2/3 times with the following error.

WARNING: [2023-01-24 13:59:30,813/MainProcess] [servicelib.long_running_tasks._task:get_task_result_old(257)]  -  Task simcore_service_dynamic_sidecar.modules.long_running_tasks.task_create_service_containers.68c5562c-a737-417a-8cb5-449637b8e784 finished with error: Task simcore_service_dynamic_sidecar.modules.long_running_tasks.task_create_service_containers.68c5562c-a737-417a-8cb5-449637b8e784 finished with exception: ''93dac263047b''
  File "/home/scu/.venv/lib/python3.9/site-packages/servicelib/long_running_tasks/_task.py", line 414, in _progress_task
    return await handler(progress, **task_kwargs)
dy-sidecar_51fb7bc0-6141-46e9-a873-f0be3f6aacea.1.w53j60bwxpkk@testmachine2    |
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/modules/long_running_tasks.py", line 119, in task_create_service_containers
    await docker_compose_pull(app, shared_store.compose_spec)
dy-sidecar_51fb7bc0-6141-46e9-a873-f0be3f6aacea.1.w53j60bwxpkk@testmachine2    |
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/core/docker_compose_utils.py", line 110, in docker_compose_pull
    await pull_images(list_of_images, registry_settings, _progress_cb, _log_cb)
dy-sidecar_51fb7bc0-6141-46e9-a873-f0be3f6aacea.1.w53j60bwxpkk@testmachine2    |
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/core/docker_utils.py", line 82, in pull_images
    await asyncio.gather(
dy-sidecar_51fb7bc0-6141-46e9-a873-f0be3f6aacea.1.w53j60bwxpkk@testmachine2    |
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/core/docker_utils.py", line 191, in _pull_image_with_progress
    if _parse_docker_pull_progress(
dy-sidecar_51fb7bc0-6141-46e9-a873-f0be3f6aacea.1.w53j60bwxpkk@testmachine2    |
  File "/home/scu/.venv/lib/python3.9/site-packages/simcore_service_dynamic_sidecar/core/docker_utils.py", line 129, in _parse_docker_pull_progress
    _, layer_total_size = all_image_pulling_data[image_name][layer_id]

Feels like a concurrency issue to me, but I might be wrong. I see you are sharing an object.

• improving dynamic-sidecar design osparc-issues#638