Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problem when fetching an image from aws ecr docker registry #1257

Closed
eliekoren opened this issue Jun 6, 2024 · 12 comments
Closed

Problem when fetching an image from aws ecr docker registry #1257

eliekoren opened this issue Jun 6, 2024 · 12 comments

Comments

@eliekoren
Copy link

Hi
After pointing SCANCODEIO_SKOPEO_AUTHFILE_LOCATION to a valid docker config file (contains the right auth)
It fails to fetch

[CalledProcessError(1, ('/opt/scancodeio/venv/lib/python3.11/site-packages/fetchcode_container/bin/skopeo', 'inspect', '--insecure-policy', '--raw', '--authfile /etc/scancodeio/config.json', 'docker://12345678.dkr.ecr.eu-west-1.amazonaws.com/devops/my_image'))]

Traceback:
  File "/opt/scancodeio/scanpipe/pipelines/__init__.py", line 191, in execute
    step(self)
  File "/opt/scancodeio/scanpipe/pipelines/__init__.py", line 231, in download_missing_inputs
    raise InputFileError(errors)

What am I missing here?
@tdruez
Copy link
Member

tdruez commented Jun 6, 2024

@eliekoren You can try to get into the container and manually run the command, we may get more insight into the problem:

docker compose run --rm  web bash
/opt/scancodeio/venv/lib/python3.11/site-packages/fetchcode_container/bin/skopeo inspect --insecure-policy --raw --authfile /etc/scancodeio/config.json docker://12345678.dkr.ecr.eu-west-1.amazonaws.com/devops/my_image

Also, it seems that you are not using the latest ScanCode.io image, as I can see python3.11 in your output where the latest image is now based on Python 3.12. This may means that your version does not include all the features related to SCANCODEIO_SKOPEO_AUTHFILE_LOCATION

@eli-koren
Copy link

eli-koren commented Jun 13, 2024 •
edited
Loading

@tdruez
took the latest
in docker compose run --rm web bash
Run /opt/scancodeio/venv/lib/python3.12/site-packages/fetchcode_container/bin/skopeo inspect --insecure-policy --raw --authfile /etc/scancodeio/config.json docker://12345678.dkr.ecr.eu-west-1.amazonaws.com/devops/my_image

got docker inspect output correctly but still see the same error in the UI

@tdruez
Copy link
Member

tdruez commented Jun 13, 2024

got docker inspect output correctly but still see the same error in the UI

I'm not sure that I understand the current state of your project.
The suggested docker compose run --rm web bash was for debugging the ability to fetch the docker image input.

Could you clarify if you used the UI to try again to run the pipeline on a new fresh Project?

@eli-koren
Copy link

Your debug suggestion (running the command from the web container) didn't find any issue and got the inspect output as expected.
Created a new project and retried the same with no luck, same error.

@tdruez
Copy link
Member

tdruez commented Jun 13, 2024

@eli-koren Could you provide the error log of the new project?

Your debug suggestion (running the command from the web container) didn't find any issue and got the inspect output as expected.

Could you also provide the output of that command? (No need to include the private information)

Lastly, could you run a docker log and see if there's any output that includes valuable information about the error?

docker compose logs web

We need more context to help you figure out what's going on ;)

@eli-koren
Copy link

eli-koren commented Jun 13, 2024 •
edited by tdruez
Loading

@tdruez

docker exec -it scancodeio-web-1 bash
app@4b87ed3e61ef:~$ /opt/scancodeio/venv/lib/python3.12/site-packages/fetchcode_container/bin/skopeo inspect --insecure-policy --raw --authfile /etc/scancodeio/auth.json docker://xxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/dev/my_image
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 56542,
      "digest": "sha256:03a93f0ed51105413264d80e0c02e918ffdfd725869f50ca8db43d9e9c4ac93f"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 71916295,
         "digest": "sha256:0049b869cecbf5a63c6c44309977fe51f061082d985e3c35ac186455e302b237"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 15369,
         "digest": "sha256:f2a1631680b58459514aeb1bd9fdb851d40d93f9569d009a813eebe04d18e1d2"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 646001461,
         "digest": "sha256:284d079c97fdaac34f8a6bef27df5c430a0d0a6745b7af0788581229a65f2e4c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 251,
         "digest": "sha256:ef0d3e9ac6dd0d46c599642d719e093be7dab70573c26fcc2d7101a03a384af6"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 249,
         "digest": "sha256:12638f3134a594cec6369d0df1bf17f85c0d7ed3971bfbccfa25b9726fd9ac0d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 6464,
         "digest": "sha256:1c6d424adf0fe6dc2c09c32f077965cca26e8b76b5cfbd34902eb9ad778e582f"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 39824715,
         "digest": "sha256:f08d775145feb19d189d26cee60e3eb34498a56feb94eb127cb04d6cc9e67d20"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 420,
         "digest": "sha256:debe506072fc15aca764425a772fe0c4e690f28c0a9f9860e29b262334b56617"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 361,
         "digest": "sha256:d960066cc0f0210d0c3d83be2ba49ab28845f6b70eb8011ba9569a6229f576aa"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 164,
         "digest": "sha256:11888bf62c583a46c8263521355ebd3d04cc7bd53708b6f4ef58368d18c9fcff"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2092,
         "digest": "sha256:2fcc0541cbbfbd8b0ed55e8275218285dfe3fd715cabeb4b7fd496cb9ff21662"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2091,
         "digest": "sha256:871b834a6f82c759b352315d78109db268f8e07f48ae9cb0b273e4556561dce3"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2090,
         "digest": "sha256:09fa9dadc2b8a20bfbfc4789a675503e1d8b6b171ce82880dc06df0977b2ad16"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 3095,
         "digest": "sha256:4e425913a320504e0be79a868b546d64a08010836ad73b7b51aeeb91cde4b177"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 162,
         "digest": "sha256:be54aa1d062ca7f0ae9cf199ce18bf9308ed3c4e18175543387b817a8c4ef8c7"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 723,
         "digest": "sha256:bcdfec8ea0a45ace2917cb828551828582a3c6c777763baf018872057ad29676"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 495,
         "digest": "sha256:0d1a605352f498fe4af73f372aa24b544e7bfee9870a689c3244ebe7beb77df3"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 696,
         "digest": "sha256:f9efa742f7e648cd69f385e9ede72dae172bb3e1ef2255a32e7b7058d182ab09"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1088,
         "digest": "sha256:978cf78632cfd7e00352f5a0ef25772f38be6e4bda8586c387ad3b61cb131e49"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 209784720,
         "digest": "sha256:06a2c76a04ba7ba18bfe6321642682c50bf7ffe840492e782496f524abb5d864"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 195,
         "digest": "sha256:700b9515e64cde1f54acd345f369ebe16c1da45970e615a6fba6e8f6d0a85f33"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 376,
         "digest": "sha256:74e6568e82498b4d56366a052ca89152a189c25263f53cb5342e26682d7c9100"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 31767,
         "digest": "sha256:fccf72aa678321e77baf125001cb5a195a8e59f4df2c42210b48cf721d511627"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 40794,
         "digest": "sha256:b2d44bc795ea9590b0fb7ac1117e0e477a45830e8bf7e9d6e53463a7eef58d56"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 189476,
         "digest": "sha256:426828a6bb709abb9e2e4d22ca2ac8980fdfef0be5820948a56f20c866373f66"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 127,
         "digest": "sha256:69fff96b8728b849fbadcae966c497214084dac7bd2e50ff000d24b81c1b8ab0"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 146,
         "digest": "sha256:c45dbb8d55678d1b2dd8b0195f8cb3bb6d77f8930f65143545b7f4f7c1ed3645"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 191,
         "digest": "sha256:30e8e8c7cff7262c2e2a2c97a4a250eb900ad27ccdf7f66e0e586ac1842216aa"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 26126185,
         "digest": "sha256:751c8655ebfe33e295c52d8eb26c519e6c6f3e7b849d492063b41c4422182fc7"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2043,
         "digest": "sha256:cf6975bee4c9eb4f4cd0a5a2d68c50b525dd82848e8fb587b49d54315d76ac0c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2639,
         "digest": "sha256:506f0b353090255a19dcf2c7279b03e1a4b77a475468a75be1756f5f4de2c658"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 5331,
         "digest": "sha256:56b3f5ee871a4ac3847e84505cbf576b040fdcf6cdb46200dd0fd9a9ec180da5"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 538616819,
         "digest": "sha256:ea9ad19f9f69d1d3e0c3858f827e38558fac75d4c9ea0ba90cf57b5925f21aa3"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 9954,
         "digest": "sha256:d5dab6304b7c25081cd399a56731aa64e5f5da9cbb0001d41888205fa1bd7d52"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 143618,
         "digest": "sha256:44ae40a8a188e2427a77166f30cc2057bee87599ddcad0d1a734edffbf87cd12"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1130,
         "digest": "sha256:4bc0a5adf260b41d52ca3409309acc0210ff80c9fd87ded82272a38f56f7d688"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 150,
         "digest": "sha256:7e6fb883008d201e2143d8e0c795285747a82ff92248fb967d48884df351e6e0"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 214408247,
         "digest": "sha256:1818aa2707a4de32db3d7d38426cab0815039479ca31f928e3a828f884e3f4ee"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 18751,
         "digest": "sha256:87d5125bb5373dc224ef021f4cc8f03d582a53e054ac80d8685000ca21455fa2"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 406,
         "digest": "sha256:72e6916081812448d179697f53bfc3656c84b0a21af3bd2d06610e438b0d7a80"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 656580,
         "digest": "sha256:694eaf3f0f8af5e78f130a8fa9df37fa14b240d5bae817f85f7006de1138fca3"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 118806,
         "digest": "sha256:49619c54e28aeaac232848cd01e7ad36cd09f510d1d85f4dcc00910718db6522"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 3272,
         "digest": "sha256:bb68e5b0f80bb078cf66ae3035ec2167f297cd3a6d3d666da4232767382edec8"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1238025,
         "digest": "sha256:e72f6b146ebae8517164982f24bf6c0d5431db49c0fad4cf28904a9ca49f3df0"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 28656097,
         "digest": "sha256:04a273b385eeebdef8346d27d2eef69480592fa8aad653b331c777698265450b"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 22200,
         "digest": "sha256:7730bada3a37a604f1a535285c1fe4221c5b6776b48eccf1c7391714d5c62201"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 62963,
         "digest": "sha256:ce447929b35debd77b2833710585426f983254c01f5eebd6181f2157c5256bb1"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 12047103,
         "digest": "sha256:922f822b4ff03a02c5acbaf57ee6e6b180ec232eae6e0a8432e0912e980aee40"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 286,
         "digest": "sha256:f2447d21bc52d7cfa36306ff7a3d9728140d3c62ad1253ef9efee4043997fd81"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1479,
         "digest": "sha256:468f6bfdeddbb1ae04cf2fb149ea8df90cd1b86d60b1ec0fd3d87a3be7718c88"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 5246831,
         "digest": "sha256:d4d87e6504cbb435f04794852cd476ce1d2caac81f3786f940f961849800abb6"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 130343383,
         "digest": "sha256:f95d9bbdbc49d9468c8d1c28a83f90157cfb833b4b47463cdad6bb9ef75b3097"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 272,
         "digest": "sha256:75bdbcfd9f9940490ba8755d1dc50d8d5e8b414f61e65f582cf5112ea19fc6f7"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 24728449,
         "digest": "sha256:3ebff47a120faab2c75dd58447c86ac4498695d9c37c1e203e1a97a30915acca"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 116034746,
         "digest": "sha256:dd6a6486d568956a30bff9e406a027a1d22b8c9d79ec45f3f86720570e4d708e"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 99350,
         "digest": "sha256:90a73267518c6532dfa5775f0f864e9ae7f9b1b0f9fc4b2db47b61364ac2d120"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 181541,
         "digest": "sha256:4d9c9686288744f62fcd5b3337f10f530c3ac3e279ab2894c08e0f3fbb938de6"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 24011,
         "digest": "sha256:fd04258c36c3b14e2bd16f0cce5cd8c6860cab3696a9370c6b28ae631f23fded"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1106084,
         "digest": "sha256:a4b94f66fef1fca2172e27295b5d4a5f6b75f2f042aa53e6636041b59c680944"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 486,
         "digest": "sha256:e62dd035cac93aefc76063fc765ae4feba29ec8a47316a7971ac5d6a5b9688fa"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 5249,
         "digest": "sha256:8c66ea1a1b1d952d70abc3ebba34189facbfcb611dc6870a3b6a7647be000051"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 935,
         "digest": "sha256:d3250c19b091f37ce13aa570590add26b5c00d2b52bb1d0bb14e25c6fbab0fda"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 155,
         "digest": "sha256:617cc459cb0c982d60d07443aff0144ffa266317f4a3d43cb21dbb085c124189"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 991,
         "digest": "sha256:8adcdb4ca15b743035290dc176a22324d9210039c542dddaf747cef809f089ef"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1487,
         "digest": "sha256:1cb86656ba14a1e02d230dfdbcf7315c42e132c3e38ad7db8417cdb7e36fe507"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 267,
         "digest": "sha256:15ab3f627c99ecf33f07fe0d8859fbb66d9c876a6c22529e6bbc626944372e5c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 443,
         "digest": "sha256:1f4fd663c9ad3465a6283237ae2be64d842ce03564e65d5c17884242395fb0cc"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2032,
         "digest": "sha256:eb93186e6a4a7a623cfa6e2fe60db5198d9696ccc9f2c49ce569f5afa5c35410"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 800,
         "digest": "sha256:4ad255126fd5d9fc50e967766ddad7dbf05680fe2e35c2f68bb42c15fe1e08ea"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 10607,
         "digest": "sha256:102dd5306f6b94915d358b24223b09f548899723f1601bcbe6ed61a13d98475a"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 252083199,
         "digest": "sha256:1317475c362c2f4de969b8af7122949922724eb6a2db38e78034a6e500a9064b"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 259,
         "digest": "sha256:95dea11a8d53f5f0ceacf3da3944cf48ecb44dbf60c18478c6124fe3bff52106"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 261,
         "digest": "sha256:fcfbcaa9b95d7f2a43854ba1834dbf73f63a10d04ad1027b5433a28bed0fe169"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 6464,
         "digest": "sha256:5431e022259450792a514cd07f0867eea949ccd43c71b52baacc333c74a4131a"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 38286162,
         "digest": "sha256:5b435fe4685594f51f6f1ce44ff810f22c1fca98639efdee49c8914420b5212d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 589,
         "digest": "sha256:469301530f3372eb1c6904d47546c023a474a5d538f028bb6a134648054622f3"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 42822307,
         "digest": "sha256:e8105c68f57b6b610934f9964796305e2a6b8d7774c638e5a521342ba3e43c32"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 3073542,
         "digest": "sha256:539f1263a4547729f407af27b0a6cd7e92dd588e17ea857ddef99050c9da5b9c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 650,
         "digest": "sha256:fd36031d504e64fb4e21b2a5e5031e33a02301e056dffe976f5865ac406ea5be"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1920706,
         "digest": "sha256:7fedcb0c3ccb7f4be494233bc5bd074d53982f61a54a625b6816a70dfa4ab4a5"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 102899,
         "digest": "sha256:5781fb4cdb220a28cfdae4d97a5b9bf35d3c7c6c4159fbbcae55cfb510f37cbc"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 301363,
         "digest": "sha256:54ac2ac527fcfbcebfba8786a13c1169465f6a9c1f8f20ae2021b6876697f67e"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 5630,
         "digest": "sha256:184b26966bf69580f6b2efc883a86d9f31a7a27528f41768d64562463febf3af"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 13197,
         "digest": "sha256:12ea1d6601e8601c4141e2090df5ad94f56d1ccef896119696630ef5640d7c90"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 434,
         "digest": "sha256:7c5907b67312a07b73b324dfe3f5afd823fdb71bf7847cd4c6f6934386e5c1bd"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1067,
         "digest": "sha256:ada14d3d14afb1bff98cfd9fa302965c39fa601abd492adba6c30bae9ecf0085"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 99638152,
         "digest": "sha256:5383e7ba589bc48daf225ad5fc8e80625093615d543b33c7ef6a4f5c899d6482"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1061,
         "digest": "sha256:4dc1d1550a2f34c367b6a756f0e01b4aa6a5d55e5bd9af4a4760b25b891e0866"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 291808146,
         "digest": "sha256:e82866797f48b073d0439cd3a3623327dacde1662dd234750a170ed7c77db1fc"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 323510465,
         "digest": "sha256:91245636b9e750c489e2dc0bdf56a3bad8362d65df9a2de681ae5f396df3e1db"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 509,
         "digest": "sha256:aa7cbe57c7d676911a26375a7775a4066440d8a55fa590f32c2c975f2c78940d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 61113842,
         "digest": "sha256:7cf79dc2922190d1e562b032fb0b44e3489269ec1a42f7c763304162571a4739"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 144667,
         "digest": "sha256:ff73a15bfb7943157456a9800221950ed6dcbe50493c82668bf9c98f44c574d9"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1611863,
         "digest": "sha256:9ca79dd886e138ff6e895900d2460977b4d63bf2e844ef3d7a699e9679b6d692"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 598351626,
         "digest": "sha256:3eb58ec0065e491437bbea885b6baf7088f2fa321c3a2804500e3e43549146ad"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 62061421,
         "digest": "sha256:6b4356c65a8c22162c60ce80913a289ab5cccaa34243dd47eda36b677f4fa70c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2177881,
         "digest": "sha256:e4b4c1ce305730db6e8c46e1596e91841b0d82f263a535df00bfcaf8f5382e08"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 616416,
         "digest": "sha256:9df6c03fd5db0a52765f2d9aaa9851d389b0657edcbfffff5e2e92a66e012f9c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 12152086,
         "digest": "sha256:d0536eb32a39de2ddd55f202988e22d6d328c5463879392adc8e4a482bd7d515"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1014832809,
         "digest": "sha256:6920e70b61f4b9eb58beabaec635598460964512afdb49a033da31893ec80d00"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 9172561,
         "digest": "sha256:f49fb72039932702296ce93abd89154b50a18ab9f6036b3c12021924ac3226bf"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 110,
         "digest": "sha256:52cb83c792b58bb33cc7ade993fadf65b899a6068a9803b662b37c0c7433a6a4"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 445,
         "digest": "sha256:cc11074384051921adb1c89fe2209b0525fec7067dd734da44e194820aea067d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 12556665,
         "digest": "sha256:d98213c63a990229448a75a3a5d982b0f4b26a074382be11481f4160d6b6265f"
      }
   ]

scancodeio-web-1  | Operations to perform:
scancodeio-web-1  |   Apply all migrations: admin, auth, authtoken, contenttypes, django_rq, scanpipe, sessions, taggit
scancodeio-web-1  | Running migrations:
scancodeio-web-1  |   No migrations to apply.
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [26] [INFO] Starting gunicorn 22.0.0
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [26] [INFO] Listening at: http://0.0.0.0:8000 (26)
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [26] [INFO] Using worker: sync
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [27] [INFO] Booting worker with pid: 27
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [28] [INFO] Booting worker with pid: 28
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [32] [INFO] Booting worker with pid: 32
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [34] [INFO] Booting worker with pid: 34
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [37] [INFO] Booting worker with pid: 37
scancodeio-web-1  | [2024-06-13 15:40:39 +0000] [44] [INFO] Booting worker with pid: 44
scancodeio-web-1  | [2024-06-13 15:40:40 +0000] [45] [INFO] Booting worker with pid: 45
scancodeio-web-1  | [2024-06-13 15:40:40 +0000] [52] [INFO] Booting worker with pid: 52

scancodeio-worker-1  | 17:37:34 default: scanpipe.tasks.execute_pipeline_task(run_pk='4ff64875-43da-4732-8e00-ce4a1b08e92f') (4ff64875-43da-4732-8e00-ce4a1b08e92f)
scancodeio-worker-1  | INFO Run[4ff64875-43da-4732-8e00-ce4a1b08e92f] Enter `execute_pipeline_task` Run.pk=4ff64875-43da-4732-8e00-ce4a1b08e92f
scancodeio-worker-1  | INFO Run[4ff64875-43da-4732-8e00-ce4a1b08e92f] Run pipeline: "analyze_docker_image" on project: "6666"
scancodeio-worker-1  | INFO 2024-06-13 17:37:34.94 Pipeline [analyze_docker_image] starting
scancodeio-worker-1  | INFO 2024-06-13 17:37:34.95 Step [download_missing_inputs] starting
scancodeio-worker-1  | INFO 2024-06-13 17:37:34.95 Fetching input from docker://xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/dev/my_image
scancodeio-worker-1  | INFO Fetching "docker://xxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/dev/my_image" using fetch_docker_image
scancodeio-worker-1  | INFO Fetching image os/arch data: ('/opt/scancodeio/venv/lib/python3.12/site-packages/fetchcode_container/bin/skopeo', 'inspect', '--insecure-policy', '--raw', '--authfile /etc/scancodeio/auth.json', 'docker://xxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/dev/my_image')
scancodeio-worker-1  | INFO 2024-06-13 17:37:34.99 docker://xxxx.dkr.ecr.eu-west-1.amazonaws.com/dev/my_image could not be fetched.
scancodeio-worker-1  | INFO 2024-06-13 17:37:35.00 Pipeline failed
scancodeio-worker-1  | INFO Run[4ff64875-43da-4732-8e00-ce4a1b08e92f] Update Run instance with exitcode, output, and end_date
scancodeio-worker-1  | 17:37:35 default: Job OK (4ff64875-43da-4732-8e00-ce4a1b08e92f)
scancodeio-worker-1  | 17:37:35 Result is kept for 500 seconds
scancodeio-nginx-1   | 10.241.85.125 - - [13/Jun/2024:17:37:35 +0000] "GET /project/6666-f50926f5/ HTTP/1.1" 200 30443 "http://10.91.9.120/project/6666-f50926f5/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" "-"
scancodeio-nginx-1   | 10.241.85.125 - - [13/Jun/2024:17:37:35 +0000] "GET /project/6666-f50926f5/resource_status_summary/ HTTP/1.1" 200 1 "http://10.91.9.120/project/6666-f50926f5/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" "-"
scancodeio-nginx-1   | 10.241.85.125 - - [13/Jun/2024:17:37:35 +0000] "GET /project/6666-f50926f5/resource_license_summary/ HTTP/1.1" 200 1 "http://10.91.9.120/project/6666-f50926f5/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" "-"

@eli-koren
Copy link 

docker --version
Docker version 23.0.2, build 569dd73

tdruez added a commit that referenced this issue Jun 18, 2024
@tdruez
Enhance the logging and reporting of input fetch exceptions #1257
1cd9fef 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jun 18, 2024
@tdruez
Enhance the logging and reporting of run_command_safely #1257
6674a61 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jun 18, 2024
@tdruez
Enhance the logging and reporting of input fetch exceptions #1257 (#1269
2389d3c 
)

Signed-off-by: tdruez <tdruez@nexb.com>
@tdruez
Copy link
Member

tdruez commented Jun 18, 2024

@eli-koren Thanks for those outputs, it helps to figure out what is going on.
I've refined the logging system to provide more insight into those issues fetching private Docker images.

You can update your ScanCode.io checkout to the latest version of the main branch using:

cd scancode.io
git pull
docker compose build
docker compose restart

Once upgraded, could retry on a new Project?
The pipeline should still fail, but we will have much more data about the issue in the "Task output" section in the UI (when using the red "Failure" button)

@eli-koren
Copy link

Thank you @tdruez
There you go:

InputFilesError encountered with the following issues:

Error 1: Error while executing cmd="('/opt/scancodeio/venv/lib/python3.12/site-packages/fetchcode_container/bin/skopeo', 'inspect', '--insecure-policy', '--raw', '--authfile /etc/scancodeio/auth.json', 'docker://xxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/dev/my_image')": "time="2024-06-18T11:21:00Z" level=fatal msg="unknown flag: --authfile /etc/scancodeio/auth.json""

Traceback (most recent call last):
  File "/opt/scancodeio/scanpipe/pipelines/__init__.py", line 247, in download_missing_inputs
    input_source.fetch()
  File "/opt/scancodeio/scanpipe/models.py", line 1668, in fetch
    downloaded = fetch_url(url=self.download_url)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/scanpipe/pipes/fetch.py", line 376, in fetch_url
    downloaded = fetcher(url)
                 ^^^^^^^^^^^^
  File "/opt/scancodeio/scanpipe/pipes/fetch.py", line 279, in fetch_docker_image
    if platform := get_docker_image_platform(docker_url):
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/scanpipe/pipes/fetch.py", line 220, in get_docker_image_platform
    output = run_command_safely(cmd_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/scancodeio/scanpipe/pipes/fetch.py", line 79, in run_command_safely
    raise subprocess.SubprocessError(error_msg)
subprocess.SubprocessError: Error while executing cmd="('/opt/scancodeio/venv/lib/python3.12/site-packages/fetchcode_container/bin/skopeo', 'inspect', '--insecure-policy', '--raw', '--authfile /etc/scancodeio/auth.json', 'docker://xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/dev/orion:my_image')": "time="2024-06-18T11:21:00Z" level=fatal msg="unknown flag: --authfile /etc/scancodeio/auth.json""


Traceback:
  File "/opt/scancodeio/scanpipe/pipelines/__init__.py", line 214, in execute
    step(self)
  File "/opt/scancodeio/scanpipe/pipelines/__init__.py", line 255, in download_missing_inputs
    raise InputFilesError(error_tracebacks)

tdruez added a commit that referenced this issue Jun 18, 2024
@tdruez
Use the --option=value syntax for run_command_safely args #1257
fcf3451 
Signed-off-by: tdruez <tdruez@nexb.com>
tdruez added a commit that referenced this issue Jun 18, 2024
@tdruez
Use the --option=value syntax for run_command_safely args #1257 (#1270)
a1fdff9 
Signed-off-by: tdruez <tdruez@nexb.com>
@tdruez
Copy link
Member

tdruez commented Jun 18, 2024

msg="unknown flag: --authfile /etc/scancodeio/auth.json""

Turns out the --authfile /etc/scancodeio/auth.json syntax works fine when directly running the skopeo executable but is problematic when calling it through the Python subprocess.

I've merged a fix in the main branch (PR #1270)

@eli-koren Could you upgrade and try again?

cd scancode.io
git pull
docker compose build
docker compose restart

@eli-koren
Copy link

@tdruez
Amazing!
Thank you for your help and quick assistance.

2024-06-18 13:12:19.29 Pipeline [analyze_docker_image] starting
2024-06-18 13:12:19.29 Step [download_missing_inputs] starting
2024-06-18 13:12:19.30 Fetching input from docker://xxxxx.dkr.ecr.eu-west-1.amazonaws.com/dev:my_image
2024-06-18 13:13:10.51 Step [download_missing_inputs] completed in 51 seconds
2024-06-18 13:13:10.64 Step [extract_images] starting
2024-06-18 13:13:25.47 Step [extract_images] completed in 15 seconds
2024-06-18 13:13:25.67 Step [extract_layers] starting
2024-06-18 13:13:50.46 Step [extract_layers] completed in 25 seconds
2024-06-18 13:13:50.46 Step [find_images_os_and_distro] starting
2024-06-18 13:13:50.47 Step [find_images_os_and_distro] completed in 0 seconds
2024-06-18 13:13:50.47 Step [collect_images_information] starting
2024-06-18 13:13:50.48 Step [collect_images_information] completed in 0 seconds
2024-06-18 13:13:50.49 Step [collect_and_create_codebase_resources] starting

@tdruez
Copy link
Member

tdruez commented Jun 18, 2024

@eli-koren Great! Thank you so much for your help finding the bug 🚀

@tdruez tdruez closed this as completed Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tdruez @eliekoren @eli-koren