(cdk-assets): support building Docker images with Podman

[fcoelho]

I'd like to use Podman to build images when using aws_ecs.ContainerImage.fromAsset("folder")

Use Case

Podman allows me to build and run containers locally without any elevated privileges while maintaining basically the same command line interface as regular Docker. My distribution even provides a /usr/bin/docker file that forwards all commands to podman so most of the time, invoking docker run ... just works.

For the commands aws-cdk currently use, everything works apart from docker inspect, which returns a different error code than docker on failure.

Proposed Solution

I've patched this file locally and have been deploying stuff with aws-cdk for a few months based on this:

-     if (e.code !== 'PROCESS_FAILED' || e.exitCode !== 1) { throw e; }
+     if (e.code !== 'PROCESS_FAILED' || ![1, 125].includes(e.exitCode)) { throw e; }

Other

As above the immediate fix is pretty simple. Reason I didn't want to open a PR just yet is because I'm not sure how to add tests for this, if it needs to be running real podman or not for example. Another option would be to just change that to checking any failure and tests could still possibly pass

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[fcoelho]

A small update to the changes I'm running locally: the endpoint used by podman login now cannot be a https:// prefixed string, even though docker supports it (this endpoint change works for both docker and podman)

diff --git a/packages/cdk-assets/lib/private/docker.ts b/packages/cdk-assets/lib/private/docker.ts
index e1fc544..3a22a10 100644
--- a/packages/cdk-assets/lib/private/docker.ts
+++ b/packages/cdk-assets/lib/private/docker.ts
@@ -41,7 +41,7 @@ export class Docker {
       await this.execute(['inspect', tag], { quiet: true });
       return true;
     } catch (e) {
-      if (e.code !== 'PROCESS_FAILED' || e.exitCode !== 1) { throw e; }
+      if (e.code !== 'PROCESS_FAILED' || ![1, 125].includes(e.exitCode)) { throw e; }
       return false;
     }
   }
@@ -68,7 +68,7 @@ export class Docker {
     await this.execute(['login',
       '--username', credentials.username,
       '--password-stdin',
-      credentials.endpoint], {
+      credentials.endpoint.replace(/^https?:\/\//, '')], {
       input: credentials.password,
 
       // Need to quiet otherwise Docker will complain

I was also having issues with SELinux in my system but seems this will sort it, I'll test it when there's a new RC out: #15742

[paulegan]

Before I found this issue I had the same problems with podman and implemented almost identical workarounds.
So I can confirm that these changes fix cdk with podman (Fedora 33, Podman 3.3.1).

For anyone searching for the errors generated, without these changes login throws:

fail: docker login --username AWS --password-stdin https://88888.dkr.ecr.eu-west-1.amazonaws.com exited with error code 125: Error: credentials key has https[s]:// prefix

and inspect throws:

fail: docker inspect cdkasset-xxx exited with error code 125: Error: error inspecting object: no such object: "cdkasset-xxx"

[madeline-k]

Hey @fcoelho , Thanks for opening this issue! I would love to see a PR with these changes from you. As far as testing, I think we can just include some unit tests for this, it doesn't need to be running real Podman, just test that the functions you are modifying behave as expected. You will need to create a test file in https://github.com/aws/aws-cdk/tree/master/packages/cdk-assets/test/private to test docker.ts since it doesn't have a corresponding test file yet.

[Mohamed-Abubucker]

Hi,

Currently i am trying to use podman and buildah with aws cdk typescript project.

i don't have docker in my local. i want to build OCI based container images.

while deploying the stack, it runs docker login and deployment get failed as docker is not there.

is there any way to instruct aws cdk to use podman and buildah instead of docker?

it really blocks my productivity.

please advise!

tech stack: Ubuntu 20.04 (over WSL2 on Windows 10), NodeJs v12, cdk 1.133.0

[timgent]

@Mohamed-Abubucker if you are still running into this issue you can just create a symlink, something like the following (adjust the podman location and where you want the docker symlink to go depending on your OS):

sudo ln -s /opt/homebrew/bin/podman /usr/local/bin/docker

[rosskarchner]

I did want to note, if you have Podman running as a user daemon, and have DOCKER_HOST pointed at it (on my machine DOCKER_HOST=unix:///run/user/1000/podman/podman.sock, then you can install plain old Docker, and use it as a client for podman. CDK seems to work just fine with the setup.

[andreabedini]

This is another workaround for the error code that doesn't require modifying the source code.

$ cat <<EOF >~/bin/docker 
#!/usr/bin/env bash

podman "$@"

if [ $? == 125 ]; then
  # echo "tricked ya!"
  exit 1;
fi
EOF
$ chmod a+x ~/bin/docker
$ export PATH=$HOME/bin:$PATH

[Pharrox]

This is another workaround for the error code that doesn't require modifying the source code.

Thanks for this! Worked pretty well for me. Did run into an issue while building Python lambda layers. For anyone else experiencing something similar, add quotes around the arguments array, so "$@". This will prevent the CMD passed to podman by the CDK from getting treated as separate arguments when shouldn't be.

[andreabedini]

Thanks @Pharrox I have edited my workaround to say "$@".

[MrArnoldPalmer]

This should be resolved by #23855

[mrgrain]

This should be resolved by #23855

FYI #23855 does NOT solve this issue. However it sounds like the fixes for both issues with podman should be compatible with docker as well, so we would likely accept PRs for it.

[JQuezada0]

I'm not really sure how to reproduce the https issue, but I encountered and fixed the other one here #24003.

[JQuezada0]

Seems that the PR I linked (#24003) is about to be closed in a week, could someone confirm that it requires the integration tests?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.