NodejsFunction: Fail to build with docker login and docker exited with status 125

[lmouhib]

Describe the bug

NodeJsFunction fail the build during cdk synth when using Python. I am not providing my own Dockerfile. The implementation is in this repo. Below is the log for the failur.

#14 [10/10] RUN /sbin/useradd -u 1000 user && chmod 711 /
#14 CACHED
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

View build details: docker-desktop://dashboard/build/sad_heisenberg/sad_heisenberg0/x0isk1zvr1xmvyt4hs4a2ssit

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
Bundling asset Stack/InteractiveSessionProvider/OnEventHandlerFunction/Code/Stage...
esbuild cannot run locally. Switching to Docker bundling.
Unable to find image 'cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab:latest' locally
docker: Error response from daemon: pull access denied for cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
jsii.errors.JavaScriptError: 
  @jsii/kernel.RuntimeError: Error: Failed to bundle asset Stack/InteractiveSessionProvider/OnEventHandlerFunction/Code/Stage, bundle output is located at /Users/yyyy/xxxxx/cdk.out/bundling-temp-e33d0e767876b374cc4c34eda8c2c07e1cc83da5543b15d1b7bae574e0fb9f6e-error: Error: docker exited with status 125
  --> Command: docker run --rm -u "504:20" -v "/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/jsii-kernel-srRVBP/node_modules/@cdklabs/aws-data-solutions-framework/lib/processing/lib/spark-runtime/emr-containers/resources/lambdas/managed-endpoint:/asset-input:delegated" -v "/Users/yyyy/xxxxx/emr-cost-tracking-bug-fix/cdk.out/bundling-temp-e33d0e767876b374cc4c34eda8c2c07e1cc83da5543b15d1b7bae574e0fb9f6e:/asset-output:delegated" -w "/" cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab bash -c "esbuild --bundle \"/asset-input/index.mjs\" --target=node20 --platform=node --outfile=\"/asset-output/index.js\" --external:@aws-sdk/*"
      at Kernel._Kernel_ensureSync (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:10491:23)
      at Kernel.sinvoke (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:9876:102)
      at KernelHost.processRequest (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:11696:36)
      at KernelHost.run (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:11656:22)
      at Immediate._onImmediate (/private/var/folders/ty/hzqp2fl51r9c1146jdmgl0mc0000gr/T/tmpva2camqw/lib/program.js:11657:46)
      at process.processImmediate (node:internal/timers:478:21)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/yyyy/xxxxx/app.py", line 10, in <module>
    EmrCostTrackingBugFixStack(app, "Stack",
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_runtime.py", line 118, in __call__
    inst = super(JSIIMeta, cast(JSIIMeta, cls)).__call__(*args, **kwargs)
  File "/Users/yyyy/xxxxx/emr_cost_tracking_bug_fix/emr_cost_tracking_bug_fix_stack.py", line 17, in __init__
    emr_eks_cluster = dsf.processing.SparkEmrContainersRuntime.get_or_create(self,
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/cdklabs/aws_data_solutions_framework/processing/__init__.py", line 1551, in get_or_create
    return typing.cast("SparkEmrContainersRuntime", jsii.sinvoke(cls, "getOrCreate", [scope, props]))
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 149, in wrapped
    return _recursize_dereference(kernel, fn(kernel, *args, **kwargs))
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/__init__.py", line 418, in sinvoke
    response = self.provider.sinvoke(
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 383, in sinvoke
    return self._process.send(request, InvokeResponse)
  File "/Users/yyyy/xxxxx/.venv/lib/python3.9/site-packages/jsii/_kernel/providers/process.py", line 342, in send
    raise RuntimeError(resp.error) from JavaScriptError(resp.stack)
RuntimeError: Error: Failed to bundle asset Stack/InteractiveSessionProvider/OnEventHandlerFunction/Code/Stage, bundle output is located at /Users/yyyy/xxxxx/cdk.out/bundling-temp-e33d0e767876b374cc4c34eda8c2c07e1cc83da5543b15d1b7bae574e0fb9f6e-error: Error: docker exited with status 125

Expected Behavior

The cdk synth should succeed

Current Behavior

CDK synth fail

Reproduction Steps

I can link

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.128.0

Framework Version

No response

Node.js Version

v20.11.1

OS

Mac

Language

Python

Language Version

3.9.6

Other information

No response

[pahud]

We can't debug your code from an external repo but looking at your provided error messages:

Unable to find image 'cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab:latest' locally
docker: Error response from daemon: pull access denied for cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.

Looks like

1. cdk-9ee7a11fc0243092baf39e8f9d149e65e683a14e5695909d84aa461dc52a6dab:latest does not exist at local
2. docker daemon was trying to pull that from your staging ecr repo and failed due to permission denied.

I am guessing this might be the permission issue.

Instead of looking at your linked code from the external repo, are you able to provide a minimal code snippet that reproduces this issue here? Simplifying the code would help us address the root cause easier.

[lmouhib]

We can't debug your code from an external repo but looking at your provided error messages:

The repo is hosted under an AWS github organisation. Also I am not providing any Dockerfile, its using the Dockerfile provided with cdk-lib to build the nodejs function.

docker daemon was trying to pull that from your staging ecr repo and failed due to permission denied
On the same code base if I used finchI successfully build and deploy, but with docker it fails with permission denied.

[pahud]

If that error only happens from https://github.com/awslabs/data-solutions-framework-on-aws please submit a ticket to that repo for their visibility.

If you believes it still happens in aws-cdk-lib please provide a minimal code snippet here so we can reproduce it.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[lmouhib]

@pahud i am one the maintainer the repo I mention. The issues is not with the repo, but happens on how CDK pull the docker image, which I have no control on, because I do not specify that image.