[aws-lambda-nodejs] Broken in BitBucket Pipelines

[misterjoshua]

There seems to be a problem with NodejsFunction in Bitbucket Pipelines. When I attempt to create a lambda using that construct and synthesize, I get an error about lscpu and another Error: EACCES: permission denied, open '/asset-output/index.js'.

Reproduction Steps

bug-stack.ts

import * as cdk from '@aws-cdk/core';
import * as nodejs from '@aws-cdk/aws-lambda-nodejs';

export class BugStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new nodejs.NodejsFunction(this, 'handler');
  }
}

bug-stack.handler.ts

export const handler = async () => {
  console.log("Hello, world!");
};

bitbucket-pipelines.yml

image: node:12
pipelines:
  default:
    - step:
        caches:
          - node
          - docker
        services:
          - docker
        script:
          - node -v
          - yarn
          - yarn cdk --version
          - yarn cdk synth
        after-script:
          - ls -la
          - ls -la .cdk.staging

Error Log

yarn run v1.22.4
warning package.json: No license field
$ cdk synth
Failed to run bundling Docker image for asset BugStack/handler/Code/Stage: Error: [Status 1] stdout: 
stderr: /bin/sh: lscpu: not found
events.js:292
      throw er; // Unhandled 'error' event
      ^
Error: EACCES: permission denied, open '/asset-output/index.js'
Emitted 'error' event on WriteStream instance at:
    at internal/fs/streams.js:376:12
    at FSReqCallback.oncomplete (fs.js:155:23) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/asset-output/index.js'
}
Subprocess exited with error 1
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Environment

• CLI Version : 1.47.0
• Framework Version: 1.47.0
• Node.js Version:
• OS : BitBucket Pipelines, based on the node:12 image
• Language (Version): TypeScript (3.7.2)

Other

This might be related to #8707, #8492

Here's what I get from ls -la in my pipeline:

+ ls -la
total 548
drwxrwxrwx.   9 root root   4096 Jun 26 17:20 .
drwxr-xr-x.   8 root root   4096 Jun 26 17:18 ..
drwxrwxrwx.   3 root root   4096 Jun 26 17:20 .cdk.staging
drwxrwxrwx.   8 root root   4096 Jun 26 17:18 .git
-rw-rw-rw-.   1 root root    135 Jun 26 17:18 .gitignore
-rw-rw-rw-.   1 root root     65 Jun 26 17:18 .npmignore
-rw-rw-rw-.   1 root root    543 Jun 26 17:18 README.md
drwxrwxrwx.   2 root root   4096 Jun 26 17:18 bin
-rw-rw-rw-.   1 root root    322 Jun 26 17:18 bitbucket-pipelines.yml
-rw-rw-rw-.   1 root root    152 Jun 26 17:18 cdk.json
drwxrwxrwx.   2 root root   4096 Jun 26 17:20 cdk.out
-rw-rw-rw-.   1 root root    130 Jun 26 17:18 jest.config.js
drwxrwxrwx.   3 root root   4096 Jun 26 17:20 lib
drwxr-xr-x. 470 root root  20480 Jun 26 17:19 node_modules
-rw-rw-rw-.   1 root root 277830 Jun 26 17:18 package-lock.json
-rw-rw-rw-.   1 root root    604 Jun 26 17:18 package.json
drwxrwxrwx.   2 root root   4096 Jun 26 17:18 test
-rw-rw-rw-.   1 root root    598 Jun 26 17:18 tsconfig.json
-rw-rw-rw-.   1 root root 195681 Jun 26 17:18 yarn.lock

In .cdk.staging.

+ ls -la .cdk.staging
total 12
drwxrwxrwx. 3 root root 4096 Jun 26 17:20 .
drwxrwxrwx. 9 root root 4096 Jun 26 17:20 ..
drwx------. 2 root root 4096 Jun 26 17:20 asset-bundle-gYtxGP

If I change the pipeline user to 1000, by changing the yml image to image: { name: node:12, run-as-user: 1000 }, the synth still fails with the same error. The only difference I can see is that the .cdk.staging and cdk.out directories have a different owner. (1000, which is the 'node' user)

---

This is 🐛 Bug Report

[misterjoshua]

I've done some digging. I think using mkdtemp to create the staged asset directories is where the problem starts.

From what I'm seeing, mkdtemp is creating the asset staging directories with mode 0700. It seems that unless "group" and "other" have read/write on that directory (i.e., mode 0777), the volume is inaccessible from the bundler's container in Bitbucket Pipelines.

I managed to get the bundler to work by writing a hacky script outside of CDK and by providing export CDK_DOCKER=$PWD/bin/docker-stub.sh in my pipeline scripts ...

bin/docker-stub.sh

#!/bin/bash

echo "Running docker $*" >>docker-stub.log

ASSET_NAME=$(egrep -o "asset-bundle-[^:]*" <<<"$*")
if [ ! -z "$ASSET_NAME" ]; then
  echo "Caught bundling: $ASSET_NAME." >>docker-stub.log

  echo "Chmodding og+rwx" >>docker-stub.log
  chmod og+rwx .cdk.staging/$ASSET_NAME &>>docker-stub.log
fi

exec docker "$@"

Updated bitbucket-pipelines.yml

image: node:12

pipelines:
  default:
    - step:
        caches:
          - node
          - docker
        services:
          - docker
        script:
          - node -v
          - yarn
          - export CDK_DOCKER=$PWD/bin/docker-stub.sh
          - yarn cdk synth
        after-script:
          - cat docker-stub.log

docker-stub.log

+ cat docker-stub.log
Running docker build --build-arg NODE_TAG=12.18.1-alpine /opt/atlassian/pipelines/agent/build/node_modules/@aws-cdk/aws-lambda-nodejs/parcel-bundler
Running docker run --rm -u 0:0 -v /opt/atlassian/pipelines/agent/build:/asset-input -v /opt/atlassian/pipelines/agent/build/.cdk.staging/asset-bundle-XSmrUb:/asset-output -w /asset-input/lib 974ee55199ba parcel build /asset-input/lib/bug-stack.handler.ts --out-dir /asset-output --out-file index.js --global handler --target node --bundle-node-modules --log-level 2 --no-minify --no-source-maps
Caught bundling: asset-bundle-XSmrUb.
Dirlist
.cdk.staging:
total 12
drwxrwxrwx. 3 root root 4096 Jun 26 20:20 .
drwxrwxrwx. 9 root root 4096 Jun 26 20:20 ..
drwx------. 2 root root 4096 Jun 26 20:20 asset-bundle-XSmrUb
.cdk.staging/asset-bundle-XSmrUb:
total 8
drwx------. 2 root root 4096 Jun 26 20:20 .
drwxrwxrwx. 3 root root 4096 Jun 26 20:20 ..
Chmodding og+rwx

One other thing worth noting is that if I rm -rf the asset directories in my script, instead of chmodding, Docker still works fine. I don't know that it's strictly necessary to pre-create the .cdk.staging asset directories, as Docker seems to be happy enough to create the host volume directories (and any parent directories necessary) on its own.

[eladb]

@jogold can you take a look?

[anilchinnam]

I am also running into this issue and unable to upgrade my cdk version from 1.36.1 to 1.58.0. I tried couple of previous versions to upgrade but the issue is been there for quite some time.

Please some one help, what is the work around on this, I tried providing the 777 permission to both .cdk.staging and .parcel-cache folders and deleted these folders before running the synth. Still getting the same error.
This is on my Mac and i have the docker running (not sure why it is looking for docker in first place for bundling)

Bundling asset hello-world-dev-ApiDynamoDbStack/dlqS3UtilFunction/dlqS3UtilFunction/Code/Stage...
[Error: EACCES: permission denied, mkdir '/.parcel-cache'] {
errno: -13,
code: 'EACCES',
syscall: 'mkdir',
path: '/.parcel-cache'
}
toSoaMonitoring format:json {
"@timestamp": "2020-08-13T21:58:10.921Z",
"env": "dev",
"cbMonIdx": "sls",
"logLevel": "error",
"msg": "Error during cdk synth"
} Error: Failed to bundle asset hello-world-dev-ApiDynamoDbStack/dlqS3UtilFunction/dlqS3UtilFunction/Code/Stage: Error: docker exited with status 1
at AssetStaging.bundle (/Users/achinnam/Applications/hello-world/hello-world-app/node_modules/@aws-cdk/core/lib/asset-staging.ts:196:13)
at new AssetStaging (/Users/achinnam/Applications/hello-world/hello-world-app/node_modules/@aws-cdk/core/lib/asset-staging.ts:93:29)