[BUG] type-safe-api: intermittent ENOTEMPTY in CodeBuild

[jstrunk]

Describe the bug

After updating my PDK project from 0.23.6(I think) to version 0.23.28, I started getting intermittent build failures in the type-safe-api typescript runtime on CodeBuild. After retrying the build stage it usually succeeds.

Expected Behavior

Since the build works consistently locally, I expected it to succeed in CodeBuild, too.

Current Behavior

news-analysis-api-typescript-runtime: npm ERR! code ENOTEMPTY
news-analysis-api-typescript-runtime: npm ERR! syscall rmdir
news-analysis-api-typescript-runtime: npm ERR! path /root/.npm/_npx/fad988c9594ac250/node_modules/cdk-nag/lib/rules/autoscaling
news-analysis-api-typescript-runtime: npm ERR! errno -39
news-analysis-api-typescript-runtime: npm ERR! ENOTEMPTY: directory not empty, rmdir '/root/.npm/_npx/fad988c9594ac250/node_modules/cdk-nag/lib/rules/autoscaling'
news-analysis-api-typescript-runtime: npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-04-22T13_08_25_937Z-debug-0.log
news-analysis-api-typescript-runtime: 👾 Task "build » pre-compile » generate" failed when executing "npx --yes -p @aws/pdk@$AWS_PDK_VERSION type-safe-api.clean-openapi-generated-code --code-path ." (cwd: /codebuild/output/src2039979916/src/packages/api/generated/runtime/typescript)
news-analysis-api-typescript-runtime: Warning: run-commands command "pnpm exec projen build" exited with non-zero status code

Reproduction Steps

const cdkVersion = "2.135.0";
const pdkVersion = "0.23.28";

const CdkContext = {
};

const project = new monorepo.MonorepoTsProject({
  devDeps: [`@aws/pdk@^${pdkVersion}`],
  name: "news-analysis",
  packageManager: javascript.NodePackageManager.PNPM,
  projenrcTs: true,
  gitignore: [".bashrc", ".profile", ".idea"],
  prettier: true,
  licenseOptions: {
    disableDefaultLicenses: true,
  },
});

const api = new TypeSafeApiProject({
  parent: project,
  name: "news-analysis-api",
  outdir: "packages/api",
  model: {
    language: ModelLanguage.OPENAPI,
    options: {
      openapi: {
        title: "News-Analysis-API",
      },
    },
  },
  infrastructure: {
    language: Language.TYPESCRIPT,
  },
  handlers: {
    languages: [Language.PYTHON],
    options: {
      python: {
        devDeps: [
          "boto3@^1.34.77",
          "boto3-stubs-lite@{version = '^1.34.77', extras = ['bedrock-runtime','dynamodb','s3','sagemaker-runtime','ssm']}",
          "moto@{version = '^5.0.4', extras = ['all']}",
          "mypy",
          "pandas@^2.2.1",
          "awswrangler@^3.7.2",
        ],
      },
    },
  },
  runtime: {
    languages: [Language.PYTHON],
  },
  library: {
    libraries: [Library.TYPESCRIPT_REACT_QUERY_HOOKS],
  },
  documentation: {
    formats: [DocumentationFormat.HTML2, DocumentationFormat.MARKDOWN],
  },
  commitGenerated: true,
});

new CloudscapeReactTsWebsiteProject({
  parent: project,
  outdir: "packages/website",
  name: "demo-website",
  typeSafeApis: [api],
  gitignore: ["public/runtime-config*.json"],
});

new InfrastructureTsProject({
  parent: project,
  name: "news-analysis-infra",
  authorName: "X",
  authorEmail: "X",
  outdir: "packages/infra",
  cdkVersion: cdkVersion,
  typeSafeApis: [api],
  deps: [
    `@aws/pdk@^${pdkVersion}`,
    "@aws-sdk/client-s3@^3.533.0",
    "avsc",
    "@types/aws-lambda",
    "@types/luxon",
    "@aws/pdk@^0.23.6",
    "@aws-cdk/aws-glue-alpha",
    "@aws-cdk/aws-lambda-python-alpha",
    "cdk-nag",
    "@aws-lambda-powertools/tracer@^2.0.2",
    "@aws-sdk/client-bedrock-runtime@^3.533.0",
    "@aws-sdk/client-s3@^3.533.0",
    "@aws-sdk/client-ssm@^3.533.0",
    "@aws-sdk/lib-storage@^3.533.0",
    "@middy/core",
    "flowp@^0.4.4",
    "luxon",
    "winston@^3.12.0",
  ],
  context: CdkContext,
  prettier: true,
  sampleCode: false,
});
project.synth();

Possible Solution

No response

Additional Information/Context

No response

PDK version used

0.23.28

What languages are you seeing this issue on?

Typescript, Python

Environment details (OS name and version, etc.)

MacOS 14.3.1

[steven-taglohner]

Having the same issue here.

[agdimech]

Hi I am sorry you are running into this issue.

• Can you please send me what your build spec looks like and also what image you are using?
• How often are you seeing this issue?
• Ideally seeing full logs would help me debug further.
• To resolve this error are you simply restarting the build action?

[agdimech]

I was able to reproduce this locally by running this command in parallel:

npx --yes -p @aws/pdk type-safe-api.clean-openapi-generated-code --code-path .

What is happening is:

Process 1 creates a new directory and starts installing the @aws/pdk dependencies in to the /root/.npm/_npx/ folder
Process 2 comes in and tries to do the same thing which is leading to a corruption in this directory. This also explains why it fails on CI but not locally as in CI the /root/.npm/_npx/ will be empty wheras locally it will most likely already have the packages installed.

I am not sure why this is presenting more now, however this would have been an issue for quite some time as apparently npx and dlx are not concurrency-safe. What is interesting is this issue is only present if the /root/.npm/_npx/ folder does not have @aws/pdk and if the folder you are currently running in doesn't have PDK installed (which it should). The resolution path should looks as follows:

1.) Check all node_modules folders starting with the current folder, traversing upward to see if @aws/pdk@$AWS_PDK_VERSION exists.
2.) Fall back to global node_modules located in /root/.npm/_npx/.

I would have assumed that 1) should be resolved as the build command would be called after performing an install. In the event it falls back to 2), this will create a new folder and is where the issue you are seeing is occuring.

Can you please confirm that you have @aws/pdk installed within your local node_modules and it matches the version listed in your $AWS_PDK_VERSION (you can find this in the .projen/tasks.json file)?

As a stop gap there are two options:

Option 1 - Preload the package into the npx cache (recommended)

Add this before your build command: npx --yes -p @aws/pdk@<YOUR_PDK_VERSION> pdk --version.

Replace <YOUR_PDK_VERSION> with the same AWS_PDK_VERSION found within .projen/tasks.json. This will ensure that the package is already present in the npx cache and when the scripts are executed within the build command, it won't try to download the package in parallel (which is where the problem stems).

Option 2 - Force your build to run serially

Append the --parallel 1 flag to your build command to force each package to build in parallel.

Note: This will slow down your build.

I would advise you to try Option 1 and report back whether this helps resolve your issue. If so I will go ahead and raise a PR to incorporate Option 1 into the PDK so you don't have to explicitly perform that step.

[agdimech]

To add to this, there may be other stages in the build process that rely on just in time packages like for example if using pnpm, there is a pnpm dlx command that is equivalent to npx which will run into the same problems. We may need to perform this precaching step for these instances also i.e: pnpm --package=@aws/pdk dlx pdk --version.

[steven-taglohner]

The issue seems to be related to the release of the latest version of pnpm, version 9. After pinning the version to version 8, the build succeeds.