[BUG] npm ci fails after installing @aws/pdk

[tmokmss]

Describe the bug

Running npm ci fails after installing pdk.

Similar problem as #381. It seems the issue has been re-introduced somewhere.

Expected Behavior

npm ci successes

Current Behavior

npm ci fails

Reproduction Steps

npm i @aws/pdk
npm ci
# npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

Possible Solution

No response

Additional Information/Context

No response

PDK version used

0.23.30

What languages are you seeing this issue on?

Typescript

Environment details (OS name and version, etc.)

macOS 14.3.1

[agdimech]

Are you able to please share a sample codebase so I can repro or provide steps to repro?

also have you tried deleting the lock file and running npm install first?

[tmokmss]

@agdimech
You should be able to reproduce it by just running the two commands in the issue description.

More specifically:

$ mkdir pdk-repro
$ cd pdk-repro
$ npm init -y
$ npm install @aws/pdk
$ npm ci
npm ERR! code EUSAGE
npm ERR!
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

And my environment:

$ npm --version
10.2.3
$ node --version
v20.10.0

also have you tried deleting the lock file and running npm install first?

This issue reproduces right after initializing a project by npm init -y.

[agdimech]

Thanks for the info.

The issue stems from the fact that bundleDependencies is not super well supported by pnpm (and most package managers) and at pack time, it just adds whatever is currently within the packages node_modules. This is problematic as most of the actual dependencies are stored in the root node_modules. We previously had a script that would link in any missing transitive bundleDependencies but this doesn't handle the case for multiple versions of the same package.

I re-worked all of this to use pnpm deploy which allows you to deploy a package to a new directory and install all deps directly in the node_modules in a flat fashion (similar to npm). This resolves the issues as now multiple package with the same version are correctly included via package node_modules nesting.

Here is the PR: #756

Apologies that you ran into this again, this should fix it once and for all :)

[tmokmss]

Great to hear that. Looking forward to the new release :)

[agdimech]

Seems that my latest fix isn't working. Upon inspecting the tar.gz there are no nested node_modules however locally I can see them. Will need to investigate into this further to see why the distributable doesn't match. Apologies.

[agdimech]

Been doing some more digging and the actual tgz file looks correct locally, but for some reason it is missing the nested node_modules on the CI publishing workflow. Could potentially be a versioning issue with pnpm/npm, will keep digging.

[agdimech]

Have tried multiple fixes on my end to no avail. This is actually a bug with npm as per: npm/cli#6787

The only fix for this at the moment is to either:

• Use and older version of npm (7.0.8 works) which is the last known version of npm that didn't have the bug 😢
• run npm i && npm i

[github-actions]

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

[github-actions]

Closing this issue as it hasn't seen activity for a while. Please add a comment @mentioning a maintainer to reopen. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

[tmokmss]

@agdimech FYI even running npm i does not fix the problem. Currently as long as @aws/pdk is installed, we cannot use npm ci by any means. Is it expected? We haven't seen any other libraries behaving like this.

# in an empty directory
npm init -y
npm i @aws/pdk
npm ci # > fail
npm i && npm ci #> fail

[FunctionalWorm]

@agdimech We are still experiencing the same issue with npm 10.9.0.
We use PDK to generate architecture diagram for our CDK services. Our CI/CD depends on npm ci so it's a blocker for us.

We would appreciate if you could offer a workaround or a fix. Thank you!

$ npm init -y
$ npm i @aws/pdk
$ npm ci
npm error code EUSAGE
npm error
npm error `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm error
npm error Missing: chalk@2.4.2 from lock file
npm error Missing: execa@0.1.2 from lock file
npm error Missing: @pnpm/logger@5.2.0 from lock file
npm error Missing: js-yaml@0.0.6 from lock file
npm error Missing: bole@5.0.16 from lock file
npm error Missing: ndjson@2.0.0 from lock file
npm error Missing: fast-safe-stringify@2.1.1 from lock file
npm error Missing: individual@3.0.0 from lock file
npm error Missing: json-stringify-safe@5.0.1 from lock file
npm error Missing: minimist@1.2.8 from lock file
npm error Missing: readable-stream@3.6.2 from lock file
npm error Missing: split2@3.2.2 from lock file
npm error Missing: through2@4.0.2 from lock file
npm error Missing: inherits@2.0.4 from lock file
npm error Missing: string_decoder@1.3.0 from lock file
npm error Missing: util-deprecate@1.0.2 from lock file
npm error Missing: safe-buffer@5.2.1 from lock file
npm error Missing: path-is-absolute@1.0.1 from lock file
npm error Missing: balanced-match@1.0.2 from lock file
npm error Missing: concat-map@0.0.1 from lock file
npm error Missing: is-arrayish@0.2.1 from lock file
npm error Missing: type-fest@0.6.0 from lock file
npm error Missing: mimic-fn@3.1.0 from lock file
npm error Missing: hosted-git-info@6.1.1 from lock file
npm error Missing: p-limit@2.3.0 from lock file
npm error Missing: lines-and-columns@1.2.4 from lock file
npm error Missing: normalize-package-data@2.5.0 from lock file
npm error Missing: type-fest@0.6.0 from lock file
npm error Missing: glob@7.2.3 from lock file
npm error Missing: fs-extra@8.1.0 from lock file
npm error Missing: @types/node@22.7.9 from lock file
npm error Missing: arg@4.1.3 from lock file
npm error Missing: typescript@5.6.3 from lock file
npm error Missing: undici-types@6.19.8 from lock file
npm error Missing: signal-exit@4.1.0 from lock file
npm error Missing: ansi-styles@3.2.1 from lock file
npm error Missing: escape-string-regexp@1.0.5 from lock file
npm error Missing: supports-color@5.5.0 from lock file
npm error Missing: color-convert@1.9.3 from lock file
npm error Missing: color-name@1.1.3 from lock file
npm error Missing: has-flag@3.0.0 from lock file
npm error Missing: @zkochan/which@2.0.3 from lock file
npm error Missing: execa@5.1.1 from lock file
npm error Missing: path-name@1.0.0 from lock file
npm error Missing: isexe@2.0.0 from lock file
npm error Missing: lru-cache@7.18.3 from lock file
npm error Missing: p-try@2.2.0 from lock file
npm error Missing: hosted-git-info@2.8.9 from lock file
npm error Missing: resolve@1.22.8 from lock file
npm error Missing: semver@5.7.2 from lock file
npm error Missing: is-core-module@2.15.1 from lock file
npm error Missing: path-parse@1.0.7 from lock file
npm error Missing: supports-preserve-symlinks-flag@1.0.0 from lock file
npm error Missing: hasown@2.0.2 from lock file
npm error Missing: function-bind@1.1.2 from lock file
npm error Missing: minimatch@3.1.2 from lock file
npm error Missing: brace-expansion@1.1.11 from lock file
npm error Missing: jsonfile@4.0.0 from lock file
npm error Missing: universalify@0.1.2 from lock file
npm error
npm error Clean install a project
npm error
npm error Usage:
npm error npm ci
npm error
npm error Options:
npm error [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
npm error [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
npm error [--include <prod|dev|optional|peer> [--include <prod|dev|optional|peer> ...]]
npm error [--strict-peer-deps] [--foreground-scripts] [--ignore-scripts] [--no-audit]
npm error [--no-bin-links] [--no-fund] [--dry-run]
npm error [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
npm error [-ws|--workspaces] [--include-workspace-root] [--install-links]
npm error
npm error aliases: clean-install, ic, install-clean, isntall-clean
npm error
npm error Run "npm help ci" for more info