Pnpm install issues

[DavidWells]

We just switched from npm to pnpm for faster installs & a monorepo setup.

It appears there are some issues with this package

> Build error occurred
Error: AWS CRT binary not present in any of the following locations:
	/Users/xyz/abc/clients/src/app/.next/bin/native/aws-crt-nodejs
	/Users/xyz/abc/clients/src/app/.next/bin/darwin-x64/aws-crt-nodejs
	 at Object.19698 (/Users//xyz/abc/clients/src/app/.next/serverless/chunks/706.js:288530:11)
    at __webpack_require__ (/Users/xyz/abc/clients/src/app/.next/serverless/pages/index.js:16366:43)

It appears that the bin is missing?

This is a related issue. aws/aws-iot-device-sdk-js-v2#119 (comment)

Stackoverflow suggested copying the bin where it needs to go https://stackoverflow.com/questions/68063950/aws-iot-sdk-js-v2-connection-problem-aws-crt-binary-not-present-in-any-of-the

Any pointers on how to solve this. I'm unfamiliar with this package and am wondering why it needs a bin to run as we are consuming the SDK in our next.js application

Thanks 😃

[bretambrose]

aws-crt-nodejs is primarily a native addon (with a parallel browser implementation based on mqtt-js and other pure js libraries). The package that gets published to npm contains binaries for OsX, 32 and 64 bit Windows, x64 linux and armv8 linux.

[DavidWells]

Hey @bretambrose thanks for the context.

It appears that the module resolution is failing in

aws-crt-nodejs/lib/native/binding.js

Line 46 in d751ed6

throw new Error("AWS CRT binary not present in any of the following locations:\n\t" + search_paths.join('\n\t'));

because of how package managers like yarn and pnpm hoist their modules to the root directory of a project. So the binding isn't in the local child node_modules folder.

We aren't using MQTT or anything that relies on this (to my knowledge). Our application uses cognito + s3 parts of the AWS SDK.

Is there any way to disable this warning?

[lukasa1993]

this issue happens with @aws-sdk/client-s3 and NPM

2021-09-04T08:42:03.897267170Z Error: Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /opt/app/node_modules/aws-crt/dist/bin/linux-x64/aws-crt-nodejs.node)
2021-09-04T08:42:03.897272311Z     at Object.Module._extensions..node (internal/modules/cjs/loader.js:1122:18)
2021-09-04T08:42:03.897276232Z     at Module.load (internal/modules/cjs/loader.js:928:32)
2021-09-04T08:42:03.897280106Z     at Function.Module._load (internal/modules/cjs/loader.js:769:14)
2021-09-04T08:42:03.897282394Z     at Module.require (internal/modules/cjs/loader.js:952:19)
2021-09-04T08:42:03.897284594Z     at require (internal/modules/cjs/helpers.js:88:18)
2021-09-04T08:42:03.897287403Z     at Object.<anonymous> (/opt/app/node_modules/aws-crt/dist/native/binding.js:55:19)
2021-09-04T08:42:03.897291584Z     at Module._compile (internal/modules/cjs/loader.js:1063:30)
2021-09-04T08:42:03.897295247Z     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
2021-09-04T08:42:03.897305734Z     at Module.load (internal/modules/cjs/loader.js:928:32)
2021-09-04T08:42:03.897317056Z     at Function.Module._load (internal/modules/cjs/loader.js:769:14)

with these dependencies:

"dependencies": {
   "@aws-sdk/client-s3": "^3.16.0",
   "@aws-sdk/s3-presigned-post": "^3.16.0",
   "@aws-sdk/s3-request-presigner": "^3.16.0",

[DavidWells]

I've tried to fix this with patch-package but no luck yet.

It turns out next.js build process runs from a .next cwd and this is causing the file here to not resolve.

After I fix the path to point to node_modules/aws-crt/dist/bin/darwin-x64/aws-crt-nodejs.node the build still failed with an error:

Error: Cannot find module '/xyz/clients/src/app/node_modules/aws-crt/dist/bin/darwin-x64/aws-crt-nodejs.node'

// Also fails without .node
Error: Cannot find module '/xyz/clients/src/app/node_modules/aws-crt/dist/bin/darwin-x64/aws-crt-nodejs'

The /xyz/clients/src/app/node_modules/aws-crt/dist/bin/darwin-x64/aws-crt-nodejs.node file exists at this path. So I'm not quite sure why this require call is failing.

Is it possible that something is corrupted in how the postinstall scripts/install.js script runs?

Currently blocked on this issue and am trying to figure out a way to disable or mock out the missing require (as our code doesn't use this, at least not to my knowledge.)

[DavidWells]

Additional context, we are running this in node v16.6.1. Perhaps the require is failing in the latest versions of node.js?

also to @lukasa1993's point, we are using the @aws-sdk/client-s3 lib

    "@aws-amplify/auth": "^4.3.2",
    "@aws-sdk/client-s3": "^3.11.0",

[lukasa1993]

Additional context, we are running this in node v16.6.1. Perhaps the require is failing in the latest versions of node.js?

also to @lukasa1993's point, we are using the @aws-sdk/client-s3 lib

    "@aws-amplify/auth": "^4.3.2",
    "@aws-sdk/client-s3": "^3.11.0",

i am using node 14 so its just lib is bad i downgraded to strict "3.24" instead of ^ and probably will never upgrade this lib it seems everything aws is just faulty and pre-alpha version

[DavidWells]

Found another thread where folks are also experiencing the "not found" issue aws/aws-sdk-js-v3#2750 (comment)

[bretambrose]

A while back, I remember trying pnpm with a CRT-based app and not being able to reproduce this issue. I tried again recently and had no luck either. The issue may still be present but I don't think it's just pnpm alone. I feel like there's additional project setup/config (bundling/stripping/packing etc...) needed at a minimum.

Closing for now, but am happy to look again at it should explicit repro be found.