[BUG] External dependencies are unavailable after deploying

[iquabius]

Describe the bug

I tried adding date-fns as a dependency and it worked fine running my function locally: npx nx run graphql:serve

But after deploying to AWS and testing I get an error because date-fns is missing:

The lambda logs:

2021-06-30T20:01:35.197Z	undefined	ERROR	Uncaught Exception	{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module 'date-fns'",
    "stack": [
        "Runtime.ImportModuleError: Error: Cannot find module 'date-fns'",
        "    at _loadUserApp (/var/runtime/UserFunction.js:100:13)",
        "    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)",
        "    at Object.<anonymous> (/var/runtime/index.js:45:30)",
        "    at Module._compile (internal/modules/cjs/loader.js:778:30)",
        "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)",
        "    at Module.load (internal/modules/cjs/loader.js:653:32)",
        "    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:585:3)",
        "    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)",
        "    at startup (internal/bootstrap/node.js:283:19)"
    ]
}

From what I understand after going through the README and the code, this plugin should generate a package.json with date-fns in dist/.serverlessPackages, right?

But the resulting package.json is:

{
  "name": "graphql",
  "version": "1.0.0",
  "description": "Packaged externals for graphql",
  "private": true,
  "scripts": {
    "package-yarn": "yarn",
    "package-npm": "npm install"
  }
}

To Reproduce

Clone my repo: https://github.com/iquabius/flow-nx-serverless-external-deps-bug

And:

1. Install dependencies: npm i
2. Deploy: npx nx run graphql:deploy
3. Try the AWS url
4. See error: "Internal server error"

Expected behavior
The function should return "Today is a <day of week>"

Screenshots
If applicable, add screenshots to help explain your problem.

Check which provider is affected:
[x] AWS
[] Azure
[] Google Cloud Platform

Check which framework is affected:
[] Angular
[] Nodejs
[x] Serverless
[] Lambda
[] Infrastructure as a code

Additional context
Add any other context about the problem here.

[wickstargazer]

i am investigating into this. Will find some free time this week to debug, thanks for the repo i can reproduce it now

[tstackhouse]

I've ran into this in the past (on older versions) and your triage is spot on, though the root cause may be different. My gut feeling is that it's coming from webpack's packaging, beyond that right now I'm not sure, I'll defer to Wick for now :)

[brian-weasner]

Any progress on this? This is happening for me with my dependencies.

I did some debugging, and it looks like the following is causing me issues.

nx-plugins/libs/nx-serverless/src/utils/webpack.stats.ts

Line 62 in 62d46c7

if (!chunk.modules) {

My webpack stats do not have modules within each of the chunks. Instead each of my modules have chunks within them.
See copied log of stats param from getExternalModules function:

  {
    "errors": [],
    "warnings": [],
    "hash": "",
    "builtAt": 1626391702947,
    "publicPath": "",
    "outputPath": "",
    "entrypoints": {},
    "namedChunkGroups": {},
    "chunks": [
      {
        "id": "src/app/common/core/jwt-authorizer",
        "rendered": true,
        "initial": true,
        "entry": true,
        "size": 7825,
        "names": [
          "src/app/common/core/jwt-authorizer"
        ],
        "files": [
          "src/app/common/core/jwt-authorizer.js"
        ],
        "hash": "cf21867fdbf405864441",
        "siblings": [],
        "parents": [],
        "children": [],
        "childrenByOrder": {}
      },
      ...
    ],
    "modules": [
      {
        "id": "aws-sdk",
        "identifier": "external \"aws-sdk\"",
        "name": "external \"aws-sdk\"",
        "index": 7,
        "index2": 5,
        "size": 42,
        "built": true,
        "optional": false,
        "prefetched": false,
        "chunks": [
          "src/app/controllers/location"
        ],
        "issuer": "",
        "issuerId": "",
        "issuerName": "",
        "issuerPath": [
          {
            "id": "",
            "identifier": "",
            "name": "",
            "profile": {
              "factory": 17,
              "building": 323
            }
          },
          {
            "id": "",
            "identifier": "",
            "name": "",
            "profile": {
              "factory": 56,
              "building": 279,
              "dependencies": 277
            }
          }
        ],
        "profile": {
          "factory": 1,
          "building": 0
        },
        "failed": false,
        "errors": 0,
        "warnings": 0,
        "assets": [],
        "providedExports": null,
        "optimizationBailout": [],
        "depth": 2
      },
      {
        "id": "crypto",
        "identifier": "external \"crypto\"",
        "name": "external \"crypto\"",
        "index": 4,
        "index2": 2,
        "size": 42,
        "built": true,
        "optional": false,
        "prefetched": false,
        "chunks": [
          "src/app/common/core/jwt-authorizer",
          "src/app/controllers/location"
        ],
        "issuer": "",
        "issuerId": "",
        "issuerName": "",
        "issuerPath": [
          {
            "id": "",
            "identifier": "",
            "name": "",
            "profile": {
              "factory": 73,
              "building": 321,
              "dependencies": 7
            }
          },
          {
            "id": "",
            "identifier": "",
            "name": "",
            "profile": {
              "factory": 9,
              "building": 397
            }
          }
        ],
        "profile": {
          "factory": 0,
          "building": 0
        },
        "failed": false,
        "errors": 0,
        "warnings": 0,
        "assets": [],
        "providedExports": null,
        "optimizationBailout": [],
        "depth": 2
      },
      ...
    ],
    "filteredModules": 0,
    "logging": {
      "webpack.buildChunkGraph.visitModules": {
        "entries": [],
        "filteredEntries": 2,
        "debug": false
      }
    }
  }

Changing getExternalModules(stats: any) to

        if (!stats.chunks) {
            return [];
        }
        const externals = new Set();
        for (const module of stats.modules) {
          if (this.isExternalModule(module) && module.chunks) {
            for (const chunkId of module.chunks) {
              if (stats.chunks.find((chunk) => chunk.id === chunkId)) {
                externals.add({
                  origin: module.issuer,
                  external: this.getExternalModuleName(module),
                });
              }
            }
          }
        }
        return Array.from(externals);

seems to properly add the dependencies to the generated package.json, subsequently allowing the builder to install them into the dist folder for deployment.

[rpaddock]

did #89 solve this issue? I'm seeing a valid package.json now, but the dependencies are not making it into the artifacts.

[brian-weasner]

It worked for me.
Note: I also have "externalDependencies": "all" set in my angular.json file (Under project-name.architect.build.options) which I believe is the option to install and package the external dependencies.