Add for support for ESM #1014

AshUK · 2020-06-11T08:36:15Z

Feature Request

Add the ability to use (ESM modules)(https://nodejs.org/docs/latest-v12.x/api/esm.html#esm_interoperability_with_commonjs)

Currently if you declare your serverless project as type module in package.js. Node will flag that InProcessRunner.js is trying to require an es module

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module

Expected behavior/code
Any ES module used as entry point should work without any further compiling (webpack/babel)

The text was updated successfully, but these errors were encountered:

cyberwombat · 2020-06-14T15:44:03Z

Looks like it should be pretty easy. I just tweaked lines around ~69 in the InProcessRunner.js dist file and seems to handle loading modules. Using node 14.x without webpack.

// const {
//   [_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
// } = await Promise.resolve().then(() => _interopRequireWildcard(import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}`)));

const {
[_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
} = await Promise.resolve().then(() => import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}.js`));

razor-x · 2021-03-29T00:50:00Z

Now that Node v14 support is released, this issue is the only blocker left (I hope) to fully use ES modules on lambda.

I tried out your suggestion as a patch-package on my skeleton project and it seems to work: https://github.com/makenew/serverless-nodejs/blob/cab54afb4bd4acf97bfa3ea2a94bea50c1b88581/patches/serverless-offline%2B6.9.0.patch

Only issue now is how to port this to the actual source code? Seems like the source already uses import:

serverless-offline/src/lambda/handler-runner/in-process-runner/InProcessRunner.js

Line 104 in 58dc8ae

const { [this.#handlerName]: handler } = await import(this.#handlerPath)

Thus I suspect this is some webpack or babel magic that transforms the import into _interopRequireWildcard which uses require under the hood. Maybe if this package was distributed as both a ES module and commonjs module we could work around the issue. Then consumers who are using type: module will get the ES module source and existing users will no see any difference.

It's possible to distribute both using exports. Old versions of Node will ignore it and fall back to main while newer versions will load the ES module if the caller supports it.

{
  "type": "module",
  "main": "dist/index.cjs",
  "exports": {
    "import": "./src/index.js",
    "require": "./dist/index.cjs"
  },
  "module": "index.js"
}

Side note, we also need a way to deal with the file extension issue since the handler must be imported with a file extension, but how do we know if it's .js or .mjs?

razor-x · 2021-03-29T05:25:56Z

I worked a bit more on this. Turns out the AWS lambda runtime always loads the handler as a commonjs module via require.

So while ESM support for this serverless-offline would be nice, it's not needed to deploy ESM to lambda today. You just need to have thin commonjs handler entry points and put all your ESM code somewhere else. (It should even be possible to put handler.js and otherthing.cjs in the same directory, but then you will not be able to set type: module in package.json).

You can see a working example of this here (note the empty package.json file): https://github.com/makenew/serverless-nodejs/tree/v7.0.0/handlers

AshUK · 2021-03-30T11:32:23Z

Turns out the AWS lambda runtime always loads the handler as a commonjs module via require.

I cannot find a reference to this, can you link the source or was this trial and error testing ?

I would like to add my thanks for your time looking in to this.

razor-x · 2021-03-30T17:24:22Z

I did not find documentation for this. I tired deploying and invoking the function as an ES module but got this error:

2021-03-29T04:49:39.330Z	undefined	ERROR	Uncaught Exception 	{"errorType":"Error","errorMessage":"Must use import to load ES Module: /var/task/handlers/todo.js\nrequire() of ES modules is not supported.\nrequire() of /var/task/handlers/todo.js from /var/runtime/UserFunction.js is an ES module file as it is a .js file whose nearest parent package.json contains \"type\": \"module\" which defines all .js files in that package scope as ES modules.\nInstead rename todo.js to end in .cjs, change the requiring code to use import(), or remove \"type\": \"module\" from /var/task/package.json.\n","code":"ERR_REQUIRE_ESM","stack":["Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /var/task/handlers/todo.js","require() of ES modules is not supported.","require() of /var/task/handlers/todo.js from /var/runtime/UserFunction.js is an ES module file as it is a .js file whose nearest parent package.json contains \"type\": \"module\" which defines all .js files in that package scope as ES modules.","Instead rename todo.js to end in .cjs, change the requiring code to use import(), or remove \"type\": \"module\" from /var/task/package.json.","","    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1080:13)","    at Module.load (internal/modules/cjs/loader.js:928:32)","    at Function.Module._load (internal/modules/cjs/loader.js:769:14)","    at Module.require (internal/modules/cjs/loader.js:952:19)","    at require (internal/modules/cjs/helpers.js:88:18)","    at _tryRequire (/var/runtime/UserFunction.js:75:12)","    at _loadUserApp (/var/runtime/UserFunction.js:95:12)","    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)","    at Object.<anonymous> (/var/runtime/index.js:43:30)","    at Module._compile (internal/modules/cjs/loader.js:1063:30)"]}

Or re-formatting so you can actually read it:

Must use import to load ES Module: /var/task/handlers/todo.js
require() of ES modules is not supported.
require() of /var/task/handlers/todo.js from /var/runtime/UserFunction.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename todo.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /var/task/package.json.

  'Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /var/task/handlers/todo.js',
  'require() of ES modules is not supported.',
  'require() of /var/task/handlers/todo.js from /var/runtime/UserFunction.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.',
  'Instead rename todo.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /var/task/package.json.',
  '',
  '    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1080:13)',
  '    at Module.load (internal/modules/cjs/loader.js:928:32)',
  '    at Function.Module._load (internal/modules/cjs/loader.js:769:14)',
  '    at Module.require (internal/modules/cjs/loader.js:952:19)',
  '    at require (internal/modules/cjs/helpers.js:88:18)',
  '    at _tryRequire (/var/runtime/UserFunction.js:75:12)',
  '    at _loadUserApp (/var/runtime/UserFunction.js:95:12)',
  '    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)',
  '    at Object.<anonymous> (/var/runtime/index.js:43:30)',
  '    at Module._compile (internal/modules/cjs/loader.js:1063:30)'

If I had to take a guess, AWS probably just updated their lambda runtime to Node 14 without changing the code that loads the handler. It's possible there may be some way to make it load the ES module, but without access /var/runtime/UserFunction.js or more documentation there is no way to know.

I bet it would be possible with a custom lambda runtime that used import over require.

Also ref: https://stackoverflow.com/a/66688901/1675295

reinhardholl · 2021-07-06T09:55:40Z

Any idea of when we can expect this support ? We are using Webpack at the moment, but if ESM modules are supported we can strip that from our process 🙏

razor-x · 2021-12-30T05:49:00Z

You can work around this by putting all your handler entry points in their own folder with an "empty" package.json. If you want ES modules in the same folder as the handlers, you can use the .mjs extension. For a working example, see https://github.com/makenew/serverless-nodejs/blob/fa148a9dd210c21736c84014c32f6da151176aba/handlers/todo.js

thejuan · 2022-01-14T02:43:37Z

Lambda now supports ESM for node 14 https://aws.amazon.com/about-aws/whats-new/2022/01/aws-lambda-es-modules-top-level-await-node-js-14/

ZebraFlesh · 2022-01-18T20:58:50Z

Any update on this issue, now that ESM is natively supported in lambda? I've got a few projects where developers went whole hog and used ESM in production without realizing their offline configs were now broken.

pgrzesik · 2022-01-19T09:45:39Z

We'd be happy to accept a PR that adds this 👍

perrin4869 · 2022-02-28T03:49:27Z

I tried removing babel/plugin-proposal-dynamic-import, which would be the first step to getting this feature out, but immediately you hit the problem that the tests fail with a segfault... I think this is the underlying issue...
I think one of the challenges here is the choice of testing framework, because jest by default mocks all imports/requires (as far as I understand), and it even requires an experimental node flag to be able to run these imports...
Any opinions? @pgrzesik

pgrzesik · 2022-02-28T12:03:41Z

Honestly I'm not an expert on the topic so I don't have any opinion, I would have to look deeper into that topic, but I definitely won't be able to dedicate time to this in the foreseeable future.

perrin4869 · 2022-02-28T12:10:37Z

I'm interested in looking further into this, but I suspect that this might involve replacing jest as the testing framework with mocha...

ZebraFlesh · 2022-02-28T13:00:12Z

vitest supports mocking esm dependencies and has a strong level of API compatibility with jest: https://vitest.dev/

perrin4869 · 2022-02-28T13:28:56Z

Ah nice, didn't know that! That could save a lot of work!!!
Although honestly the minimalism of mocha might be worth the extra effort, especially since this is not a front-end lib, and I don't think the test suites here are doing much or any mocking

pgrzesik · 2022-02-28T14:37:18Z

If there would be a need for migration to another test runner, then mocha would be the preffered direction as it's used extensively in the Framework and related projects.

ZebraFlesh · 2022-02-28T14:45:09Z

especially since this is not a front-end lib

I did forget that vitest is more aimed at front end code, so perhaps that's not so great a suggestion.

RavenHursT · 2022-03-01T23:50:53Z

Well.. now I know why the previous developer on the project I just inherited was transpiling a node project in 2022 🤦

Stack traces on exceptions in our CloudWatch logs are essentially useless, even w/ --source-maps=true in our babel build command 😢

vishnup95 · 2022-03-27T19:15:51Z

Did anyone get serveless-offline working with type:module in package.json. Maybe I am missing something (I have babel installed, but not sure if I have configured it just right?). I keep getting the error. @reinhardholl Sorry to ping you here a slighly old issue, but could you please help out with babel/webpack config you used?

✖ Error [ERR_REQUIRE_ESM]: require() of ES Module /home/vishnu/Documents/XXX/XXX/.webpack/service/functions/kafka-starter/index.js from /home/vishnu/Documents/XXX/XX/node_modules/serverless-offline/dist/lambda/handler-runner/in-process-runner/InProcessRunner.js not supported.

winston0410 · 2022-05-12T14:26:28Z

Still not able to use a package with type:module, for example: https://www.npmjs.com/package/pkg-dir.

Setting type:module will get an error from serverless-offline now. Any solution to this?

perrin4869 · 2022-05-12T14:44:54Z

Looks like it should be pretty easy. I just tweaked lines around ~69 in the InProcessRunner.js dist file and seems to handle loading modules. Using node 14.x without webpack.

// const {
//   [_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
// } = await Promise.resolve().then(() => _interopRequireWildcard(import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}`)));

const {
[_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
} = await Promise.resolve().then(() => import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}.js`));

You can apply the patch here using patch-package for now, it's a bit of a maintenance pain but I've been using it for a while without problems.
I did get started on a PR but I'm stuck for now... Jest is a real mess...

RavenHursT · 2022-05-13T08:33:50Z

Looks like it should be pretty easy. I just tweaked lines around ~69 in the InProcessRunner.js dist file and seems to handle loading modules. Using node 14.x without webpack.
// const {
//   [_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
// } = await Promise.resolve().then(() => _interopRequireWildcard(import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}`)));

const {
[_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
} = await Promise.resolve().then(() => import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}.js`));
You can apply the patch here using patch-package for now, it's a bit of a maintenance pain but I've been using it for a while without problems. I did get started on a PR but I'm stuck for now... Jest is a real mess...

Gotta love how much "unit testing" "speeds up development" 🤪

QAnders · 2022-05-15T18:47:24Z

Any update on this, and of course the common dependent plug-ins, e.g. offline, prune, ssm offline?

With node 16 now available for Lambda and more and more modules moving to ESM we'd like to start using ESM for new projects...

jlarmstrongiv · 2022-06-30T18:38:50Z

Related aws/aws-lambda-nodejs-runtime-interface-client#44

higherorderfunctor · 2022-07-21T16:08:32Z

Looks like it should be pretty easy. I just tweaked lines around ~69 in the InProcessRunner.js dist file and seems to handle loading modules. Using node 14.x without webpack.

// const {
//   [_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
// } = await Promise.resolve().then(() => _interopRequireWildcard(import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}`)));

const {
[_classPrivateFieldLooseBase(this, _handlerName)[_handlerName]]: handler
} = await Promise.resolve().then(() => import(`${_classPrivateFieldLooseBase(this, _handlerPath)[_handlerPath]}.js`));

This patch broke in 9.x

RichiCoder1 · 2022-08-02T22:55:55Z

Here's the new patch:

diff --git a/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/InProcessRunner.js b/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/InProcessRunner.js
index f42be20..81286dd 100644
--- a/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/InProcessRunner.js
+++ b/node_modules/serverless-offline/src/lambda/handler-runner/in-process-runner/InProcessRunner.js
@@ -53,7 +53,16 @@ export default class InProcessRunner {
       // eslint-disable-next-line import/no-dynamic-require
       ;({ [this.#handlerName]: handler } = require(this.#handlerPath))
     } catch (err) {
-      log.error(err)
+      if (err.code?.includes('ERR_REQUIRE_ESM')) {
+        try {
+          const { [this.#handlerName]: esHandler } = await import(`file://${require.resolve(this.#handlerPath)}`);
+          handler = esHandler;
+        } catch (err) {
+          log.error(err)
+        }
+      } else {
+        log.error(err)
+      }
     }
 
     if (typeof handler !== 'function') {

And a much more thorough fix that covers more cases is coming soon with #1397

dnalborczyk · 2022-08-10T13:37:38Z

resolved in: #1534

justingrant mentioned this issue Aug 10, 2021

Error trying to import ESM Temporal polyfill in a CJS environment js-temporal/temporal-polyfill#29

Closed

perrin4869 mentioned this issue Apr 14, 2022

[Feature]: Allow regular node dynamic import calls from jest test files jestjs/jest#12439

Closed

perrin4869 mentioned this issue May 10, 2022

Add support for ESM modules #1397

Closed

stefanullinger mentioned this issue Jun 9, 2022

NextjsSite hangs on deploy (ESM issue I think) sst/sst#1775

Closed

dnalborczyk added the feature label Jul 7, 2022

dnalborczyk mentioned this issue Aug 9, 2022

feat: use aws-lambda-ric UserFunction.js #1534

Merged

dnalborczyk closed this as completed Aug 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add for support for ESM #1014

Add for support for ESM #1014

AshUK commented Jun 11, 2020

cyberwombat commented Jun 14, 2020

razor-x commented Mar 29, 2021 •

edited

razor-x commented Mar 29, 2021

AshUK commented Mar 30, 2021 •

edited

razor-x commented Mar 30, 2021 •

edited

reinhardholl commented Jul 6, 2021

razor-x commented Dec 30, 2021

thejuan commented Jan 14, 2022

ZebraFlesh commented Jan 18, 2022

pgrzesik commented Jan 19, 2022

perrin4869 commented Feb 28, 2022

pgrzesik commented Feb 28, 2022

perrin4869 commented Feb 28, 2022

ZebraFlesh commented Feb 28, 2022

perrin4869 commented Feb 28, 2022

pgrzesik commented Feb 28, 2022

ZebraFlesh commented Feb 28, 2022

RavenHursT commented Mar 1, 2022 •

edited

vishnup95 commented Mar 27, 2022

winston0410 commented May 12, 2022

perrin4869 commented May 12, 2022

RavenHursT commented May 13, 2022

QAnders commented May 15, 2022

jlarmstrongiv commented Jun 30, 2022

higherorderfunctor commented Jul 21, 2022

RichiCoder1 commented Aug 2, 2022 •

edited

dnalborczyk commented Aug 10, 2022

Add for support for ESM #1014

Add for support for ESM #1014

Comments

AshUK commented Jun 11, 2020

Feature Request

cyberwombat commented Jun 14, 2020

razor-x commented Mar 29, 2021 • edited

razor-x commented Mar 29, 2021

AshUK commented Mar 30, 2021 • edited

razor-x commented Mar 30, 2021 • edited

reinhardholl commented Jul 6, 2021

razor-x commented Dec 30, 2021

thejuan commented Jan 14, 2022

ZebraFlesh commented Jan 18, 2022

pgrzesik commented Jan 19, 2022

perrin4869 commented Feb 28, 2022

pgrzesik commented Feb 28, 2022

perrin4869 commented Feb 28, 2022

ZebraFlesh commented Feb 28, 2022

perrin4869 commented Feb 28, 2022

pgrzesik commented Feb 28, 2022

ZebraFlesh commented Feb 28, 2022

RavenHursT commented Mar 1, 2022 • edited

vishnup95 commented Mar 27, 2022

winston0410 commented May 12, 2022

perrin4869 commented May 12, 2022

RavenHursT commented May 13, 2022

QAnders commented May 15, 2022

jlarmstrongiv commented Jun 30, 2022

higherorderfunctor commented Jul 21, 2022

RichiCoder1 commented Aug 2, 2022 • edited

dnalborczyk commented Aug 10, 2022

razor-x commented Mar 29, 2021 •

edited

AshUK commented Mar 30, 2021 •

edited

razor-x commented Mar 30, 2021 •

edited

RavenHursT commented Mar 1, 2022 •

edited

RichiCoder1 commented Aug 2, 2022 •

edited