Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify Eleventy to work with ECMAScript Modules (ESM) by default #836

Open
zachleat opened this issue Dec 29, 2019 · 33 comments
Open

Modify Eleventy to work with ECMAScript Modules (ESM) by default #836

zachleat opened this issue Dec 29, 2019 · 33 comments
Labels
enhancement needs-votes A feature request on the backlog that needs upvotes or downvotes. Remove this label when resolved. research-needed

Comments

@zachleat
Copy link
Member

zachleat commented Dec 29, 2019

Node 13 projects can switch to ECMAScript Modules using "type": "module" in package.json, using .mjs files, or --input-type https://nodejs.org/docs/latest-v13.x/api/esm.html#esm_enabling

This causes problems with Eleventy, which uses require and CommonJS internally.

Here it is failing on a config file require:
image

Without a config file, it fails on 11ty.js files too:
image


Explore whether or not this is a possibility. Might need a major version bump? Might want to be prepared for Node 14 stable. We’re currently at Node 8+ right now but it exits maintenance very soon so we’ll need a major version bump to at least do Node 10+: https://nodejs.org/en/about/releases/

@zachleat zachleat added enhancement research-needed needs-votes A feature request on the backlog that needs upvotes or downvotes. Remove this label when resolved. labels Dec 29, 2019
@justinfagnani
Copy link
Contributor

justinfagnani commented Dec 29, 2019

I think you may be able to get support without a major version bump, though the support would only work in versions of Node with module support itself, which seems fine.

The first thing to change would be where the JavaScript template engine performs the actual require():

getInstanceFromInputPath(inputPath) {

That will need to become async, but luckily it's internal to the template engine and only called from two already async methods.

Since you can import() a CJS module in Node with JS modules support, to detect and load a module, I think there's only two things that need to be done:

  1. Detect if the environment supports modules
  2. If so, use import() to load all JS templates, CJS or standard JS

import() only works in >= 13.2, but the syntax is valid from 10 on (not sure the exact version). So, if you support only Node 10+, this should be pretty straightforward. If you want to still support 8+, you'll need the import() expression in a file you only require after detecting module support.

As for that, I'm not sure the best way. you could just key off the Node version, but that would leave off environments in 12 using flags. That might be ok. You could also try to require a file with import(), and if that works, then try to import something, and if that works your'e in an env that supports modules.

That's the basics, but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules, at least without spinning up a new VM context to run the templates in and writing loader and linker functions to make it all go.

@zachleat
Copy link
Member Author

zachleat commented Dec 30, 2019

Awesome, this info is very valuable—thank you!

but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules

Whoa, hmm—that would be a huge limitation. We need require/import cache invalidation to get new versions of templates during a --serve or --watch.

Not too much info on the docs either: https://nodejs.org/api/esm.html#esm_no_code_require_cache_code

@justinfagnani
Copy link
Contributor

justinfagnani commented Dec 30, 2019

Yeah, that's the thing to figure out before any of the other work... I wonder how big of a change it would be to spin up a new VM instance for every hot reload?

But... the module support in vm is still marked experimental, and requires a flag, so I think this would be something for a future major version of eleventy.

@zachleat zachleat added this to To do in Support ESM and .cjs Files via automation Mar 20, 2020
@georges-gomes
Copy link

georges-gomes commented Mar 25, 2020

Hi!

ES module support would be so much nicer (at least for me). I started my own little SSG in ES Module only to achieve that because I thought 11ty couldn't make the switch. As @justinfagnani said maybe there is a possibility...

Here are the things I learn in the process and I would be happy to contribute a few things if I'm good enough :D

(the following are my observation on node 13)

  1. There is no require cache for ES Modules like for commonJS.
    what has been recommended to me is to use the internal v8 "Debugger.setScriptSource" to replace code live. After few hours of research I came up with this https://gist.github.com/georges-gomes/6dc743addb90d2e7c5739bba00cf95ea
    It works most of the time but it fails quite often specially when you start modifying import/exports.
    Bugs are open on v8 for better ES modules of this API.
    Also, the code is replaced hot so none of the top level side effects are re-executed.
    I think this is bad for our purpose here.

  2. You can call import multiple times if you change the file name. On HTTP adding query params can do the trick of re-importing the module but it doesn't work on files. Node accept the query params on files but doesn't reimport. I think it's bug again. So you can still copy files and rename them then call import again...

  3. You can import commonJS modules from an ES module.

import { createRequire } from "module";
const require = createRequire(import.meta.url);
const es_dev_server = require("a-cjs-module");

I think it would be a good practice to have eleventy boot code in ESModule and load "templates" in cjs with this when required.

Conclusions:
Like @justinfagnani (but it took me days to come to this :)), I think that a spawning a new VM for every hot reload is a better way to go in order to solve this issue and have a single code base for both one-shot generate and serve/watch operations.

I hope this helps
Cheers

@TimvdLippe
Copy link

TimvdLippe commented Mar 31, 2020

I was able to make ES imports work with the following:

node -r esm node_modules/.bin/eleventy

It uses https://github.com/standard-things/esm to make both require and import {} from work in .11ty.js files.

@lennyanders
Copy link

lennyanders commented Jun 13, 2020

I needed the following command on Windows to make ESM working with 11ty:

node -r esm node_modules/@11ty/eleventy/cmd.js

Also, with the following esm config, I got export default for the 11ty config (.eleventy.js) working:

{
  "esm": {
    "cjs": {
      "dedefault": true
    }
  }
}

@thelucid
Copy link

thelucid commented Nov 13, 2020

Any news on this? Would be great to be able to use import in Eleventy.

@kuworking
Copy link

kuworking commented Nov 14, 2020

I was able to make ES imports work with the following:

node -r esm node_modules/.bin/eleventy

It uses https://github.com/standard-things/esm to make both require and import {} from work in .11ty.js files.

Do you have any repo that can be looked at?
Does your setup allow for using something like this in a .11ty.js file?

import { css } from '@linaria/core'

exports.data = {
  title: '',
  date: '',
  templateEngineOverride: '11ty.js,md',
}

exports.render = data =>
...

@thelucid
Copy link

thelucid commented Nov 14, 2020

@TimvdLippe Does this allow imports in standard ‘.js’ data files?

@thelucid
Copy link

thelucid commented Nov 14, 2020

@TimvdLippe Does the .11ty.js do something special over just calling your file 1-2.js? Pardon my ignorance, as I've just been using things.js in the global data directory, and want to use imports within these.

@TimvdLippe
Copy link

TimvdLippe commented Nov 14, 2020

@thelucid We specify it as input here: https://github.com/ChromeDevTools/debugger-protocol-viewer/blob/c12e43d2054d074ac07f8990c4b4be54a172c3cf/.eleventy.js#L11 but other than that we don't do anything special.

@thelucid
Copy link

thelucid commented Nov 15, 2020

I'm hitting issues with using imports within the data files, in your case _data. So something like _data/things.js doesn't work with imports.

@thelucid
Copy link

thelucid commented Nov 15, 2020

@TimvdLippe Got it working. Thanks so much. It should be nice to see eleventy sport this by default, so that imports work out of the box.

@justinfagnani
Copy link
Contributor

justinfagnani commented Nov 16, 2020

I'm not sure eleventy should support esm directly, since it's pretty non-standard in its capabilities. I'd like to see Node's VM modules support to stabilize and eleventy can use that to load projects and still support watch mode by creating a fresh context.

@thelucid
Copy link

thelucid commented Nov 16, 2020

I see, it that why reloads seem to have stopped working with ‘esm’?

@Zearin
Copy link

Zearin commented Nov 27, 2020

Recent related experience

I recently started trying to port a project that used a deep _data/ directory (yes, the same name! ☻), *.mjs files, along with import and export.

I had to rename a lot of files, and grep my files for uses of import and export, which took a few passes.

It wasn’t particularly awful. But it was tedious to do. Also, I was personally a little disappointed about having to return to CommonJS syntax, as I’ve switched over to ES module syntax everywhere I can.

If not for *.js, at least *.mjs?

Eleventy already makes many decisions based on file extensions. Simply telling Eleventy “If you see any *.mjs files, parse them as ES modules” would leave existing behavior as-is, while still allowing users to opt-in to ES modules by changing a file extension.

Would an extension-based approach address concerns raised above

@reubenlillie
Copy link

reubenlillie commented Jan 20, 2021

I’ve tested the solutions by @lennyanders (#836 (comment)) and @kuworking (#836 (comment)), using the build command syntax from the former and the esm configuration suggestion from the latter.

.esmrc.json:

{
  "cjs": {
    "dedefault": true
  }
}

The code is working in the eleventy-dot-js-blog starter kit if you’d like to peruse.

@yklcs
Copy link

yklcs commented Feb 17, 2021

Any news/update on this? Is it being developed at all? Support for even just Node 14+ would be great.

@flaki
Copy link

flaki commented Feb 21, 2021

The code is working in the eleventy-dot-js-blog starter kit if you’d like to peruse.

Unfortunately using ESM will cause issues on latest Node.js versions (v13 and up, as well as the latest backports on v12) for any dependencies that use type: module, and the ecosystem is moving in a direction where the number of these will only increase. Given that ESM has long been neglected & unmaintained, a fix is also rather unlikely at this point.

@flaki
Copy link

flaki commented Feb 22, 2021

but I see that you also delete the require cache as some point. I'm not sure you can do that at all with JS modules

Whoa, hmm—that would be a huge limitation. We need require/import cache invalidation to get new versions of templates during a --serve or --watch.

ESM imports do not use the require cache in Node.js. For cache-busting the good ole' query/fragment URL method can be applied, as I was writing this up Aral published a comprehensive how-to blog post on precisely this.

I was working on a simple proof-of-concept fork to see if I could add rudimentary ESM support and I managed to get this working:

Getting --build to work is the easier one to tackle, see here. One basically needs to async-ify the code path and swap out the require for dynamic import() calls.

Of course for commonjs support one still needs to use require (e.g. for .cjs files), and that needs detection logic. At the very basic level, 11ty needs to detect/be aware of the default module system of the codebase it is working on.not needed, see Gil's note below

--serve and --watch uses a different code path and is a bit more complicated.

If the sync require in @11ty/dependency-tree needs to be asyncified there's a pretty long cascade, but it can be done:

  • sync require() in getDependenciesFor() in @11ty/dependency-tree
  • called from getCleanDependencyListFor() (exported, recursive) in @11ty/dependency-tree
  • called from getJavaScriptDependenciesFromList(), addDependencies() in EleventyWatchTargets.js (all sync)
  • called from _initWatchDependencies() in Eleventy.js, but it's is already async!

The bigger issue here isn't the async nature of dynamic import()-s but that:

  • 11ty's dependency-tree relies on require.children to map out dependencies
  • require.cache is used for cache invalidation

As mentioned above, require.cache cannot be used for invalidation, but this can be worked around. Unfortunately there seems to be no way to access module resolution (and the list of dependencies) for ES modules, so this needs another solution. In my proof-of-concept I swapped out @11ty/dependency-tree for the npm dependency-tree package, that is mentioned in the README of that internal package, and I managed to configure it so that that it would provide the list of dependencies. It is also a sync call so this could be done without async-ifying everything upstream.

I would be happy to work/contribute to a more fleshed out solution of the above @zachleat if you think this is a viable direction.

@giltayar
Copy link

giltayar commented Feb 22, 2021

Small comment: I believe you don't have to deal differently (in terms of importing) with CJS and ESM, because Node.js allows you to use import to load CJS too. So just load everything with import.

Another 2 cents on cache-busting, which I implemented using a loader. I wrote a long technical note on it here: https://dev.to/giltayar/mock-all-you-want-supporting-es-modules-in-the-testdouble-js-mocking-library-3gh1

@alexpeattie
Copy link

alexpeattie commented May 3, 2021

Just a quick note that I think this issue will become more pressing as more popular libraries move over to being ESM only. One example I ran into recently is unist-util-visit (5M downloads per week on NPM) which is now ESM only so can't be used in an Eleventy project 😢 .

I'd be very happy to contribute to this, but I don't know if the maintainers have a sense yet of how this best should be tackled?

@nhoizey
Copy link
Contributor

nhoizey commented May 4, 2021

I now have the issue with slugify v2.0.0 also.

Sindre explains the situation here: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

@Zearin
Copy link

Zearin commented May 4, 2021

Look at this from the Node v16 docs:

https://nodejs.org/dist/latest-v16.x/docs/api/esm.html#esm_node_imports

mtsknn added a commit to koodikrapula/koodikrapula.fi that referenced this issue Jun 24, 2021
Kudos to Lenny Anders for his comment on GitHub:
11ty/eleventy#836 (comment)
@justinfagnani
Copy link
Contributor

justinfagnani commented Aug 20, 2021

Another idea of how to accomplish this while --experimental-vm-modules is still uncertain to move forward: Could 11ty watch .js files with something like nodemod and restart a child process every time they change? It'd be expensive compared to modifying the require cache, but it would only have to be done for .js files - templates, HTML, CSS, etc., could trigger a rebuild in the existing process. @zachleat

aomarks added a commit to lit/lit.dev that referenced this issue Sep 1, 2021
The lit-dev-tools package is currently CommonJS, because mostly it
is used for Eleventy plugins, and Eleventy doesn't support ES
modules (11ty/eleventy#836).

We want ES modules for this new redirect checker script, because it
needs to import some ES modules, and that is difficult to do with
TypeScript, because TypeScript doesn't allow emitting an actual
`import` statement, which is how CommonJS -> ESM interop works
(microsoft/TypeScript#43329).

We also an't really have a mix of CommonJS and ESM in the same package,
because the {"type": "module"} field has to be set. We could use
.mjs extensions, but TypeScript won't emit those.

So the simplest solution seems to be to just have two packages!
aomarks added a commit to lit/lit.dev that referenced this issue Sep 1, 2021
Custom script for checking lit.dev redirects.

It would be nice if we could use the 3rd party link checker we already have for this somehow, but it doesn't support checking for anchors (see stevenvachon/broken-link-checker#108 -- understandable since it would require DOM parsing) which is one of the main failure cases.

Fixes #467 (since we shouldn't need comments if we have the redirects checked in CI).

As part of this, I created a new lit-dev-tools-esm package.

The existing lit-dev-tools package is currently CommonJS, because mostly it is used for Eleventy plugins, and Eleventy doesn't support ES modules (11ty/eleventy#836).

We want ES modules for this new redirect checker script, because it needs to import some ES modules, and that is difficult to do with TypeScript, because TypeScript doesn't allow emitting an actual import statement, which is how CommonJS -> ESM interop works (microsoft/TypeScript#43329).

We also can't really have a mix of CommonJS and ESM in the same package, because the {"type": "module"} field has to be set to one or the other in the package.json. We could use .mjs extensions, but TypeScript won't emit those.

So the simplest solution seems to be to just have two packages.
@mahnunchik
Copy link

mahnunchik commented Dec 6, 2021

Any news in 1.0.0?

@klauss194
Copy link

klauss194 commented Mar 10, 2022

+1

@noelforte
Copy link

noelforte commented Apr 1, 2022

I've been following this for a bit and wanted to chime in; has anyone tried porting their 11ty site to pure ESM aside from using something like esm? Or is using a module loader like that the most elegant way to start moving towards ESM? I'm seeing a lot of modules I depend on start moving over to ESM and am trying to create a plan for how I eventually want to tackle this on my site, but not if there's official word from Zach et. al. contributors on the status of this...

One other thought I had that I'd be curious to know if any others have tried— taking advantage of Eleventy's new programmatic API and writing a short module that calls the API to handle Eleventy until official ports to ESM are out. Curious to know what people watching this thread think!

@j-f1
Copy link

j-f1 commented Apr 1, 2022

I’ve been using the import() feature in Node.js, which works well except that Eleventy can’t clear out the ESM cache, so editing an ESM file will not change your build until you restart the eleventy --serve command.

@giltayar
Copy link

giltayar commented Apr 2, 2022

@j-f1 just use a cache buster. Something like this: await import(./my-module.mjs?buster=${Math.random()}`). Now every time the import is called, it will not use the cache.

(not very efficient memory-wise, as the older modules are not GC-ed, but it will work)

@thescientist13
Copy link

thescientist13 commented Jun 20, 2022

Just wanted to share I ran into this issue while writing a plugin for 11ty similar to what was captured in #836 (comment), as the main dependency of the plugin uses ESM and has type="module" in package.json.

FWIW, I had to do something similar for cache busting when I migrated a medium-ish project of mine to ESM and had to solve this same problem myself. This thread is Node.js land was helpful and echos similar solutions posted here, and from my own experience can vouch for the Workers approach. Though I didn't significantly benchmark it, it has been solid.


Update: as a temporary measure, I am able to bundle my dependency as CJS with Rollup so for now will just ship a CJS version of that alongside ESM, but otherwise, it is now working with Eleventy! 🥳

@bennypowers
Copy link

bennypowers commented Aug 24, 2022

I have a work-in-progress branch at https://github.com/bennypowers/eleventy/tree/esm which converts the source to esm in a mostly-mechanical way. the parts which rely on calculating a module dependency graph based on the require cache do not work, and will likely require a fundamental redesign, unless i'm missing something (which i probably am)

But the source should be cjs to support require

that can be handled with a rollup build and package.json exports field

edit: if you want to take part, contributions on that branch are welcome as far as i'm concerned.

adilsonfsantos added a commit to adilsonfsantos/Portfolio-11ty that referenced this issue Sep 11, 2022
- Build wont work due to [11ty not supporting ESM](11ty/eleventy#836) need to wait until 2.0.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement needs-votes A feature request on the backlog that needs upvotes or downvotes. Remove this label when resolved. research-needed
Projects
Development

No branches or pull requests