-
-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add esm experimental-loader support to @babel/register #15456
base: main
Are you sure you want to change the base?
Conversation
Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/54047/ |
i came across ./packages/babel-core/src/vendor/import-meta-resolve.js... maybe that has better import resolution code. |
hrm... so support goes back to node 12 biggest change is |
the loader seems to be in a different context, i'm handling this now (tests aren't done). cjs loading is explicitly not supported by esm loaders and needs to be handeled by the application context (aka the usual way). for cjs workers will likely be needed (setup/config can be done via messageport) but the esm part likely won't need any workers, since the loader itself will be the worker, so it may make sense to get rid of that eventually |
Gulpfile.mjs
Outdated
@@ -705,6 +705,12 @@ gulp.task( | |||
gulp.series("copy-dts", () => buildRollupDts(dtsBundles)) | |||
); | |||
|
|||
gulp.task("build-other", () => | |||
getFiles(`packages/babel-register/src/loader.mjs`, {}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no idea how to make it keep just this particular file as ESM. theoretically since I'm re-exporting via loader.mjs this might aswell be a cjs file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We initially wanted to wait until Node.js stabilizes their API, but since it's taking so long I'm ok with relying on the current experimental one.
packages/babel-register/loader.mjs
Outdated
export const resolve = major >= 16 && _resolve; | ||
export const globalPreload = major >= 16 && _globalPreload; | ||
|
||
export const getSource = major < 16 && _getSource; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is all experimental anyway, I would prefer to only support Node.js 16&18 and simplify our implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The backwards compatibility - at least down to node 14 - consists of renamed functions only plus getFormat, which is 5 lines.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note also about this particular snippet: What I tried to accomplish was not exposing the old functions on node 16+, since that causes unnecessary warnings about old functions being used. But node 14 didn't like it.
Edit: fixed it now
edit: tests fixed |
I've released a built package here if anyone wants to try: https://github.com/mkg20001/babel/releases It worked on the node 14 project i tried it on, but might break on some edge case that I'd like to know about |
ah windows, forgot about that... most likely just path seperator stuff and node 12 doesn't do await in top-level |
Sorry I didn't fully understand what you mean, can we use similar usage?
Don't worry, I can help if you don't have a windows device. |
// skip everything if below version 12 | ||
if (major < 12) { | ||
// eslint-disable-next-line no-global-assign | ||
it = it.skip; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use a new variable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ide doesn't detect it as mocha test then, for running it from ui
|
||
flag[resolved] = true; | ||
|
||
// console.log('map', resolved) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: remove
Co-authored-by: liuxingbaoyu <30521560+liuxingbaoyu@users.noreply.github.com>
we already do, because we also use environment variables to configure. we can autoload .babelrc like babel-node, but maybe we can just extend babel-node to support esm aswell. edit: babel-node is the tool that would have a usage most similar to esm-loader's cli |
if (type) return type; | ||
// Otherwise, (if not at the root) continue checking the next directory up | ||
// If at the root, stop and return false | ||
return dir.length > 1 && getPackageType(resolvePath(dir, "..")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is likely going to break on windows, since dir will end up being d: or something. is there a function to detect if we have a root folder for a certain drive letter? or just dir.split(path.sep).length === 1?
edit: otherwise this will loop forever, calling path.dirname and expecting it to return just "/" eventually
} | ||
} | ||
|
||
export async function resolve(specifier, context, nextResolve) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we handle experimental network imports? (aka http(s):// as import path)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you expand on why we need a resolve
hook? Babel should only care about transforming the already resolved files, delegating to Node.js for the actual resolution.
This is what already happens with the require
hook, where we only hook into the loading process and not in the resolution process.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it is necessary. According to my understanding, esm-loader allows multiple and sequential processing, and https
has another loader to provide support.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, I don't think we should really hook into how resolve
behaves. The Node.js docs provided an example coffeescript ESM loader: https://nodejs.org/api/esm.html#transpiler-loader, maybe we can start from that example.
const ext = extname(url); | ||
if (ext === ".cjs") return "commonjs"; | ||
if (ext === ".mjs") return "module"; | ||
const isFilePath = !!ext; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is from the docs and is based on the simple assumption that directories don't have file extensions, i guess we should stat here (and maybe cache the stat call)... but since the call below is perfectly fine with the file being not found it may be not necesarry
how is a file with .cjs extension supposed to be handled by babel? we can autodetect the type and pass it along as parameter to the worker. currently it just detects it as a module because that's the result we get from the worker after transformation. we need to get the type from the worker after transformation, as the code might be transpiled to commonjs (in the case of the loader this would mean that we'll return commonjs as the type and node then uses the pirates hook to load the file) |
}; | ||
|
||
if (env.BABEL_REGISTER) { | ||
const { default: opts } = await import(pathToFileURL(env.BABEL_REGISTER)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this breaks on node 12
/* eslint-disable import/extensions */ | ||
|
||
// this is a workarround to make sure global._BABEL_ESM_REGISTER is set to true before anything else gets loaded | ||
import { readFile, access } from "node:fs/promises"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nodejs/node#37583
nodejs/node#38006
I personally prefer to use the sync version, because it is more convenient to use and there is no obvious performance loss.
NOTE: This is just my opinion, I'm not sure what other reviewers think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The async version allows Node.js to call out loader with multiple files in parallel, but if the promise-based version is so much slower we could use the callback-based version.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There should be little difference in the new version of nodejs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Initially I went with not using it because I thought existsSync was deperacted. turns out only exists is, not existsSync. But having it async if it doesn't make a difference speed-wise is better then.
async function exists(path) { | ||
try { | ||
await access(path, constants.R_OK); | ||
return true; | ||
} catch (error) { | ||
// do nothing | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/nodejs/node/blob/e5c1fd7a2a1801fd75bdde23b260488e85453eb2/lib/fs.js#L290-L308
Looks like existsSync
is better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 for existsSync
. Node.js had an asynchronus exist
API, that had been deprecated because checking if a file exists was effectively a sync operation anyway.
Files that are modules could be transformed into commonjs, the user might
want to do things like not specify a file extension (as is common in
typescript) and resolve needs to account for that, etc.
liuxingbaoyu ***@***.***> schrieb am Mo., 6. März 2023, 15:19:
… ***@***.**** commented on this pull request.
------------------------------
In packages/babel-register/src/loader.mjs
<#15456 (comment)>:
> + return next.url;
+ } catch (error) {
+ // do nothing
+ }
+}
+
+async function exists(path) {
+ try {
+ await access(path, constants.R_OK);
+ return true;
+ } catch (error) {
+ // do nothing
+ }
+}
+
+export async function resolve(specifier, context, nextResolve) {
I don't think it is necessary. According to my understanding, esm-loader
allows multiple and sequential processing, and https has another loader
to provide support.
—
Reply to this email directly, view it on GitHub
<#15456 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3AO2J5EORR6V7T5IZTQ43W2XW67ANCNFSM6AAAAAAVFABQTA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
I'd prefer to not to this (especially by default), and instead stick to the default semantics of Node.js. If users want different behaviors, they can use multiple loaders for that. Babel should only handle transpilation. |
Per spec things first go through resolve and that then returns whether the file is commonjs or a module (with load only being called for format=module), so we need to return it there anyways. It is planned to allow CJS loading with experimental loaders eventually. We need our own resolve as nodeJS's resolve won't resolve for example a .ts file, because that's not an extension it recognizes as valid. It'll just throw ERR_UNKNOWN_FILE_EXTENSION.
https://nodejs.org/api/esm.html#transpiler-loader But that aside, typescript can be compiled to both cjs and esm both under the .ts extension, depending on which settings are used. Say you have a bunch of typescript in a package that is "type": "module", that is where that code is really necesarry. esm-loader has it's own resolve function aswell. https://github.com/esbuild-kit/esm-loader/blob/develop/src/loaders.ts#L98 Also, typescript can be written with import statements, but then end up being compiled to cjs in a package that isn't "type": "module". (Typescript is just a good example, it applies to pretty much everything) I'd be open to taking the esm-loader logic as a base or having something like pirates but for esm that handles the more complicated parts and simply gives us files to transpile, but until that exists we need our own resolve logic. I tried to re-use as much as possible directly from the node internals, ending up with what is currently this loader's resolve function. Ain't pretty but it works. |
Fixes #1, Fixes #2
This adds experimental-loader/ESM support to @babel/register, which is a much wanted feature
I'm aware of experimental-loader being unstable, I'd still find it cool if it would be available at least behind a flag. Since it has to be loaded with --experimental-loader it should be clear that it's prone to breakage.
The code is basically "let's just get it working". Unstructured first attempt, for the project I tested it on it appears to work.
Since there is no way I'm aware to load @babel/register "the old way" and specify the config along, I added an environment variable which can be used to specify one. I'm open to better solutions. (edit: there's now a way to do it via ipc between loader and application, but it may be async and BABEL_REGISTER automatically preloads config. for esm configs there's --import, but it's not in any lts yet nodejs/node#40110)
BABEL_REGISTER=$PWD/register.mjs node --experimental-loader @babel/register/loader.mjs src/bin.ts