-
Notifications
You must be signed in to change notification settings - Fork 27k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for transpile packages installed via npm #3319
Conversation
@giuseppeg This looks like a good approach. I'd like to see a test case as well. |
@@ -206,6 +218,10 @@ export default async function createCompiler (dir, { buildId, dev = false, quiet | |||
loader: 'emit-file-loader', | |||
include: [dir, nextPagesDir], | |||
exclude (str) { | |||
if (shouldTranspileModule(str)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This works for sure.
For the SSR version, we may need to rewrite the require('my-module')
pointing to this emitted file.
And if that requires a ES6+ module, how we are going to handle it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This works for SSR since node will look for .next/dist/node_modules
first and pick up the transpiled version in it.
And if that requires a ES6+ module, how we are going to handle it?
During my testing my-modules
was depending on another es6 module (in the same package) and both were transpiled and emitted to .next/dist/node_modules/my-modules
.
Later today or tomorrow (when I have time basically) I will add those to the PR together with the changes to the readme.
As I said this solution still doesn't work with linked packages though - I will try to tackle that at the end and see if it is easy/doable to fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This works for SSR since node will look for .next/dist/node_modules first and pick up the transpiled version in it.
That's pretty interesting. Nice work.
Looking forward to take this in.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@arunoda edit: in order for this to work we need to copy the package.json
over too. Do you have any idea how to do so? Or would you rather rewrite the imports/requires?
Do we want to support some sort of signal in the module config that it definitely won't have to be transpiled? For example, if you require a module that has a devDependency on |
@rauchg @giuseppeg transpiring nested modules is tricky. We look for Then this is clear. |
@arunoda added an example.
Currently when imported, nested modules are transpiled and this is problematic.
Need to escape the original pattern probably.
This is tricky, I wouldn't do that but rather have people add a regexp for it. |
I was trying this today. But I'm having errors. I think the SSR version is not using the transpiled version. Here's how I use this:
|
@arunoda yeah see my comment #3319 (comment) If you copy the |
@giuseppeg ah ha. |
@giuseppeg @arunoda Thanks for your work on this! I've been able to work around the underlying issue in my own projects by customizing the I took a slightly different approach in my own PR (#3470). There I've exposed a way to change the underlying |
@ScottPolhemus I think that your solution suffers from the same issues as the one from this PR. Is SSR working for you? Also I personally think that with your solution one would easily end up having a ton of code transpiled or emitted since it is easy to involuntarily write regexps that are too permissive – whitelisting most of the times is better than blacklisting. That said I might be missing something and be (happily) wrong. |
Actually @giuseppeg, unless I'm missing something I don't think the error that @arunoda ran into above is related to SSR after all. It looks like your example page (
The actual error message was pretty misleading, but with this change your example runs fine for me. I don't feel too strongly about using a whitelist vs. exposing the underlying |
Yup it works if you require the actual module. The problem though is that many people will rely on the
This is great, better expose an API so you can always change the underlying implementation. Otherwise you are locked to that solution and can't make a change without breaking stuff. |
By the way if you think that it could be useful we could discuss about an option to use the regexps for blacklisting i.e. make it configurable to support both. |
|
||
<p><details> | ||
<summary><b>Examples</b></summary> | ||
<ul><li><a href="./examples/with-es6-npm-module-pages">Transpile NPM modules</a></li></ul> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: fix this link
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
} | ||
|
||
const packageJsonToCopy = {} | ||
const copiedPackageJson = {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure if this is persisted across builds (when rebuilding)
@@ -28,6 +28,13 @@ module.exports = function (content, sourceMap) { | |||
callback(null, code, map) | |||
} | |||
|
|||
if (query.copyPackageJson) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should/could be a separate loader
server/build/webpack.js
Outdated
function getPackageJsonPath (filePath) { | ||
let filePathDirname = dirname(filePath) | ||
if (filePathDirname === dir) { | ||
return join(dir, 'package.json') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to check whether this file exists or probably return undefined
in this case.
Is this PR functional @giuseppeg ? or still some work to be fully functional ? |
@maxs15 functional? yep. Done? Not sure, it needs to be reviewed, polished and test for a few edge cases. Feel free to try it out locally. |
Ok awesome work @giuseppeg, waiting for that feature for a long time ! |
@giuseppeg do you think there is any way of being able to transpile a file outside of node_modules ? Thanks |
It works but the issue that I have now is that the SSR doesn't require the compiled file in the .next directory. |
This would be great if it could do that, as things like Yarn workspaces should then be easier to work. |
Thanks to @giuseppeg’s work in #3319
* Speed up next build * Document webpack config * Speed up next build * Remove comment * Add comment * Clean up rules * Add comments * Run in parallel * Push plugins seperately * Create a new chunk for react * Don’t uglify react since it’s already uglified. Move react to commons in development * Use the minified version directly * Re-add globpattern * Move loaders into a separate variable * Add comment linking to Dan’s explanation * Remove dot * Add universal webpack * Initial dev support * Fix linting * Add changes from Arunoda's work * Made next dev works. But super slow and no HMR support. * Fix client side hot reload * Server side hmr * Only in dev * Add on-demand-entries client + hot-middleware * Add .babelrc support * Speed up on demand entries by running in parallel * Serve static generated files * Add missing config in dev * Add sass support * Add support for .map * Add cssloader config and fix .jsx support * Rename * use same defaults as css-loader. Fix linting * Add NoEmitErrorsPlugin * Add clientBootstrap * Use webpackhotmiddleware on the multi compiler * alpha.3 * Use babel 16.2.x * Fix reloading after error * Remove comment * Release 5.0.0-univeral-alpha.1 * Remove check for React 16 * Release 5.0.0-universal-alpha.2 * React hot loader v4 * Use our static file rendering machanism to serve pages. This should work well since the file path for a page is predictable. * Release 5.0.0-universal-alpha.3 * Remove optional loaders * Release 5.0.0-universal-alpha.4 * Remove clientBootstrap * Remove renderScript * Make sure pages bundles are served correctly * Remove unused import * Revert to using the same code as canary * Fix hot loader * Release 5.0.0-universal-alpha.5 * Check if externals dir exist before applying config * Add typescript support * Add support for transpiling certain packages in node_modules Thanks to @giuseppeg’s work in #3319 * Add BABEL_DISABLE_CACHE support * Make sourcemaps in production opt-in * Revert "Add support for transpiling certain packages in node_modules" This reverts commit d4b1d9b. In favor of a better api around this. * Support typescript through next.config.js * Remove comments * Bring back commons.js calculation * Remove unused dependencies * Move base.config.js to webpack.js * Make sure to only invalidate webpackDevMiddleware one after other. * Allow babel-loder caching by default. * Add comment about preact support * Bring back buildir replace * Remove obsolete plugin * Remove build replace, speed up build * Resolve page entries like pages/day/index.js to pages/day.js * Add componentDidCatch back * Compile to bundles * Use config.distDir everywhere * Make sure the file is an array * Remove console.log * Apply optimization to uglifyjs * Add comment pointing to source * Create entries the same way in dev and production * Remove unused and broken pagesGlobPattern * day/index.js is automatically turned into day.js at build time * Remove poweredByHeader option * Load pages with the correct path. * Release 5.0.0-universal-alpha.6 * Make sure react-dom/server can be overwritten by module-alias * Only add react-hot-loader babel plugin in dev * Release 5.0.0-universal-alpha.7 * Revert tests * Release 5.0.0-universal-alpha.10 * Make sure next/head is working properly. * Add wepack alias for 'next' back. * Make sure overriding className in next/head works * Alias react too * Add missing r * Fragment fallback has to wrap the children * Use min.js * Remove css.js * Remove wallaby.js * Release 5.0.0-universal-alpha.11 * Resolve relative to workdir instead of next * Make sure we touch the right file * Resolve next modules * Remove dotjsx removal plugins since we use webpack on the server * Revert "Resolve relative to workdir instead of next" This reverts commit a13f3e4. * Externalize any locally loaded module lives outside of app dir. * Remove server aliases * Check node_modules reliably * Add symlink to next for tests * Make sure dynamic imports work locally. This is why we need it: https://github.com/webpack/webpack/blob/b545b519b2024e3f8be3041385bd326bf5d24449/lib/MainTemplate.js#L68 We need to have the finally clause in the above in __webpack_require__. webpack output option strictModuleExceptionHandling does that. * dynmaic -> dynamic * Remove webpack-node-externals * Make sure dynamic imports support SSR. * Remove css support in favor of next-css * Make sure we load path from `/` since it’s included in the path matching * Catch when ensurepage couldn’t be fulfilled for `.js.map` * Register require cache flusher for both client and server * Add comment explaining this is to facilitate hot reloading * Only load module when needed * Remove unused modules * Release 5.0.0-universal-alpha.12 * Only log the `found babel` message once * Make sure ondemand entries working correctly. Now we are just using a single instance of OnDemandEntryHandler. * Better sourcemaps * Release 5.0.0-universal-alpha.13 * Lock uglify version to 1.1.6 * Release 5.0.0-universal-alpha.14 * Fix a typo. * Introduce multi-zones support for mircofrontends * Add section on css
* Speed up next build * Document webpack config * Speed up next build * Remove comment * Add comment * Clean up rules * Add comments * Run in parallel * Push plugins seperately * Create a new chunk for react * Don’t uglify react since it’s already uglified. Move react to commons in development * Use the minified version directly * Re-add globpattern * Move loaders into a separate variable * Add comment linking to Dan’s explanation * Remove dot * Add universal webpack * Initial dev support * Fix linting * Add changes from Arunoda's work * Made next dev works. But super slow and no HMR support. * Fix client side hot reload * Server side hmr * Only in dev * Add on-demand-entries client + hot-middleware * Add .babelrc support * Speed up on demand entries by running in parallel * Serve static generated files * Add missing config in dev * Add sass support * Add support for .map * Add cssloader config and fix .jsx support * Rename * use same defaults as css-loader. Fix linting * Add NoEmitErrorsPlugin * Add clientBootstrap * Use webpackhotmiddleware on the multi compiler * alpha.3 * Use babel 16.2.x * Fix reloading after error * Remove comment * Release 5.0.0-univeral-alpha.1 * Remove check for React 16 * Release 5.0.0-universal-alpha.2 * React hot loader v4 * Use our static file rendering machanism to serve pages. This should work well since the file path for a page is predictable. * Release 5.0.0-universal-alpha.3 * Remove optional loaders * Release 5.0.0-universal-alpha.4 * Remove clientBootstrap * Remove renderScript * Make sure pages bundles are served correctly * Remove unused import * Revert to using the same code as canary * Fix hot loader * Release 5.0.0-universal-alpha.5 * Check if externals dir exist before applying config * Add typescript support * Add support for transpiling certain packages in node_modules Thanks to @giuseppeg’s work in #3319 * Add BABEL_DISABLE_CACHE support * Make sourcemaps in production opt-in * Revert "Add support for transpiling certain packages in node_modules" This reverts commit d4b1d9b. In favor of a better api around this. * Support typescript through next.config.js * Remove comments * Bring back commons.js calculation * Remove unused dependencies * Move base.config.js to webpack.js * Make sure to only invalidate webpackDevMiddleware one after other. * Allow babel-loder caching by default. * Add comment about preact support * Bring back buildir replace * Remove obsolete plugin * Remove build replace, speed up build * Resolve page entries like pages/day/index.js to pages/day.js * Add componentDidCatch back * Compile to bundles * Use config.distDir everywhere * Make sure the file is an array * Remove console.log * Apply optimization to uglifyjs * Add comment pointing to source * Create entries the same way in dev and production * Remove unused and broken pagesGlobPattern * day/index.js is automatically turned into day.js at build time * Remove poweredByHeader option * Load pages with the correct path. * Release 5.0.0-universal-alpha.6 * Make sure react-dom/server can be overwritten by module-alias * Only add react-hot-loader babel plugin in dev * Release 5.0.0-universal-alpha.7 * Revert tests * Release 5.0.0-universal-alpha.10 * Make sure next/head is working properly. * Add wepack alias for 'next' back. * Make sure overriding className in next/head works * Alias react too * Add missing r * Fragment fallback has to wrap the children * Use min.js * Remove css.js * Remove wallaby.js * Release 5.0.0-universal-alpha.11 * Resolve relative to workdir instead of next * Make sure we touch the right file * Resolve next modules * Remove dotjsx removal plugins since we use webpack on the server * Revert "Resolve relative to workdir instead of next" This reverts commit a13f3e4. * Externalize any locally loaded module lives outside of app dir. * Remove server aliases * Check node_modules reliably * Add symlink to next for tests * Make sure dynamic imports work locally. This is why we need it: https://github.com/webpack/webpack/blob/b545b519b2024e3f8be3041385bd326bf5d24449/lib/MainTemplate.js#L68 We need to have the finally clause in the above in __webpack_require__. webpack output option strictModuleExceptionHandling does that. * dynmaic -> dynamic * Remove webpack-node-externals * Make sure dynamic imports support SSR. * Remove css support in favor of next-css * Make sure we load path from `/` since it’s included in the path matching * Catch when ensurepage couldn’t be fulfilled for `.js.map` * Register require cache flusher for both client and server * Add comment explaining this is to facilitate hot reloading * Only load module when needed * Remove unused modules * Release 5.0.0-universal-alpha.12 * Only log the `found babel` message once * Make sure ondemand entries working correctly. Now we are just using a single instance of OnDemandEntryHandler. * Better sourcemaps * Release 5.0.0-universal-alpha.13 * Lock uglify version to 1.1.6 * Release 5.0.0-universal-alpha.14 * Fix a typo. * Introduce multi-zones support for mircofrontends * Add section on css
}) | ||
} | ||
|
||
function getPackageJsonPath (filePath) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider using https://www.npmjs.com/package/find-root instead
function getPackageJson (filePath) {
const pkgRoot = findRoot(filePath)
const packageJsonPath = path.resolve(pkgRoot, 'package.json')
return packageJsonPath
}
I'm currently refactoring into next.js. I had this working previously by deleting the This is how I'm transpiling node_modules by patching in config.module.rules = config.module.rules.map(rule => {
if (rule.use && rule.use.loader === 'babel-loader') {
delete rule.exclude
rule.include = (filePath) => {
const shouldTranspile = pathIsInside(filePath, __dirname) || hasPkgModule(filePath)
return shouldTranspile
}
}
return rule
}) Rather, I'm trying to do this. I was doing the same thing in my webpack config previously and it was working fine. I confirmed the include rule is returning true for the module I'm trying to transpile but I'm getting Maybe consider providing an option to transpile based on package.json.module also? |
@brettstack with Next 5 and universal webpack probably we need to revisit this solution. I think that many things can go wrong when one tries to transpile external dependencies so I would understand if the folks at Zeit decided to discard this idea. Anyway are you trying to patch Next 5's? |
You can use #3732 |
Maybe helpful, this is how meteor tackles the problem in it's newest beta: |
@timneutkens feel free to close this PR if you think that this solution won't work with the universal webpack rework |
👍 |
Fixes #706
Emits the transpiled modules in
.next/dist/node_modules
so that it works for SSR'd pages.@arunoda / @nkzawa in case you like this solution I'll finish the PR.
If you really want I can try to make symlinks work too.