New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[@rollup/plugin-commonjs] Circular dependency is not resolved optimally by plugin-commonjs for protobufjs causing runtime error #988
Comments
From reading the code it seems that commonjs plugin processes modules one-file at a time, so it doesn't really have the ability to detect cycles during its execution. The breakup of cycles happens upstream in rollup itself ( |
However the code generated by commonjs plugin begs for a question whether it should actually use dynamic import in cases where
the plugin generates a top level imports block with
and
I suppose conversion of requires into imports should be done per scope (top-level vs functions) and for the latter might need to use dynamic import e.g.
and in that case the sorting of modules could better resolve the araising circular dependencies, since actually the static imports do not have a cycle here, it's only the dynamic imports that create them. |
I also encountered this problem when using Vite to build project. Is there any way to solve? ❤❤ |
According to the prompted https://github.com/mmacfadden/rollup-protobufjs#notes // vite config
export default defineConfig({
resolve: {
alias: [
{
find: 'protobufjs',
replacement: 'protobufjs/dist/protobuf.min.js',
}
],
}
}); |
This is just switching to a subset of the library, which doesn't trigger the problem mentioned here. I was also using a |
The example works if you replace the commonjs plugin with the alternative cjs plugin found at: rollup/rollup#4195 (comment) Demonstration using a slightly more interesting test case: diff --git a/index.js b/index.js
index 2383a47..a9e06a7 100644
--- a/index.js
+++ b/index.js
@@ -4,0 +5 @@ const root = protobuf.Root.fromJSON(awesome);
+toMessage("Hello").forEach(x => console.log(x, JSON.stringify(String.fromCharCode(x))));
Note that protobuf uses Bundle size using
Bundle size using
Edit: corrected |
@kzc it looks promising, any hint on how to solve errors like this (from my larger project) with the custom plugin? should I add aliases or something?
|
I've never used |
It's also possible to use both |
I just verified that specifying native module name(s) in the the
|
Thanks, the options ( It failed however on a different |
Again, I don't know anything about these libraries, but it looks like https://github.com/grpc/grpc-node/blob/master/PACKAGE-COMPARISON.md Have you tried using According to https://github.com/grpc/grpc-node/tree/master/packages/proto-loader documentation: Usageconst protoLoader = require('@grpc/proto-loader');
const grpcLibrary = require('grpc');
// OR
const grpcLibrary = require('@grpc/grpc-js'); Out of curiosity, how was your system successfully built before? Webpack or esbuild? |
I'm actualy using As for my build process - the circular dependency problem I know webpack handles properly to bundle the For the project experiencing this latest problem I don't bundle the files, since it's an Electron app that can just load individual files from disk on demand - I suspect webpack would still handle it well though. |
So far an isolated test case using It would be nice to have the proper handling of circular dependencies incorporated into |
We are finalizing a new version of the commonjs plugin in #1038. It is pre-published as |
@lukastaegert I tried to apply a change of
to the reproduction repository https://github.com/kskalski/rollup-protobufjs, but the effect is that build step no longer produces any output without really emitting any errors (the step |
Yes, you also need to have the very latest node-resolve. I will include a check for that in the final version. |
Ok, updating node-resole fixed the no-emit problem. However the old error caused by uninitialized object related to order of imports evaluation still remains. I pushed packages update to the repo. |
The test repo appears to have been updated with the wrong version of the commonjs plugin. When this patch is applied to the test repo in https://github.com/kskalski/rollup-protobufjs and using the latest node-resolve plugin, the new commonjs plugin will produce output just 10% of the time. Non-determinisitic output is produced when it does work. diff --git a/index.js b/index.js
index 2383a47..9a35b5e 100644
--- a/index.js
+++ b/index.js
@@ -25 +25,3 @@ export function toMessage(awesomeField) {
-}
\ No newline at end of file
+}
+
+console.log(JSON.stringify(Array.from(toMessage("Hello")).map(x=>[x,String.fromCharCode(x)])));
diff --git a/package.json b/package.json
index ed00ee3..415fd8d 100644
--- a/package.json
+++ b/package.json
@@ -17 +17 @@
- "@rollup/plugin-commonjs": "22.0.0.0",
+ "@rollup/plugin-commonjs": "22.0.0-0", Using
Using
(A rollup warning about the of the use of Using
Using commonjs plugin with
When the new commonjs plugin fails, rollup will exit entirely without errors or warnings and without any output, and with a 0 exit code - incorrectly indicating success. I was unable to isolate how an error escapes from being caught and reported. Before the bug in the plugin is fixed it would be helpful if rollup could detect such anomalous scenarios and report an error. Edit: updated second example to correct a typo in the option name |
Thanks, I updated the repo following your patch and can confirm non-deterministic behavior of |
Thanks for spotting, there was a serious issue with the cycle detection logic. This should be fixed now, I released a new version 22.0.0-1. Please also look if there is still indeterminate output. |
Thanks, the issue coming from circular dependency seems to be resolved with 22.0.0-1 |
@kskalski I noticed that kskalski/rollup-protobufjs@d985b9f in your test case also works with Just curious... does |
Indeed an extended test case with using At this point I didn't test the new plugin version in my original Electron project, since the problem appears in a standalone case with just trying to import |
I can confirm that https://github.com/kskalski/rollup-protobufjs/tree/grpc works with
but fails with
This new @lukastaegert Is there any way you can detect when a plugin causes rollup itself to exit without error or warning and produce no output? |
I fear this is not possible. The reason is that the event loop is basically exhausted while the build is not done, there is no error involved. Basically you have promises waiting for each other in a circular way. This can happen if you use a This is something the plugin actually does, but I thought I finally have the cycle detection figured out, obviously this is not the case. Maybe I need to rethink the entire algorithm here. The previous issue was that I had assumed that the modules in a cycle are loaded in the "right" order, but this is certainly not the case. Will need to see what the new issue is about. Another point to fix is that the cycle detection is always running while it should only be running when strictRequires is "auto". |
Do you think this plugin is reaching a fundamental limit of in-flight Promises causing NodeJS to exit unexpectedly? Or is it just a case of NodeJS will exit when there are no more pending events to process? If the latter, perhaps an artificial pending event can be introduced to gate the process. |
I think this is the case. What we could do is indeed introduce something timeout based, but this will not solve the underlying issue, in the best case it will just "hang" instead of exiting. |
I think I fixed the issue now. I assume it was caused by some premature optimizations I built into the original algorithm. Instead of trying to debug this, I switched now to a much simpler cycle detection algorithm that does not try to cache too much. I worked fine for the example given (and also in a complex project of my own). Published as @rollup/plugin-commonjs@22.0.0-2 and also updated the "beta" tag. |
@lukastaegert I just wanted to chime in and mention that the beta version of the commonjs plugin fixed a circular dependency issue for me! I was attempting to build a third-party React package with Rollup. The package uses styled-components, and it was crashing when it hit circular dependencies. I was able to build the package with webpack but not Rollup. Updating to the beta version allowed me to build the package with Rollup. Thanks for the great work! |
I'd also like to chime in with another example. I tried switching a CRA project to Vite and encountered it breaking in the production build because of this bug happening to xmlbuilder. https://github.com/dantman/rollup-commonjs-circular-xmlbuilder-bug I tested it with the beta and it fixed the issue. So I am encouraged there is a solution to it and ~~all it requires is Vite to update a package. However, I am extremely discouraged by the fact that this confirmed fix for a bug that makes rollup unusable to users of certain libraries has been "beta" for 3.5 months. I can't report this as a bug and ask them to bump the package, because the package doesn't even have an official release. And because it's a major and the dependency is indie Vite I can't even workaround this by installing the beta in my package as Vite will continue to use the older version. Edit: Edit2: I retract my retraction. Vite does use |
Expected Behavior
Code runs without error, in particular code from protobufjs
object.js
is executed before its reference infield.js
thus avoiding use of uninitialized variableReflectionObject
Circular dependency reported during build, but resolved in favor of executing required (unconditionally specified at root of the module) dependencies first before "loose" dependencies (
require
s placed in function bodies / conditionals?)Actual Behavior
Error during runtime
The order of circular dependency resolution is not optimal, since modules that are not on critical path are executed before those on critical path.
Additional Information
In particular I tracked the offending cycle as
node_modules\protobufjs\src\object.js -> node_modules\protobufjs\src\util.js -> node_modules\protobufjs\src\type.js -> node_modules\protobufjs\src\namespace.js -> node_modules\protobufjs\src\object.js
however:
object.js
has a hard dependency at topvar util = require("./util");
,namespace.js
has hard dedepdency at topvar ReflectionObject = require("./object");
util.js
has only loose dependencies ontype.js
(and alsoenum.js
androot.js
), like those:so in fact the optimal breaking of the cycle would be something like:
type.js -> namespace.js -> object.js -> util.js
(
util.js
thus executed first,type.js
last)The text was updated successfully, but these errors were encountered: