Replies: 6 comments 6 replies
-
very nice article, hope you all can take rspack further |
Beta Was this translation helpful? Give feedback.
-
gained much |
Beta Was this translation helpful? Give feedback.
-
提一个小问题,复用ast 节省时间会多吗。我之前有在webpack里尝试复用,但后续发现parse ast耗时不长,主要是操作ast的时间占了大头 |
Beta Was this translation helpful? Give feedback.
-
很感同身受的一篇文章 👍🏻,特别是从服务上层的视角,阐述各个工具的优缺点那一大段 |
Beta Was this translation helpful? Give feedback.
-
有点不理解的是:都是对模块的处理。为什么webpack要硬拆分成loader和plugins。我认为rollup的概念更清晰。就只需要了解插件API的各个生命周期就行了。这样对编写自定义插件上手难度更低🤔 |
Beta Was this translation helpful? Give feedback.
-
One of the best articles I've ever read! |
Beta Was this translation helpful? Give feedback.
-
chinese version
Before embarking on the development of Rspack, we explored various build tools and frameworks, including extensive use of Webpack, Vite, esbuild, and Rollup in real-world production environments.
To provide some context, our team, known as the WebInfra Team, is responsible for overseeing the company's suite of front-end build tools and frameworks. Some of these are open-source, while others are proprietary. Our portfolio includes:
ModernJS Builder: A build engine for common front-end applications
Garfish & Vmok: Universal solutions for micro frontends
ModernJS Framework: A progressive React framework
PIA: A high-performance web native development framework
Module Tools: A scheme for building common libraries
Rspress: A documentation solution
Lynx Speedy: A build tool for the cross-platform Lynx framework
Web Doctor: A diagnostic analysis tool for builds
This background informs our perspective on the design trade-offs involved in bundler technology and why we decided to create Rspack.
The Complexity of Underlying Build Tools: Daily Challenges and Operational Differences
The underlying build tools in all these frameworks and utilities are complex. A significant portion of our daily on-call responsibilities involves addressing user issues related to these build processes.
Key Differences Between Internal Infra Team and Open-Source Community Operations:
Scope of Responsibility: Open-source teams often focus on single-point solutions like Next.js or React-Native. In contrast, our team has a broader mandate. We aim to manage multiple solutions cost-effectively, minimize user switching costs between frameworks and tools, and facilitate the integration of various solutions, such as supporting both SSR and micro frontends.
On-Call Obligations for Business Teams: Unlike just addressing issues, our on-call duty requires rapid business response. Most problems are resolved within 24 hours, and almost all are solved within a week. This dual focus necessitates high-speed iterations for our solutions while also enabling us to address business-side issues in a cost-effective manner.
Challenges with Webpack: Performance and Debugging
Our initial large-scale deployment of build tools heavily relied on Webpack, a trend that continues with our open-source Modern.js project. Webpack's scalability is its strongest suit, accommodating almost all our build requirements. However, it comes with its own set of challenges.
Debugging Woes
Webpack operates much like a black box, making debugging a cumbersome process. When the business team encounters build-related issues, troubleshooting becomes a significant hurdle. This often requires intervention from the Infra Team, increasing the on-call pressure on them. This was a key motivator behind the development of Web Doctor, aimed at alleviating this on-call stress.
Performance Limitations
Performance has always been a sticking point for Webpack. We've experimented with various optimization strategies, including swc-loader, esbuild-loader, thread-loader, cache-loader, MFSU, and Persistent Cache. While these solutions may offer some relief, they fall short in handling large projects efficiently. Moreover, these optimizations often make the build process even more opaque. For example, Persistent Cache relies heavily on well-configured business build dependencies, esbuild-loader lacks support for ES5 downgrades, and cache-loader can lead to outdated products if the cache isn't cleared properly.
Challenges with Dual-Engine Approach: Webpack and Vite
The dual-engine approach, where the underlying engine can switch between Vite and Webpack, seemed promising initially. It offered a unified configuration and plugin layer, which partially addressed the performance issues we faced with Webpack. However, this approach introduced a new set of complexities.
Plugin Reusability
The plugin mechanisms for Rollup (used by Vite) and Webpack are fundamentally different. While there are attempts to create universal plugins, like
unplugin
, these solutions are still in their infancy and lack the robustness to support complex frameworks like Modern.js. This leads to a codebase filled with conditional logic to load different plugins based on the configuration, making it less maintainable.Performance in Large Projects
Vite's performance in large-scale projects is not up to the mark. The overhead from thousands of network requests during development can lead to significant delays, especially during hot module reloading (HMR). Rollup's performance also leaves room for improvement, sometimes even falling behind Webpack, which has the advantage of a persistent cache.
Inconsistency Between Development and Production
Many internal teams end up using Vite for development and Webpack for production due to Rollup's limitations in product optimization. This creates a significant divergence between development and production environments, making it harder to ensure consistent behavior.
In summary, while the dual-engine approach solves some problems, it introduces new challenges that make it less than ideal for our needs. We're still in search of a more holistic solution that can offer both performance and flexibility without compromising on either.
Rollup: Strengths and Weaknesses
We've extensively used Rollup in library build and early Lynx build scenarios, revealing both its strengths and weaknesses.
Strengths:
Weaknesses:
watch
feature is mediocre.Esbuild
Esbuild has effectively addressed two major shortcomings we encountered with Rollup: CommonJS support and performance. Esbuild considers CommonJS as a first-class citizen, providing robust support for it. Additionally, esbuild's performance is excellent, making it a strong alternative to Rollup, particularly for library builds. This is why tools like tsup, which is built on esbuild, are considered better alternatives to tsdx, which relies on Rollup. It's also worth noting that our ModernJS Module Tools are currently using esbuild as the underlying layer.
However, our experience with esbuild has also revealed several challenges.
onTransform hook
makes it difficult to chain different transform extensions. This limitation also affects post-processing chunks, as there's norenderChunk hook
for tasks like custom minification.onLoad
andonResolve
hooks. This results in an O(n) complexity problem in large projects.Load
phase, which introduces compile-time overhead.You can see these struggles in remixDue to these challenges, we've decided to focus on developing a Rust-based bundler. The goal is to create a tool that combines the strengths of Webpack, Rollup, and esbuild while addressing their respective weaknesses, providing a more efficient and robust build process.
How We Developed Rspack
Initially, our approach to developing the Rust Bundler was not to emulate Rust Webpack. Instead, we aimed to address some of the existing issues with esbuild. Our starting point was essentially a Rust version of esbuild and Rollup, but with built-in support for HMR, CommonJS, and Bundle Splitting. Given that we had already encapsulated a complete framework based on esbuild and had run it in a production environment for a year, we wanted to maintain compatibility with esbuild's interface while resolving its limitations in HMR and bundle splitting. The 'legacy' branch of Rspack still retains this original design, aimed at being compatible with the esbuild interface.
However, as we delved deeper into the development process, completing the first version of our Rust Bundler based on the Rollup architecture and reaching the project's Proof of Concept stage, we encountered a series of issues. These challenges forced us to reevaluate our architectural choices, ultimately leading us to adopt an architecture more aligned with Webpack.
This shift was driven by the need for a more robust and flexible solution, capable of addressing the complexities and varied requirements we had identified. The Webpack architecture offered us the extensibility and customization options that were lacking in our initial approach, making it a more suitable foundation for Rspack.
First-Class Citizen Approach in a Language-Agnostic Manner
Rollup's architecture is designed to support only ESM as a first-class citizen. This means that other module systems like CommonJS, or even non-JS module systems, need to be converted to ESM to be compatible. This conversion process introduces a host of issues. One prevalent problem is the inconsistent resolve logic across different module systems. Other inconsistencies include default values for
sideEffects
, chunk generation logic, and more.Because Rollup converts all modules into ESM, it doesn't adequately distinguish between different types of modules at its core layer. This lack of differentiation forces developers to rely on plugins and employ hacky logic to implement functionalities. For example, each conversion from CommonJS to ESM often requires annotating the original CommonJS code to ensure compatibility and functionality.
This approach is problematic for several reasons. It not only complicates the build process but also makes it difficult to maintain and extend the system. It's a significant issue when dealing with large, complex codebases that use a mix of different module systems or when trying to integrate third-party libraries that may not be ESM-compatible.
In contrast, our aim with Rspack is to be language-agnostic and to treat all module systems as first-class citizens. This approach allows for greater flexibility and reduces the need for complex conversions or hacky workarounds, making the build process more efficient and robust.
Note
Contrary to the intuition of many people, Webpack is language agnostic like Parcel, while Rollup is only a first-class citizen with
Javascript . This may also be the most overlooked point of Webpack 5 , which supports more first-class citizen modules.
Plugin API Design
From the inception of the Rust Bundler project, one core requirement has remained constant: the need to support plugins written in JavaScript. This requirement stems from our understanding of business needs, particularly the necessity for scalability. A bundler that lacks scalability is difficult to implement effectively in a business environment, making the design of the plugin API a critical issue for us.
The two primary factors we considered in the API design were performance and composability. Upon evaluation, we found that Rollup's API fell short of meeting these needs. Rollup's architecture, which is more aligned with treating JavaScript as a first-class citizen, didn't offer the flexibility and performance we required for a diverse and scalable plugin ecosystem.
Our aim is to develop an API that not only allows for high performance but also supports a composable architecture. This would enable plugins to be easily combined and configured, thereby enhancing the bundler's flexibility and adaptability to various business requirements. This focus on performance and composability is part of our broader strategy to make Rspack a robust, efficient, and scalable solution.
Note
Simple API is useful for adoption but maybe hard for scaling.
Module Conversion
Module transformation is a core feature that all bundler plugins must address. While all bundlers offer plugins to handle this, the implementations differ. For example, Rollup uses
transform
, while Webpack usesloader
for this purpose.Upon a comprehensive analysis of module transformation functionality, we identified that this is essentially a three-dimensional requirement:
svg
tojsx
or vice versa.Each of these dimensions has its own set of challenges and requirements, and the plugin architecture needs to be flexible enough to accommodate a wide range of use-cases and performance considerations.
Taking the
svgr
plugin as an example helps illustrate the complexity involved in module conversion. The primary function of thesvgr
plugin is to convert an SVG file into a React component. Breaking this down into the three elements we discussed:/.svg$/
, meaning it will only process files with an.svg
extension.@svgr/core
to transform the SVG content into the corresponding JSX component.This example underscores the intricacy of module conversion logic. Each step—filtering, converting, and changing the module type—has its own set of challenges and considerations. The plugin architecture must be robust enough to handle such complexities efficiently.
In Rollup, the three dimensions of module conversion—filtering, transforming, and changing the module type—are bundled into a single
transform hook
. This design choice leads to several significant issues:transform hook
is executed within the hook itself. This means that to perform filtering, every module incurs the overhead of Rust-to-JS communication. In scenarios involving tens of thousands of modules, especially with Hot Module Replacement (HMR), this overhead becomes substantial. Esbuild addresses this by performing filtering first and only then executing the JS callback. Notably, Esbuild uses Golang's regex for filtering to avoid the Golang-to-JS call overhead, although this can lead to user confusion due to differences between Golang and JS regex.Loss of User Flexibility: With Rollup, the filter logic is hard-coded into the
transform
hook, making it difficult for users to modify the filter logic externally. For instance, if you later decide to process files with different extensions but the same SVG content, you'd have to modify the Rollup plugin itself. In contrast, with Esbuild or Webpack, you can directly modify the filter logic.Loss of Combinability in Module Conversion Logic: Rollup's architecture, which is geared towards JavaScript, requires additional steps to convert the JSX file generated by
@svgr/core
into a JS file. This means that the plugin has to handle the JSX-to-JS conversion logic, even if similar functionality exists in other plugins. For example,vite-plugin-svgr
uses Esbuild to convert JSX to JS, duplicating functionality and introducing another dependency.After extensive research on almost all plugin APIs for module conversion, we found that only Parcel and Webpack's APIs adequately address these issues. Discussion Link
Parcel
Parcel takes a more modular approach to handle the three dimensions of module conversion:
pipeline
to define the filtering logic. You can specify which transformers to use for different types of files directly in the configuration.transform plugin
to define how the conversion should take place. The plugin API allows you to retrieve the asset's source code and source map, run it through a compiler, and set the results back on the asset.asset.type
.This modular approach provides a high degree of flexibility and control, allowing for more complex and varied module conversion scenarios. It also avoids the issues seen in Rollup, such as high-frequency callback communication and loss of user flexibility.
Webpack
Webpack
Webpack's approach to module conversion is also modular but differs in implementation details:
rule.test
to specify which files should be processed by a particular loader.Converter (Transformer) : Webpack employs loaders for the actual conversion logic, such as
@svgr/webpack
for SVG to React component conversion.Module Type Conversion: Webpack uses
inlineMatchResource
for this. It's less intuitive compared to directly modifyingasset.type
, and although there's a proposal for a more intuitive "virtual resource," it hasn't been implemented yet.AST Reuse
The reuse of Abstract Syntax Trees (ASTs) between different module transformations is a crucial design aspect for performance. Parsing overhead is often a significant bottleneck, and reusing ASTs can substantially mitigate this issue. Here's how Esbuild and Rollup approach AST reuse:
Esbuild
Esbuild takes the most straightforward approach: it doesn't support module conversion operations, so the question of AST reuse is moot. Esbuild's parse, transform, and minify steps all share the same AST. This is a key reason why Esbuild's performance is superior to other bundlers. The trade-off is limited scalability due to the lack of support for custom transformations.
Rollup
Rollup allows for the return of ASTs in both its
load
andtransform
hooks. The ASTs must conform to the standard ESTree format. If a standard ESTree AST is returned, Rollup can internally reuse the AST, avoiding the need for duplicate parsing. This offers a more flexible approach compared to Esbuild but requires that plugins adhere to the ESTree standard for maximum efficiency.Note
One advantage of rollup to reuse AST is that rollup only supports JavaScript, which means that only the standard ESTree AST
data structure needs to be considered, but this is not tried for Parcel and Webpack .
Parcel
Parcel's approach to AST reuse is particularly well-conceived. It provides detailed guidelines on how to achieve AST reuse, a design aspect that early versions of Rspack also borrowed from. Parcel's design addresses a significant challenge in AST reuse: how to handle scenarios where string transformations and AST transformations intersect.
For general transformers, there are four possible cases:
Managing the interplay between these different types of transformations is a complex task. Parcel handles this complexity well by allowing transformers to specify both their input and output types and managing the conversions internally. This ensures that ASTs are reused whenever possible, optimizing performance without sacrificing flexibility.
Parcel's approach offers a balanced solution, providing both performance optimization through AST reuse and the flexibility to handle various transformation types. This makes it a robust choice for scenarios requiring both performance and adaptability.
Webpack
Webpack does offer the capability to return an AST in its loaders to facilitate AST reuse. However, this feature has not gained much traction in the community due to several limitations:
babel-loader
andswc-loader
, making it challenging to implement AST reuse effectively in Webpack.These constraints have resulted in Webpack's AST reuse feature being underutilized, despite its potential for performance optimization. It's a feature that exists but hasn't been effectively operationalized due to these community and technical barriers.
Beyond module conversion and reusing AST, we also considered various plugin design aspects like reducing the Rust-to-JS communication frequency, avoiding redundant parsing overhead and virtual modules. After a comprehensive evaluation, we found that Webpack's architecture aligns better with our goals for customization and performance in the development of Rust Bundler. We opted for a Webpack-based approach primarily because it's a lower-risk option; almost no one on the team has experience with Parcel, making Webpack a safer bet.
Webpack Design Exploration
Despite committing to the Webpack architecture, the implementation journey involved several detours, particularly around the concept of first-class citizen support.
Initial Approach and Challenges
Influenced by Esbuild and facing incomplete
loader
support, we initially extended first-class citizen support to various JavaScript extensions likejsnext, ts, tsx, jsx
. While this expedited business-side implementation, it introduced several issues:codegen
doesn't occur immediately post-AST conversion, making it impossible to obtain the converted code for bundle splitting analysis. This leads to inaccuracies, especially when transformations like injectingbabel
andswc
runtimes cause significant code changes.swc
forts
andjs
files led to transformation errors in some modules, such ascore.js
.swc-loader
for transformations, it results in modules undergoing secondary conversions, causing problems.Future Considerations
Given these challenges, Rspack is considering retracting the first-class citizen support for
ts, tsx, jsx
and other such modules. Instead, it plans to allow users to handle compilation viabuiltins:sw``c-loader
. This approach aims to align with Webpack's handling of first-class citizens while ensuring core layer stability.Codegen Architecture
Initial Approach: AST-Based Codegen
In the early stages, we adopted an AST-based
codegen
scheme, contrasting with Webpack's dependency-based string replacement approach. While the AST-based scheme offered better performance by avoiding repeated parsing overhead, it diverged significantly from Webpack's architecture.Challenges
Runtime
andtreeshaking
logic are tightly coupled with the dependency information derived from string replacement. Using an AST-based approach disrupted this harmony.codegen
complicated the subsequent implementation of a persistent cache, adding layers of complexity that were hard to manage.Transition to String Replacement: Version 0.3
Given these challenges, we transitioned from an AST-based
codegen
architecture to a string replacement-based architecture in version 0.3. This move aimed to align more closely with Webpack's architecture, facilitating better integration with features likeRuntime
,treeshaking
, and persistent caching.TreeShaking Architecture
Initial Approach: AST-Based Scheme
Initially, our
TreeShaking
approach was influenced by our AST-basedcodegen
scheme. We borrowed from Esbuild's AST-friendlyTreeShaking
method. However, this approach later revealed several limitations.Challenges
Reexport and Multi-Entry Optimization: The AST-based approach struggled with
TreeShaking
forreexport
andmulti-entry
scenarios. Esbuild'sTreeShaking
optimization lags behind Webpack's, as highlighted in Esbuild GitHub Issue #2049.Transition to Webpack's Implementation
Due to these challenges, we transitioned our
TreeShaking
implementation to align with Webpack's. This move aimed to leverage Webpack's more advanced and in-depthTreeShaking
optimization strategies.Ongoing Exploration
While we've made this transition, several questions and challenges remain, and we continue to explore solutions to further refine our
TreeShaking
architecture.Beyond Webpack: The Road Ahead for Rspack
Out-of-Box Solution
Webpack's most glaring issue is its poor development experience, especially for newcomers. Setting up a project from scratch with Webpack can be daunting, especially when compared to the ease of using Vite. With the decline in community support for create-react-app, there are fewer upper-layer solutions built on Webpack. Recognizing the importance of an out-of-the-box experience, we plan to collaborate with Modern.js to enhance this aspect.
Diagnostics and Debugging
Webpack operates as a black box for most users, lacking essential debugging tools. To address this, we'll continue to refine Web Doctor to improve the debugging experience for both Rspack and Webpack. Additionally, we'll integrate more debugging information directly into Rspack to offer an out-of-the-box debugging experience.
Optimization
While Webpack is often considered a leader in product optimization, there's still room for improvement. Webpack's bundle splitting, code splitting, and tree shaking are all module-based, limiting the optimization techniques that can be applied. We aim to explore function-level granularity for these optimization methods to further enhance runtime performance.
Portable Cache and Remote Build Cache
Within our organization, we have numerous large-scale applications and Monorepo setups. Webpack currently lacks support for portable caching capabilities, making it challenging to implement distributed build cache sharing. This feature is crucial for large applications and Monorepo setups where build speed is a key factor. We plan to focus on developing this capability in the future.
By addressing these areas, we aim to not only serve as a drop-in replacement for Webpack but also to push the boundaries of what's possible in the Webpack ecosystem to deliver a superior user experience.
Beta Was this translation helpful? Give feedback.
All reactions