Known issue of porting Webpack Sync APIs

[h-a-n-a]

TL;DR

This only affects Sync APIs which calls into Node thread-safe functions.

Background

Before we dive into the problem, let me show you the plugin architecture and NAPI a little bit for better understanding of the problem.

Rspack is divided into two parts node_binding and rspack_core. In the crate of rspack_core, we have a plugin driver to iterate through every registered hooks. node_binding has an adapter for calling Node registered plugins, but still, they are called in rspack_core. No matter if a plugin is a Rust based one or Node based one, rspack_core will just do the job.

The Node plugin adapter basically converts any JavaScript plugin hooks into Rust. There is where NAPI comes to the play. We know that JS is running single-threadedly on Node. So NAPI comes up with a function type ThreadSafeFunction to solve interoperability issue between multi-threaded language and single-threaded language.

Here is how it is going to play: for each thread-safe functions defined on the Node side, you can invoke it on any thread with supported data types, however, the node side cannot assure you if this function call can be called directly as there might be some tasks running on the main thread that are not finished yet. So a queue is introduced for each thread-safe function and if the stack is empty, then Node will pop the queue and run the job enqueued by the previous thread-safe function call from the Native side. This call is always asynchronous for the native side.

Problem

When I was reviewing this PR #3403, there's a problem when we are trying to port a Webpack Sync API with JS hooks call. So basically, if a JS method which under the hood is a native method call and the native method call calls any JS hook will cause the deadlock. For this PR, there's an API called compilation.getAssetPath and this API will call the compilation.hooks.assetPath hook under the hood. Furthermore, compilation.hooks.assetPath, intentionally, can be registered on both native and JS side. Here's how it looks like:

This API is supposed to be a sync API. However, like what I have mentioned above, thread-safe functions can be called on any thread and you need wait until the main thread stack is empty and call back to Rust. Even if we try to use tokio's blocking method, this still does not work since the sync API is blocked by the native call and main thread stack is not empty, so the real thread-safe function call will never be called, thus Rust will be waiting for the result forever.

Potential proposals

Technically speaking, we have three ways to solve this common issue.

1. Changing Sync API calls to Async one

This will free the main thread stack so that the thread-safe queue can be proceed. However, this changes the original API of webpack to async

2. Calling JS Hooks on the JS side

Calling JS hooks on the JS side will not cause interoperation between Node and Rust. This is also the original webpack's implementation. However, there is a hidden contract for this solution: You cannot register that hook on the native side. If you break that contract, this will result in the result being completely not right.

3. Using normal function instead

The problem is basically due to the architecture of thread-safe function. It is similar to the mpsc(multiple producer single consumer) architecture. So you can change the current thread-safe function to the normal JS function instead. But this will also change the plugin driver's architecture.

[hardfist]

for solution 1 we can add another async_hook and mark the sync hook unusable

[h-a-n-a]

This will affect the codebase a lot. So I suggest us to settle down the solution in the first place.