[node-api] Creating a reference-counted env-less JavaScript object that can be shared between threads

[alshdavid]

What is the problem this feature will solve?

Introduction & Use Case

Hi all,

At Atlassian, we have a build tool written in Nodejs that makes heavy use of worker threads. It follows the best practice conventions for communicating/sharing data between Worker threads however we are running into significant limitations.

For suitable data, we use manually managed pages of SharedArrayBuffers and serialize JavaScript data into custom data structures, taking a "copy-on-read" & "copy-on-write" approach when the data is modified. For data that cannot be stored in SABs, we either postMessage the whole structure to the worker threads (copying it), then manually synchronize it by sending diffs back to the main thread via postMessage which propages those changes to all worker threads - or we store the data on the main thread and use getters/setters that wrap postMessage.

However, despite our best efforts, the application consumes around 60 - 100gb of ram (~ 10gb per thread, depending on how many threads we use) and we see diminishing performance gains after about 6 worker threads due to the overhead of de/serializing + message overhead.

In addition, the worker thread glue code is immensely complex and difficult to maintain.

We have also tried to use an external in-memory db to share data however, the serialisation/deserialisation overhead makes it perform poorly.

While JavaScript itself is a fantastic language and performs well, we recognize that the single-threaded nature of the runtime likely makes it unsuitable for our use case (inherited codebase). We are rewriting the tool in Rust; however, due to the complexity of the project, that will likely take 2+ years and in the meantime, we are looking for solutions to alleviate the cost of running the existing project.

Proposal

Note: I realize there are inherent limitations with V8 isolates that may prevent this from being possible but I am not an expert in v8 APIs so I lean on the team's expertise to help understand if this is feasible.

Is it possible to add n-api functions that allows the creation of a zero-copy napi_value that can be used between envs?

Perhaps an approach similar to the reference-counted threadsafe_function approach is more feasible:

NAPI_EXTERN napi_status 
napi_create_transferrable_object(
  napi_env env,
  napi_transferrable_object* result
)

NAPI_EXTERN napi_status 
napi_aquire_transferrable_object(
  napi_env env,                       // Can be obtained from any env
  napi_transferrable_object* value,
  napi_transferrable_handle* result,
)

NAPI_EXTERN napi_status 
napi_set_key_transferrable_object(
  napi_env env,
  napi_transferrable_handle* value,
  napi_value key,       // structuredClone(key)
  napi_value value,    // structuredClone(value)
)

napi_get_value_transferrable_object(/* ... */)
napi_release_transferrable_object(/* ... */)
napi_ref_transferrable_object(/* ... */)
napi_unref_transferrable_object(/* ... */)

• The host env defines a napi_transferrable_object, producing a reference-counted handle to the object
• Any env could acquire anapi_transferrable_object
• Only one env can acquire the object at a given time
• If the object is acquired by an env, acquiring a object in another thread would block until the object is released
• keys and values are still copied when set/get to/from the transferrable
• Extra: the object's memory usage does not count towards heap/max-old-space-size
  • Values obtained from the object are cloned into the current env and dealt with by GC as normal

How this solves my use case

The idea is that this API would facilitate synchronization of mutations to a JavaScript object between Worker threads, allowing me to share a simple JavaScript object to act as state between isolates without copying it or syncronizing the value with a postMessage based RPC implementation.

Deep properties / Complex types

Given how dynamic JavaScript values can be, perhaps a value can only be transferrable if the object is capable of being passed into structuredClone.

// Assume a native addon that exposes a high-level API wrapping the native functions
const handle: number = myNativeAddon.createTransferrable() 

worker.postMessage(handle)

const guard: HandleGuard<Map<any, any>> = await myNativeAddon.aquireTransferrable(handle)
guard.set('foo', 'bar')     // The key and value must go through `structuredClone()`
console.log(guard.get('foo')). // "bar"
myNativeAddon.releaseTransferrable(guard)

Async

It would be good if acquiring a transferrable value was an async operation, as it would avoid blocking the thread waiting for the value to be released.

GC considerations

I understand that a napi_value is tied to an env and is managed by that context's GC (by V8).

I'm wondering if, when a value is marked as "transferrable", much like the napi_threadsafe_function API, is it possible that the value is removed from v8 GC and instead managed with a reference-counted smart pointer?

Other thoughts

Given the performance/memory of the tool in JavaScript is actually fine when single threaded but slow due to the inability to use multiple threads, having the ability to share/syncronize data between threads may be enough to avoid needing to rewrite the project.

What is the feature you are proposing to solve the problem?

Addition of napi functions that facilitate syncronization of JavaScript values between threads

What alternatives have you considered?

• SharedArrayBuffer
  • Still require copying data to write it
  • Not all data can be stored
  • Complex synchronization due to the need for Atomic
• Synchronization via postMessage
  • Requires copying data
  • Complex diffing logic
• Centralizing data in an external in-memory database
  • Performs poorly due to serialization/deserialization overhead
  • Still requires copy-on-read and copy-on-write
• Rewriting in another language
  • Not practical in the short/medium term

[bakkot]

You may be interested in the shared structs proposal, though even if it does eventually solve your problem it will not do so in the near term.

[bnoordhuis]

V8 has preliminary support for shared structs, gated on the --harmony_struct flag, but it's all very tentative, undocumented, very lightly tested, and not at all like how it's described in the TC39 proposal. Without knowing more about Atlassian's specific use case, it's hard to say whether it's a good fit.

W.r.t. a napi_value that's somehow shared between envs: I won't bore you with details but alas, that cannot possibly work.

I don't have good suggestions off the bat, not without knowing more about the application (what it does, how it does it, etc.) I don't want to turn this into an upsell but it feels like you'd be better off with an expert consultant than a bug tracker issue.

[alshdavid]

V8 has preliminary support for shared structs, gated on the --harmony_struct flag

That's super interesting!

I don't have good suggestions off the bat, not without knowing more about the application (what it does, how it does it, etc.)

I appreciate your insight nonetheless. It's a bundler forked from Parcel so we inhert their WorkerFarm approach (but we dropped support for process based workers and only use Nodejs workers).

Our use case is pretty simple. We have a few graph structures that are essentially some objects/arrays that look approximately like;

const graphAdjacencyList  = []
const graphWeights = []

function addNode(graph, edgeFrom, edgeTo, weight) {}
function walk(graph, callback) {}

The way we work with the graphs depends on the phase of the build the tool is in. For example, when the graph is read-only we have to copy the whole graph (the graph is 5gb) to the threads because traversing over postMessage is too slow - the compromise is memory usage.

One solution we are looking at is storing the graph data using a memory mapped file (lmdb) and storing the data as protobuf (protobuf.js). That would ensure the data is accessible across all threads without copying, but read/write operations would require copying and ser/de.

I don't want to turn this into an upsell but it feels like you'd be better off with an expert consultant

Haha, I'm sold and have made the suggestion - sadly I can't authorize that

[bnoordhuis]

There are a number of examples in https://github.com/nodejs/node/tree/v24.2.0/deps/v8/test/mjsunit/shared-memory - they're the tests that have a // Flags: --harmony-struct comment. Caveat emptor: very experimental, use at own risk.

The flag adds two new globals, SharedStructType and SharedArray, plus Atomics.Mutex and Atomics.Condition; Atomics.compareExchange etc. now also work on shared struct fields and shared array elements.

const MyStruct = new SharedStructType(["foo", "bar"], "MyStruct") // second arg is the cross-isolate canonical name
const s = new MyStruct
s.foo = 42
Object.assign(s, {foo: "ok", bar: 1337})
if (1337 === Atomics.compareExchange(s, "bar", 1337, 42)) { /* ok */ }

Shared structs/arrays can be sent over with postMessage. Fields/elements can only contain primitives or other shared structs/arrays.

If you're storing lots of strings, --shared_string_table may help (or not - you'll have to measure.)

I think (but am not 100% sure) things like await Atomics.Condition.waitAsync(cv, mtx) should Just Work(TM) in node. Quick testing locally suggests it does.

[alshdavid]

If you're storing lots of strings, --shared_string_table may help (or not - you'll have to measure.)

Yes, strings account for the vast majority of our memory usage. I was trying to share them with TextEncoder, TextDecoder and SharedArrayBuffer but that wasn't a very good solution as they are still copied on read and (I think) there is a conversion to utf16 -> utf8 -> utf16.

I also investigated using the streams API to remotely read/write a centrally stored string but I think that suffers the same limitations.