New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion/Tracking SnapshotCreator support #13877

Closed
bmeck opened this Issue Jun 22, 2017 · 47 comments

Comments

Projects
None yet
9 participants
@bmeck
Member

bmeck commented Jun 22, 2017

Relevant earlier discussion on #9473 .

v8::SnapshotCreator is a means to capture a heap snapshot of JS, CodeGen, and C++ bindings and revive them w/o performing loading/evaluation steps that got to there. This issue is discussing what would be needed for tools like webpack which run many times and have significant startup cost need in order to utilize snapshots.

CC: @hashseed @refack

  • All C++ bindings available as intptr_t
  • C++ addon API/declaration to register external intptr_ts
  • CLI flags for --make-snapshot and --from-snapshot
  • JS API to declare main() functions for snapshots (save to v8::Private::ForApi($main_symbol_name)).
  • JS API to declare vm.Context for snapshot
  • Serializer for
    • ArrayBuffer/TypedArrays
    • HandleWrap
    • ObjectWrap
    • Timer
    • STDIO
    • require.cache paths?
  • File format for snapshots
    • C++ Addon inlining / symlinking

The v8 API might be able to have some changes made (LANDED)

Right now the v8 API would need a --make-snapshot CLI flag since v8::SnapshotCreator controls Isolate creation and node would need to use the created isolate.

Since all JS handles need to be closed when creating the snapshot, a main() function would need to be declared during snapshot creation after all possible preloading has occurred. The snapshot could then be taken when node exits if exiting normally (note, unref'd handles may still exist).

Some utility like WarmUpSnapshotDataBlob from v8 so that the JIT code is warm when loaded off disk also relevant.

@refack

This comment has been minimized.

Member

refack commented Jun 22, 2017

I'm working on first one.

@mscdex

This comment has been minimized.

Contributor

mscdex commented Jun 22, 2017

Does this still really matter now with Ignition (and TurboFan)?

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 22, 2017

@mscdex less so, but allows for rudimentary equivalence of features of pkg and does have impact since you don't need to run any require on startup.

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 22, 2017

Some numbers from local investigation: https://twitter.com/bradleymeck/status/875758792805404673

On a Mac 64bit:

  • 2MB~ is minimal snapshot size (also why setting max old space < 3MB doesn't work)
  • 10e6 globals serialized into a snapshot w/ different hidden classes (global[i++]={[i]:i}) is 88MB, no significant change in speed when calling Isolate::New w/ this vs 2MB.
  • Isolate::New time seems to float around 10ms on local machine (consistent w/ v8 builtin default snapshot)
@refack

This comment has been minimized.

Member

refack commented Jun 22, 2017

If the v8::SnapshotCreator was able to use the existing Isolate, we could use this for pure JS post-mortem dumps, right?

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 22, 2017

@refack yes, but the snapshot blob is platform and build specific unlike core dumps

@refack

This comment has been minimized.

Member

refack commented Jun 22, 2017

But we'll end up with a live system that could be dynamically interrogated. Sounds like an amazing feature, and if V8 stabilize the blob format it seems possible they could implement portability.
/cc @nodejs/post-mortem @nodejs/v8

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 22, 2017

Not sure what you mean by "live", all v8::Handles need to be closed in order to serialize a blob, can't have JS on the stack.

@refack

This comment has been minimized.

Member

refack commented Jun 22, 2017

Not sure what you mean by "live", all v8::Handles need to be closed in order to serialize a blob, can't have JS on the stack.

Ok not live, suspended animation, but AFAICT you could reignite the event loop and use the preloaded code for example to rerun a JS function to see what it's doing.
For example if a production server enters an err state, before you restart it you "suspend" it, take the blob to your dev machine and fiddle with the state in JS land. Mind blown!

@sebmarkbage

This comment has been minimized.

sebmarkbage commented Jun 23, 2017

Related: Proposal for snapshots as a Realm API for wider support in the ecosystem and getting libraries to consider the requirements.

I built a mode into Prepack to compile snapshots of basic Node CLIs. One limitation that I hit was that things like the type of a socket becomes hard coded. You might imagine that taking a snapshot of a CLI that prints to the console and then running it with node mysnapshot > file.txt or something would work but now that stdout is a different type and a lot of the JS streams around stdio has be rewired. Now Prepack can automatically encode multiple branches over abstract values but you'd probably want to manually handle such cases.

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 23, 2017

@sebmarkbage I think the custom serializer deserializer functions in v8 could be enough to handle this. Even with that though, libs could check tty info that gets invalidated when main runs after being deserialized, like you said.

@hashseed

This comment has been minimized.

Member

hashseed commented Jun 23, 2017

Some comments:

  • The isolate provided by the snapshot creator is special in that it prepares for serialization. This basically means that code objects will include meta data to mark external references so that the serializer can find them.
  • Not everything on the heap can be serialized. Most importantly, optimized code and typed arrays.
  • Snapshot for Realm API is different than the start up snapshot in so far that the start up snapshot bundles isolate and context snapshot, while for the Realm API the snapshot will only contain the context, decoupled from the isolate. This could prove to be more complicated since there need to be a way to untangle a context from the isolate it belongs to, into a form that can be transplanted into any arbitrary isolate.
  • Snapshot creator can be used to also include compiled code into the snapshot. For this, check out the implementation of V8::WarmUpSnapshotDataBlob. If it confuses, I can elaborate further.
  • We can't capture a snapshot that includes the stack. This in combination with some of the above points makes snapshot as mechanism for crash dump unsuitable, imo. Also unsuitable for suspending an isolate, as it contains more state than what's on the heap. It really is just to put V8 into a predefined start up state.
  • V8 team is unlikely to stabilize the snapshot format for the same reasons the bytecode format will not be stable.

(edited by @TimothyGu to correct a typo)

@sebmarkbage

This comment has been minimized.

sebmarkbage commented Jun 23, 2017

@bmeck My point was more that the JS libs internal to Node's runtime already does that. Ideally they'd be updated to avoid that problem.

@bnoordhuis

This comment has been minimized.

Member

bnoordhuis commented Jun 23, 2017

Not everything on the heap can be serialized. Most importantly, optimized code and typed arrays.

I wondered about that. Means it's probably not useful for node because typed arrays (Buffers, which are instances of Uint8Array) are used pervasively throughout core.

@hashseed

This comment has been minimized.

Member

hashseed commented Jun 23, 2017

Adding support for serializing typed arrays should be fairly easy, as long as the deserializer can allocate the backing stores outside of V8's heap, I think. I'm not familiar with how Buffer is implemented. Would it work out-of-the-box if Uint8Arrays can be serialized?

@TimothyGu

This comment has been minimized.

Member

TimothyGu commented Jun 23, 2017

@hashseed Yes it would work, but can be a bit unwieldy if no special case is designated. Node.js Buffers can be pooled, and multiple Buffers can share one ArrayBuffer, but with different offsets and lengths. If Buffer serialization were to be implemented, only the visible view should be stored rather than the entire ArrayBuffer. This is what we are doing with the Serializer/Deserializer APIs (see https://github.com/nodejs/node/blob/master/lib/v8.js#L148-L165). Otherwise everything is the same.

/cc @addaleax who implemented the Serializer/Deserializer binding.

@hashseed

This comment has been minimized.

Member

hashseed commented Jun 23, 2017

@TimothyGu that would require V8's serializer to recognize and introduce special case for Buffers, which doesn't sound like a good idea to me. Why is it necessary to serialize views into separate ArrayBuffers? I would suggest to deserialize Buffers into the exact same state as they've been serialized.

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 23, 2017

I am with @hashseed here, I think the entire pool would want to be serialized since a snapshot shouldn't change the data (which is leaking the pool).

@TimothyGu

This comment has been minimized.

Member

TimothyGu commented Jun 23, 2017

@bmeck @hashseed The default pool size (Buffer.poolSize) is 8192 bytes, so the underlying ArrayBuffer is 8192 bytes until it is used in full. In case I need to serialize 4096 consecutively allocated Buffers, each with two bytes, they would actually be stored on the same ArrayBuffer. To serialize Buffers into the same state as they've been initialized, I would have to store 4096 * 8192 = 33554432 bytes instead of just 4096 * 2 = 8192 bytes.

I'm not sure how startup snapshots are different from V8's Serializer, and the calculation above only applies to Serializer. If different ArrayBufferViews in a startup snapshot are allowed to share a single ArrayBuffer, then the issue would of course be solved. If not, I do see the argument for keeping the Buffers in the same state (and TBH I'm fine with that), but it just seems to be a bit wasteful.

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 23, 2017

I think there is a disconnect here. I am not sure why you need to change the pool for all the buffers, that seems like a shallow copy which would also be bad.

@hashseed

This comment has been minimized.

Member

hashseed commented Jun 23, 2017

The start up snapshot is supposed to put the context in a predefined state while bypassing any initialization scripts. It should reach the same state as if the initialization script ran.

Assuming that serializing ArrayBuffer is implemented, there is not going to be a duplication you described above. The serializer can recognize objects that it already visited before and deserialize to the same object graph. After deserialization we would still require just 8192 bytes. I think you are confusing the ValueSerializer with the snapshot serializer. The former is fairly new, while the latter has existed for years, and been continuously improved. We are talking about the latter.

@TimothyGu

This comment has been minimized.

Member

TimothyGu commented Jun 24, 2017

@hashseed Ah that was partially what I was asking – if they operate differently. Seems like they do, so thanks for the explanation :)

@mrkmarron

This comment has been minimized.

mrkmarron commented Jun 28, 2017

We have been working on snapshot/restore in ChakraCore for a while in the context of time-travel debugging. Our current implementation supports full serialization/deserialization of most ES5/6 constructs to a stable and (mostly) engine agnostic format. Snapshot performance is a big concern for us and the current implementation is able to extract/restore the full JS application state on the order of 10's of milliseconds.

We don't yet support linking this up with the native data in Node as, mentioned above, (1) we need to track down and associate all the native reference locations for snapshot/restore and (2) there are challenges in restoring some native state such as file handles or sockets. The first issue is mainly an engineering challenge, an intern of mine experimented with this for migrating live JavaScript/HTML applications (section 5.2). The perf. numbers are on the slow side since it was a proof-of-concept effort. The second issue requires some thought in the API for inflation -- e.g. symbolic values for startup code, user provided hooks, report error on next access and let recovery logic take over, etc. -- and I think having some specific scenarios here to guide design would be very useful.

Using snapshots for application startup performance and migration are things I have been wanting to do more work on, and we have most of the needed JavaScript side parts already implemented, so I am definitely happy to support work on this.

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 29, 2017

@mrkmarron we can discuss need for ChakraCore here as well. Do you think an intptr_t[] is enough for ChakraCore to setup pointers to when doing serialization? In pseudocode~

intptr_t externals = { READ_FILE }; // provided by node

JSFunction readFileFn = { // made by runtime env / C++
   intptr_t internal_fields = { READ_FILE }
};

snapshot_create(...) {
   // walk ...
       for (field, i of obj.internal_fields) {
           serialized.internal_fields[i] = externals.indexOf(field);
       }
   // ... walk
}
snapshot_load(...) {
   // walk ...
       for (serialized_external_i, i of obj.internal_fields) {
           deserialized.internal_fields[i] = externals[serialized_external_i];
       }
   // ... walk
}
@mrkmarron

This comment has been minimized.

mrkmarron commented Jun 30, 2017

Hi @bmeck, if I understand your design correctly I think it should work for recording basic JS->Native references in the snapshot. My paraphrasing of your design is that it uses the index in a fixed by the host array of intptr_t as the stable way to give an identity to a native reference in the snapshot phase so it can be looked up in the inflate phase. This should definitely be do-able with the current ChakraCore snapshot/inflate code.

My only thought here is that this works well when the snapshot is assumed to be at a very stable point (e.g., right after Node loads but before event loop starts) but it can be a bit brittle and difficult to debug if we start to want to use this in more complex scenarios. We had a similar issue with primitive objects from the core runtime (like the undefined object or builtin functions) where we cannot create them and need to give them "well known identities" in the snapshot. We ended up using simple string names, which is slightly lower performance but much easier to debug and more flexible, and I have been satisfied with the result. So, I feel like having a typedef-able setup for the identifier tokens, at least to allow for debugging, might be good.

Also, what are you thinking in of the the Native->JS references? We currently just use the pointer values cast to integer values (#defined as TTD_PTR_ID) in the snapshot and then have a map TTD_PTR_ID -> Js::Object when inflating. We could either look to pass that map back explicitly or provide and API the host can use like Js::Object JsRTMapSnapshotObjectId(TTD_PTR_ID).

I'll plan to put some cycles into the JS->Native implementation next week.

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 30, 2017

@mrkmarron I think the map could be fine as long as it doesn't explicitly point to a memory address once serialized.

For v8, the general idea is to us Private symbols that are not visible to JS and attach them to the global scope:

Context global;
Function require = makeJSRequireFunction();
global->Set(Private::ForAPI("require"), require); // ~= Symbol.for but not visible to JS
// ... take snapshot
// ... revive snapshot
Context global = snapshot.GetContext(0);
Function require = global->Get(Private::ForAPI("require"));

This roughly equates to us manually making the map ourselves so that seems fine. Not having to do all the Get/Sets and being able to get an array would be nice.

@hashseed

This comment has been minimized.

Member

hashseed commented Jun 30, 2017

You could store everything in an array and store that single array via private symbol.

@bmeck

This comment has been minimized.

Member

bmeck commented Jun 30, 2017

I guess that could work but a dictionary object would closely map the string based map @mrkmarron is talking about and is probably more readable.

@mrkmarron

This comment has been minimized.

mrkmarron commented Jul 6, 2017

I was able to do some experimenting with this and add the following JsRT methods in my branch (here):

  1. JsRuntimeSerializeSetExternal(const char* name, void* ptr) which registers a external name/value for snapshot/restore. This seemed easier to use than trying to have the host build a global array and then pass it all at once.
  2. JsRuntimeSerializeSnapshot(const char* location, size_t locationLength) which takes a snapshot and stores it at the given location in the filesystem.
  3. JsRuntimeSerializeInflate(const char* location, size_t locationLength) which inflates the snapshot from the location in the filesystem.

For testing purposes I hooked these into the ch.exe host (similar to d8). In ch I added support to launch a JS file and take a snapshot after loading and a second mode that will load a snapshot and then start execution of code from another JS file.

ch.exe -CreateImg=. c:\Chakra\ScratchJS\SnapInflate\init.js
Runs the code in init.js (below) and produces a snapshot in .\_ttdlog

ch.exe -LoadImg=. c:\Chakra\ScratchJS\SnapInflate\launch.js
Loads the previously created snapshot and produces:

Hello World!
Hello Universe!
Done after timeout!

From the code you can see we recreate all the JS heap and relink the console write and timer hooks so that the launch.js file will run successfully.

For some simple experiments I took 100k objects with a live heap size of ~15MB the time to load/parse the snapshot is 44ms and then it takes 64ms for the inflation. For sizes of 250K/~40MB the times are 106ms/136ms and for 500K/~80MB the times are 206ms/243ms.

This is slower than I would like but we have been optimizing for snapshot time and not too much about inflate costs so there is a lot of room to bring these times down. In addition to the full inflation mode where new contexts and objects are created from scratch we also have a re-use inflate mode. In many cases an application will not modify (or only change slightly) most objects. So, we simply reset and re-use as much of the existing JS heap as possible when re-inflating. This may be interesting from a serverless application perspective where the goal is to reuse an existing JS process, quickly setting it into a known initial state and ensuring state from a previous execution is not leaked into a later one.

//init.js source
let location = ['World', 'Universe'];

const size = 100000;

var dataBlob = [];
for (let i = 0; i < size; ++i) {
    dataBlob.push({ num: i, value: i.toString() });
}

function sayHello(location) {
    WScript.Echo('Hello ' + location + '!');
}
//launch.js source
WScript.SetTimeout(function() {
    WScript.Echo('Done after timeout!');
}, 20);

for(let i = 0; i < location.length; ++i) {
    sayHello(location[i]);
}
@hashseed

This comment has been minimized.

Member

hashseed commented Jul 6, 2017

JsRuntimeSerializeSetExternal sounds interesting. Does it require the external references to be added in the same order for both taking and inflating the snapshot? If not, how do you make sure the required external references have been added for inflating?
I think having one global array has the advantage that for inflating from snapshot we can just index into the provided array without the overhead of having to manage one.
@bmeck is it difficult to create the array of external references exact the same way for both creating and using the snapshot? Does using a name to external reference mapping have big advantages over requring the embedder to hand over the same global array?

@bmeck

This comment has been minimized.

Member

bmeck commented Jul 6, 2017

@hashseed not difficult, just boilerplate. Needing them prior to actually taking the snapshot is a bit of a problem for C++ addons / templates that get instantiated on demand.

@hashseed

This comment has been minimized.

Member

hashseed commented Jul 6, 2017

But even with an API that works with name to external reference mapping, you would need to know before using the snapshot that you'd have to add the external references for certain on-demand instantiated addons / templates, right?

I'm just trying to figure out whether such an API makes sense in V8 as well. It would not benefit Blink (we are actually moving to the opposite direction to reduce overhead of importing external references upon using a snapshot). If it does not actually take away work in Node it probably makes no sense to add this API and the involed complexity.

@bmeck

This comment has been minimized.

Member

bmeck commented Jul 6, 2017

correct, we would probably use a container format that keeps this data around in addition to the startup blob.

container {
  container_protocol_version: ...
  addons: [
     // shared library data (can be loaded via dlopen mechanisms)
     // bikeshed: well known `export C snapshot_externals` symbol to grab list of externals
  ],
  snapshot_blob: void* // serialized snapshot
}

load_snapshot(container) // concatenates node internal externals and addon externals to get full array
@hashseed

This comment has been minimized.

Member

hashseed commented Jul 6, 2017

That's similar to what I had in mind. The flexibility provided by mapping via name would be unnecessary given the complexity that would introduce, I guess.

@mrkmarron

This comment has been minimized.

mrkmarron commented Jul 6, 2017

I don't have a strong opinion on the array vs. single entry design, the single entry just seemed simplest while I was prototyping but changing to arrays is not hard.

However, I definitely think there is a lot of value using stable names for the mapping. Unless there is a large performance implication or other serious concern I see having a stable "obvious" key for the mapping as a much better choice than relying on an implicit and subtle indexing invariant for the API (particularly as it will be public facing).

@mrkmarron

This comment has been minimized.

mrkmarron commented Jul 7, 2017

I am planning to spend some cycles next week on cleaning up the ChakraCore code I have and looking at how to integrate with Node as the host. @bmeck do you have any sort of implementation in Node yet? If so I can use this as a starting point.

@hashseed

This comment has been minimized.

Member

hashseed commented Jul 19, 2017

@bmeck
I'm looking into snapshot support for TypedArrays at this point. In V8 we differ between those with on-heap and off-heap backing store.

On-heap ones are garbage collected like any other object, and are used for small TypedArrays. I assume that Node.js doesn't use them because they can be relocated.

Off-heap ones are allocated on the C++ heap, and can be either internal or external. See v8::ArrayBufferCreationMode. Internal ones are tracked by V8's GC and freed if no longer referenced. For external ones V8 assumes them to be managed by the embedder.

For the serializer, hooking up internal off-heap TypedArrays are fine. But hooking up external ones to what ever the embedder uses to track them would require a new API. Does Node.js use any external ones?

@bmeck

This comment has been minimized.

Member

bmeck commented Jul 19, 2017

@hashseed

Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), data, length);
is the only thing I think that might be a problem. However, I don't think Node itself ever changes the underlying buffer once created. Addons might, but if throwing an error on encountering them is sane that might be a good solution.

@mrkmarron

This comment has been minimized.

mrkmarron commented Jul 25, 2017

It was able to spend some time on this last week updating the snapshot/inflate code in ChakraCore to deal with some of the additional external data references that appear when running in a Node host. At this point the largest problem I am running into is getting Node to reset the list of native pointers before inflate. I worked around this with a pretty hard-coded hack but it would be great if there was a cleaner implementation of the Node host side code.

I'll be at Node Summit this week if anyone else is going to be there and wants to discuss in more detail.

@hashseed

This comment has been minimized.

Member

hashseed commented Aug 10, 2017

@bmeck, you mentioned that there are instances of ArrayBuffer with external off-heap backing store. Do you have any pointers for me on that? This may be solved simply by adding more logic to the array buffer allocator on Node side.

@bmeck

This comment has been minimized.

Member

bmeck commented Aug 10, 2017

@hashseed I think most of the cases are fine, but Buffer.allocUnsafe and Buffer.allocUnsafeSlow are probably not able to be made internalized to my knowledge.

Since we never use kExternalized; you can get a list like below easily from grepping the source.

  • Allocated but look to act like Internalized
    ArrayBuffer::New(isolate, fields_ptr, fields_count * sizeof(*fields_ptr));

    node/src/async-wrap.cc

    Lines 543 to 546 in 4796fbc

    Local<ArrayBuffer> uid_fields_ab = ArrayBuffer::New(
    isolate,
    uid_fields_ptr,
    uid_fields_count * sizeof(*uid_fields_ptr));
  • Are these just for Domains ?

    node/src/node.cc

    Line 1209 in 4796fbc

    ArrayBuffer::New(env->isolate(), fields, sizeof(*fields) * fields_count);

    node/src/node.cc

    Line 1254 in 4796fbc

    ArrayBuffer::New(env->isolate(), fields, sizeof(*fields) * fields_count);
  • No clue how to fix this NAPI call which always makes externals

    node/src/node_api.cc

    Lines 2885 to 2886 in 4796fbc

    v8::Local<v8::ArrayBuffer> buffer =
    v8::ArrayBuffer::New(isolate, external_data, byte_length);
  • No clue how to fix Buffer.unsafe*
    Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(), data, length);
  • fs.stat avoids copy into heap, but fundamentally acts like internalized

    node/src/node_file.cc

    Lines 1424 to 1426 in 4796fbc

    Local<ArrayBuffer> ab = ArrayBuffer::New(env->isolate(),
    fields,
    sizeof(double) * 2 * 14);
  • v8 module hooks up to v8 C++ Heap statistics. Could repopulate after revival?

    node/src/node_v8.cc

    Lines 147 to 149 in 4796fbc

    ArrayBuffer::New(env->isolate(),
    env->heap_statistics_buffer(),
    heap_statistics_buffer_byte_length));

    node/src/node_v8.cc

    Lines 195 to 197 in 4796fbc

    ArrayBuffer::New(env->isolate(),
    env->heap_space_statistics_buffer(),
    heap_space_statistics_buffer_byte_length));
@refack

This comment has been minimized.

Member

refack commented Aug 10, 2017

It seems

node/src/node.cc

Lines 1249 to 1257 in 4796fbc

// Values use to cross communicate with processNextTick.
uint32_t* const fields = env->tick_info()->fields();
uint32_t const fields_count = env->tick_info()->fields_count();
Local<ArrayBuffer> array_buffer =
ArrayBuffer::New(env->isolate(), fields, sizeof(*fields) * fields_count);
args.GetReturnValue().Set(Uint32Array::New(array_buffer, 0, fields_count));
}
is used to comunicate the state of the Tick queue between JS and C++ lands.
But AFAICT it could be de-abstracted since fields_count === 2 always.

@addaleax

This comment has been minimized.

Member

addaleax commented Aug 10, 2017

I’m kind of jumping in here without having read the full context yet, but if it’s about the AB backing store not being on the heap: It should be pretty easy to remedy that by creating the ABs in JS and then passing them to C++ instead of the other way around, right?

@bmeck

This comment has been minimized.

Member

bmeck commented Aug 10, 2017

@addaleax the issue is the kExternalized ArrayBuffers are not managed by the v8 heap; that is correct. We can still use C++ to make kInternalized ArrayBuffers and do in node_buffer.cc:

node/src/node_buffer.cc

Lines 304 to 307 in 4796fbc

ArrayBuffer::New(env->isolate(),
data,
length,
ArrayBufferCreationMode::kInternalized);

The snapshot serialization does not currently have a way to revive kExternalized ArrayBuffers.

@hashseed

This comment has been minimized.

Member

hashseed commented Aug 29, 2017

To reset the discussion on ArrayBuffers...

We have three kinds of ArrayBuffer backing stores:

  1. on V8's heap and managed by V8's GC
  2. off-heap, but registered to V8's GC, and therefore managed by V8's GC
  3. off-heap and external, i.e. not managed by V8's GC.

We can serialize and deserialize the first two kinds just fine. For off-heap backing stores, the deserializer simply calls the array buffer allocator.

The third kind is the one that the serializer currently cannot deal with, since we assume that the embedder is managing its lifetime. Maybe we should just assume that the embedder provided an array buffer allocator that correctly registers the backing store wrt lifetime management?

@bmeck

This comment has been minimized.

Member

bmeck commented Aug 29, 2017

Maybe we should just assume that the embedder provided an array buffer allocator that correctly registers the backing store wrt lifetime management?

If that is added to the documentation, that seems fine.

@bmeck

This comment has been minimized.

Member

bmeck commented Jan 17, 2018

closing in favor of #17058

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment