New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: speeding up Node.js startup using V8 snapshot #17058

Open
hashseed opened this Issue Nov 15, 2017 · 60 comments

Comments

Projects
None yet
@hashseed
Copy link
Member

hashseed commented Nov 15, 2017

I recently went through Node.js bootstrapping code, and think that we could make it a lot faster using V8 snapshot. I wrote a design doc that captures the main points.

This is somewhat separate from the discussion on using V8 snapshot to capture arbitrary initialized state, discussed here. The main difference is that the set of native modules is known upfront, and there is no ambiguity about the native bindings that need to be known to V8's serializer/deserializer.

I'm doing this as sort of a side project, so it may take some time for me to make progress. Any help is welcome.

@mhdawson

This comment has been minimized.

Copy link
Member

mhdawson commented Nov 17, 2017

Sounds great. Is there a repo/branch people can take a look at the changes you have so far for this work ?

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 17, 2017

I have a development branch here. All tests still pass, because they are not affected. Running node snapshot attempts to create a snapshot blob. It currently still fails, but I have collected almost all native references. There is still a long way to go though.

@juancampa

This comment has been minimized.

Copy link

juancampa commented Nov 18, 2017

Hi @hashseed, this is amazing and very needed, thanks for working on this. I asked on #17103 but it seems pertinent to this thread (since it's an RFC) so I'll ask here as well.

Would these changes allow for taking snapshots in arbitrary points in time? as opposed to right after startup? If not, do you mind explaining if such a thing would even be feasible. Seems like the guys from Chakra have been working on something like it for Time Travel Debugging (as mentioned by @mrkmarron: #13877 (comment))

To elaborate a little, I'm thinking something like CRIU, where you can send a signal to the process (e.g. kill -USR2 <pid> or some other mechanism) and have the process serialize it's state (to a predefined location maybe) and optionally exit. Then later in time you could restore that serialized state by providing it to a new node process (e.g. node --restore <state-file>)

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 18, 2017

It would, once finished, help achieve that goal. Current V8 API is not laid out to serialize arbitrary isolates at arbitrary times, but something in that direction would be thinkable.

However, there are quite a few C++ objects allocated by native modules. Each module would have to provide a way to serialize/deserialize its objects, and we would need a way to dispatch to each module for serialization/deserialization. This hurdle applies to Node with Chakra too.

Another issue is that there is no good way to serialize/deserialize the stack, including C++ frames and local variables that hold onto objects on the JS heap.

@mrkmarron

This comment has been minimized.

Copy link

mrkmarron commented Nov 19, 2017

Hi @hashseed, I was able to take a quick look at the document and the commits. As you say it looks like there is a lot of work left but this looks like a great start. Please let me know if there is anything that might be useful to help and I will try to make some time. Otherwise, I'll definitely keep an eye on the progress here.

Thanks for this effort!

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 19, 2017

Hi Mark,

this is more of a solve-issues-as-we-go approach. Once I have a roughly working version done, there will be some more careful auditing necessary to see which part of initialization needs to be done after snapshotting because they are runtime-dependent, for example gated by command line args. Maybe that's something you can dig into, if you have the time?

@mrkmarron

This comment has been minimized.

Copy link

mrkmarron commented Nov 20, 2017

Sure, I have looked into this type of thing in the past. The two most common approaches seem to be either (1) making this type of startup a pure programmer responsibility or (2) using symbolic execution (ala prepack). Neither solution is 100% satisfactory but I'll try to spend some time investigating to get a better understanding of things.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 20, 2017

I updated the branch. So far running node snapshot creates a snapshot_blob.bin file into the working directory. With the file in place, running node uses the snapshot, deserializes an isolate and a context, and then immediately quits (because I haven't spent time on making the deserialized context work yet :D).

For some reason I wasn't able to reproduce the 400ms start up for vanilla Node.js anymore, but rather 200ms. I wonder whether I made some mistake with the build. But when I instrumented my branch with uv_hrtime() I was at least able to reproduce 4x speed up.

I'll carry on to make the deserialized context actually work.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 20, 2017

@addaleax @bnoordhuis @TimothyGu is there any reason we store the environment in a v8::External object to pass to function templates as data, instead of getting the current context from the isolate and the environment from there?

v8::Externals are tricky to serialize. They should belong to the context. Function and object templates belong to the isolate, but can own v8::External objects as data. With the current API there is no good way serialize templates as part of the context.

v8::External objects are also kinda hacky. They look like JavaScript objects, but are special in that they are context-independent and don't have a constructor.

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Nov 20, 2017

is there any reason we store the environment in a v8::External object to pass to function templates as data, instead of getting the current context from the isolate and the environment from there?

@hashseed See commit 7e88a93, it's explained in the commit log.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 20, 2017

I see. So these functions are fundamentally bound to the context. Would it help if FunctionCallbackInfo can return the function's creation context?

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Nov 20, 2017

Yep, that should work if it's also done for PropertyCallbackInfo.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 20, 2017

Quite a few yaks to shave... I think I'll just pile things on this branch first and later disentangle into single PRs.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 20, 2017

@bnoordhuis for PropertyCallbackInfo the Holder should suffice? I don't think there is a way to move an interceptor from one object to another.

As for FunctionCallbackInfo, the corresponding class in v8::internal, FunctionCallbackArguments, actually already has a callee argument in the constructor. It's just never used or stored anywhere. That can be fixed.

Edit: looks like removing callee was intentional. Guess I'll solve it differently: FunctionTemplates and ObjectTemplates that reference some data that is context-dependent (including v8::External) must be serialized as part of the context.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 21, 2017

@bnoordhuis I just verified with @verwaest that inside a function template callback, the current context is indeed already the context of the function being called.

That means we don't need to wrap the environment in a v8::External anymore.

@verwaest-zz

This comment has been minimized.

Copy link
Contributor

verwaest-zz commented Nov 21, 2017

This isn't true yet for PropertyCallbackInfo and Interceptors though. We could do the same there as we do for "lazy accessor pairs" (function template based accessors).

@liqyan

This comment has been minimized.

Copy link
Contributor

liqyan commented Dec 21, 2017

Hi hashseed, i have an experimental implemention here. It roughly works.

Just like points you wrote in design doc, i do:

  • splitting up node_bootstrap.js's startup into two functions: bootstrap and new startup, for node_mksnapshot, bootstrap is the entry, for node booting from snapshot, the latter is the entry.
  • registering external references to SnapshotCreator, All is c++ function, and node::Environment*

and some more:

  • support registered Persistent/Global handles serialization
  • distinguish Persistent/Global handles from Context aware or not, registering the former to Context, registering the latter to Isolate
  • while booting from snapshot, first allocate memory for node::Environment without initialization, register it to Isolate, then using placement new to create node::Environment after Context has been created
  • change the behavior of external string serialization. External string will still be external string if it's ExternalStringResource is registered. v8 SnapshotCreator converts external string to internal string. node's builtin modules are all exteranl strings, after booting from snapshot, they are internal strings whose chars allocated in v8 heap. These internal strings can't be shared if node fork/exec or just fork child processes

some remain issues:

  • external references registering and c++ function binding looks like redundantly
  • setupGlobalConsole consumes significant time ( 14ms, vs total running time 39ms with snapshot on my workstation), but node_mksnapshot can't run with it currently(crashed), node booting from snapshot must restore the status below:
    • one ChannelWrap and one SignalWrap object will be created
    • two TTYWrap object will be created, and some uv tty operations will be executed

Running the command time ./node -e "", with snapshot ther result is 39ms, without is 88ms, on my workstation.

howto build

 
./configure --with-node-snapshot
make
cd out/Release
./node_mksnapshot --startup-blob=snapshot_blob.bin

Hope this helps.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Dec 21, 2017

Wow. That's a big surprise. I'll definitely check out your implementation once I have some time.

I must admit that I haven't actually pursued my prototype much further in the last couple of weeks since I haven't had time and had to wrap up a few other projects at the end of the year.

When you say "external references registering and c++ function binding looks like redundantly" do you mean it's not necessary? Is that because you do not set up any function templates when creating the snapshot? Function templates has been one of the biggest blockers for me since they point to the Environment via data pointer, which makes them no longer context-independent.

Cutting the startup time to less than half is a bit less than what I expected, but definitely a significant win! We should try to get this merged, maybe incrementally.

@liqyan

This comment has been minimized.

Copy link
Contributor

liqyan commented Dec 22, 2017

@hashseed

When you say "external references registering and c++ function binding looks like redundantly" do you mean it's not necessary?

What i want to say is c++ functions binding in node builtin addon initialization procedure, and registering these c++ functions to SnapshotCreator while creating snapshot or to Isolate while booting from snapshot, do similar things, like env->SetMethod(target, name, api_callback), or external_references[idx]=api_callback. If there's a way to do these things in one mechanism, it'd be great.

Cutting the startup time to less than half is a bit less than what I expected, but definitely a significant win! We should try to get this merged, maybe incrementally.

As i said, setupGlobalConsole consumes significant time. Once node_mksnapshot can be executed with it, the startup time will be 1/4.

@hashseed hashseed referenced this issue Dec 22, 2017

Closed

Call with V8 #454

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Dec 22, 2017

I opened an issue on V8 to track implementing serializer/deserializer support for external strings: https://bugs.chromium.org/p/v8/issues/detail?id=7240

I have some other ideas than the one you implemented. I'll work on that and land that upstream.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Dec 22, 2017

I took a look at the changes that affect V8:

  1. deps: v8: serialize/deserialize external string table correctly
  2. deps: v8: support ser/des global/eternal handles
  3. deps: v8: support visit external string

To support external strings, I came up with this upstream change. Does that solve the issue that commits (1) and (3) solve? I'm not entirely sure what commit (1) fixes.

As for commit (2), I think it goes into the right direction, but is a bit over-engineered. You are putting global and eternal handles into fixed arrays before serialization, so that you can re-create global and external handles after deserialization. You distinguish between ones that are context-independent and ones that are context-dependent. A few thoughts on that:

  • With this approach, do we really need to distinguish between global and external handles, if they are both stored in fixed arrays?
  • Should context-dependent eternal handles even exist in the first place?
  • Assuming that we have no context-dependent eternal handles, we could deserialize eternal handles into a std::vector of pointers that we explicitly visit during GC, like the partial-snapshot-cache. Getting external handles from that std::vector would be simple, as that's essentially the same data structure the eternal handle implementation already uses.

I would love if you could contribute (2) to upstream V8. Due to legal reasons I can't just copy your code and submit it upstream.

What i want to say is c++ functions binding in node builtin addon initialization procedure, and registering these c++ functions to SnapshotCreator while creating snapshot or to Isolate while booting from snapshot, do similar things, like env->SetMethod(target, name, api_callback), or external_references[idx]=api_callback. If there's a way to do these things in one mechanism, it'd be great.

Do you mean to say that you would like to have a way to add external references late? How late? At the point of SnapshotCreator::CreateBlob? That should be doable and simply an API change.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Jan 29, 2018

Great to see you are making progress!

I do agree that Node.js' bootstrapping is not ideal and by far not as isolated as it should be. I'm looking forward to the proposal and will take a look at your current experimental implementation in the meantime.

@liqyan

This comment has been minimized.

Copy link
Contributor

liqyan commented Jan 31, 2018

Problem

Currently lib/internal/bootstrap_node.js looks like this:

(function(process) {
 function startup() {
  // node booting
  // test arguments and run user code
 }

 ...

 startup();
});

There's no clear boundary between node booting and running user code. And the state that node bootstrap finished varies due to runtime arguments (command line arugments, system environment variables, etc.).

To use snapshot to speed up node startup, we have to make sure there's a consistent state between creating snapshot and using snapshot. This means we have to eliminate all the effects of runtime arguments.

Proposal

To achive the consistent state, this proposal suggests node should have boot procedure like this:

(function(process) {
   function setupNode() {
     // run things that don't depend on runtime arguments

    ...

    preloadModules();
    return startup;
  }

  var postpones = [];
  function postSetupNode() {
    // run things that depend on runtime arguments
    // invoke each function in postpones and clear it
  }

  function startup() {
    postSetupNode();

    ...

    // test arguments and run user code
  }

  ...

  return setupNode();
});

setupNode and postSetupNode establish the javascript runtime together.
The main criteria is setupNode must not access runtime arguments directly or indirectly. This means:

  • Native module reqiured by setupNode must not access runtime arguments during "require" directly or indirectly. If we support preloadModules while creating snapshot, it means all native modules.
  • setupNode must not invoke functions exported by native modules.
  • setupNode must not invoke functions defined in lib/internal/bootstrap_node.js.

Native Modules

To satisfy criteria 1 we have to refact native modules like this:

'usr strict'

// do init things
// do export

maybe_postpone_require(() => {
  // things access runtime arguments directly or indirectly in previous top level function
});

maybe_postpone_require is accessable via NativeModule.wrapper. It postpones functions passed to it until postSetupNode when executing setupNode, and invokes functions passed to it immediately then.

  let postponed_fns = [];
  let maybe_postpone_require = function(postponed_fn) {
    postponed_fns.push(postponed_fn);
  }

  function postSetupNode() {
    // reset maybe_postpone_require
    maybe_postpone_require = function(postponed_fn) {
      postponed_fn();
    }

    postponed_fns.forEach(function(fn) {
      fn();
    });

    postponed_fns = undefined;
  }

  NativeModule.wrapper = [
    '(function (exports, require, module, internalBinding, maybe_postpone_require, process) {',
    '\n});'
  ];

Only native modules are involved since user modules use another version require.

Split up setup

To satisfy criteria 2 and 3, we have to split those setup functions into 2 parts: one part that can be invoked by setupNode, the rest will be invoked by postSetupNode.

Binding addons

Some initialization funcition of builtin addons do more thing besides setting up the addon object, just look at crypto:

void InitCrypto(Local<Object> target,
                Local<Value> unused,
                Local<Context> context,
                void* priv) {
  static uv_once_t init_once = UV_ONCE_INIT;
  uv_once(&init_once, InitCryptoOnce);

  ...

}

We should register these things while creating snapshot, and run them while boot from snapshot. I'll explain this in another proposal about the c++ part.

Check access violation

All accesses to runtime arguments are via process object(?). So we can set up interceptor for process object to check access violation during booting.

@liqyan

This comment has been minimized.

Copy link
Contributor

liqyan commented Jan 31, 2018

Struggling with english language, I'm afraid I have not expressed myself clearly. Please do tell me if you don't understand.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Jan 31, 2018

I definitely agree with this proposal. Currently, Node.js startup includes parts that depend on command line flags or env vars, and some internal modules allocate C++ objects.

These have to be factored out so that we have a phase of initialization that initializes Node.js in a deterministic way, followed by a phase that is affected by command line flags and env vars. This would mirror what we have in V8.

I'm not entirely sure wrt C++ objects. If they are referenced by V8 objects via embedder fields, we can serialize them, as you have it in your experimental branch.

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Jan 31, 2018

I just brought this up at the V8/TSC call. @jasnell mentioned that he has some plans/ideas to disentangle bootstrapping. James, do you have any specifics that you want to share? I guess this is something we could already get started with even before the necessary V8 API becomes available on master. I would definitely be willing to help.

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Jan 31, 2018

Yeah, the key issue is that node.js startup is a Tangled mess of spaghetti code right now. From the way cli args and env vars are processed, to the order of start up events, etc. I've been looking at introducing a proper command line args parser and start up options isolation that will be tied to the Environment rather than using global state, then will be separating out the initialization of the process object. Once that is done, we should be able to refactor the bootstrap script into distinct phases. The key challenge with the snapshot is that any snapshot is going to be tightly coupled with cli/env args at startup if we do it right now so separating these out first will be critical.

@liqyan

This comment has been minimized.

Copy link
Contributor

liqyan commented Feb 1, 2018

@hashseed

I'm not entirely sure wrt C++ objects. If they are referenced by V8 objects via embedder fields, we can serialize them, as you have it in your experimental branch.

I don't think we have to concern wrt C++ objects. There're 4 wrt C++ objects alive when node complete bootstrap, all others live for temporary. These 4 objects are:

  • 2 TTYWrap objects associate with stdout and stderr
  • 1 SignalWrap object listens signal 'SIGWINCH' of tty
  • 1 ChannelWrap object is defaultResolver in lib/dns.js which is not used during bootstrap

Since we don't need to open tty during bootstrap, there will be no wrt C++ objects once refactor is done.

@hashseed

I guess this is something we could already get started with even before the necessary V8 API becomes available on master. I would definitely be willing to help.

@jasnell

I've been looking at introducing a proper command line args parser and start up options isolation that will be tied to the Environment rather than using global state, then will be separating out the initialization of the process object. Once that is done, we should be able to refactor the bootstrap script into distinct phases.

I think we could refactor the bootstrap script simultaneously, by setting up interceptor to process object for checking runtime arguments access. See here

@liqyan

This comment has been minimized.

Copy link
Contributor

liqyan commented Feb 7, 2018

Status update:
Most native modules used during bootstrap have been modified that those procedures that accessing runtime arguments have been postponed until runtime arguments accessing is allowed.

Most tests passed except some message tests that strictly compare exception stacks, and some tests that treating native module as normal module by using relative path, such as require('../../lib/buffer').

code is here

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Feb 7, 2018

Status update on this side: I've started working on refactoring the native bootstrap lifecycle.... starting with how configuration and command line options are handled.... specifically, moving away from use of global variables to using a NodeOptions class associated with the Environment. From there, I will be working through a refactoring of how the process object is bootstrapped.

@liqyan

This comment has been minimized.

Copy link
Contributor

liqyan commented Mar 7, 2018

Hi @jasnell, would you please update the status of working on refactoring native bootstrap?

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Apr 13, 2018

This seems to be blocked by current bootstrapping code being incompatible with snapshotting. I think we have a chicken-egg problem here. Without clean bootstrapping code, we don't get snapshotting. Without snapshotting, there is little incentive to clean up bootstrapping. Though I think there has been some progress with refactorings.

How about we approach this incrementally. I think snapshotting right after lib/internal/bootstrap/loaders.js might already work?

@devsnek

This comment has been minimized.

Copy link
Member

devsnek commented Apr 13, 2018

@hashseed can you come up with a sort of action item list that contributors can use to figure out what changes need to be made? I assume it's along the lines of separating loading all our code from stuff like checking cli arguments and such?

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Apr 13, 2018

I guess a list would look something like this:

  • Collect v8::FunctionCallback, external string resources, etc. into a null-terminated intptr_t* array that can be passed to v8::SnapshotCreator. This would look like this from @liqyan's prototype.
  • Implement persisting and restoring eternal handles from the snapshot, like here.
  • Implement a new build step to create the snapshot. Node.js would run up to bootstrapping and then capture a snapshot. This could either be between executing two scripts, e.g. lib/internal/bootstrap/loaders.js and lib/internal/bootstrap/node.js, or wrapping bootstrapping in an async function so that at the point where we create the snapshot, there is no V8 frames on the stack.
  • Make sure that before snapshotting, we only perform steps that are independent from CLI flags and environment variables. Ideally, we would move as much as possible to before snapshotting, but this could be done incrementally. The proposal laid out by @liqyan looks reasonable to me.
  • Implement a way for Node.js bootstrapping to continue from deserializing from a context.
  • Implement a way to serialize and deserialize C++ objects referenced by the snapshot, like done here.
@joyeecheung

This comment has been minimized.

Copy link
Member

joyeecheung commented Apr 14, 2018

We could also go over the list produced by running process._rawDebug(process.moduleLoadList.join('\n')) and try not to load unnecessary modules during bootstrapping. For example there are a lot of modules that seem to be irrelevant to bootstrapping in there (e.g. dns, tcp), most of them are eagerly loaded because of the instantiation of the global console and we are not really careful about how we load the built-in modules in the built-in modules. Even acorn was loaded during bootstrapping before #19863

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Apr 14, 2018

With snapshotting preloading modules becomes advantageous because bootstrapping becomes a lot cheaper.

It can't hurt to have a list of FunctionCallbacks already.

@devsnek

This comment has been minimized.

Copy link
Member

devsnek commented Apr 16, 2018

sorta a sidenote... with this separation i wonder if we can somehow also mark all the functions we create before user code runs as native for toString purposes

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Apr 16, 2018

sorta a sidenote... with this separation i wonder if we can somehow also mark all the functions we create before user code runs as native for toString purposes

That's not exposed, but could easily be implemented. All you need to do is to mark script type as v8::internal::Script::TYPE_NATIVE.

Last time I brought this up I was told that exposing the source of core libs is intended. And I agree that it is useful.

@Trott

This comment has been minimized.

Copy link
Member

Trott commented Oct 26, 2018

Is this an issue that should remain open? Or is this close-able at this time?

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Oct 26, 2018

This is a slow going one but it's still active (even if it doesn't appear to be ;)...). I'll leave it up to you on whether the issue should remain open

@Trott

This comment has been minimized.

Copy link
Member

Trott commented Oct 26, 2018

This is a slow going one but it's still active (even if it doesn't appear to be ;)...). I'll leave it up to you on whether the issue should remain open

That sounds like it should be left open for now.

@Trott

This comment has been minimized.

Copy link
Member

Trott commented Nov 25, 2018

(How do we know when this is complete and can be closed?)

@hashseed

This comment has been minimized.

Copy link
Member Author

hashseed commented Nov 25, 2018

We would be done once we use snapshots for Node.js start up :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment