Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Node.js to run with Microsoft's ChakraCore engine #4765

Closed
wants to merge 9 commits into from

Conversation

@kunalspathak
Copy link
Contributor

kunalspathak commented Jan 19, 2016

(Note from the CTC (Fishrock123): This thread is expected to garner a lot of attention. Comments that are not productive to discussing the technical aspects may be removed.)


Enable Node.js to optionally build and run on Microsoft's ChakraCore JavaScript engine.

We implemented a V8 API shim (aka chakrashim) which takes advantage of ChakraCore runtime hosting APIs (JSRT). This shim implements most essential V8 APIs so that the underlying JavaScript engine changes are transparent to Node.js and native addon modules written for V8.

Here is the summary of commits :

  • chakrashim source code
  • ChakraCore v1.1.0.1 source code
  • build script changes to build Node.js with ChakraCore on Windows OS for x86, x64 and ARM
  • node-gyp changes to build Node.js with ChakraCore on Windows OS for x86, x64 and ARM 1
  • gyp changes to to build chakrashim and ChakraCore on Windows OS for x86, x64 and ARM 2
  • node source code changes needed to run on ChakraCore
  • unit test updated for ChakraCore

1 node-gyp changes won't directly be submitted via PR, but will be first submitted to upstream branch as pointed out in nodejs/node-gyp#873. They are included in PR so reviewers can build and test Node.js with chakracore. Node-gyp changes are divided in two commits, first is based on changes needed to build node.js with chakracore for x86 and x64 for Windows OS and other based on changes need to support building it for ARM architecture for Windows OS.

2 gyp changes are also divided in two commits. One for x86/x64 and other for ARM.

ChakraShim implements essential V8 APIs needed by Node.js on top of the
Chakra runtime hosting API (JSRT). This enables Node.js and other native
addon modules written for V8 to build and run with the Chakra JavaScript
engine.
Source code of [chakracore](https://github.com/Microsoft/ChakraCore.git)
that lights up Node.js for Chakra.
Building Node.js with Chakra produces chakracore.dll along with other binaries
that is needed by node.exe to function.
@Fishrock123

This comment has been minimized.

Copy link
Member

Fishrock123 commented Jan 19, 2016

@kunalspathak Thanks for the PR! This has a very large impact so please expect review to take a considerable amount of time. :)

@daekano

This comment has been minimized.

Copy link

daekano commented Jan 19, 2016

VERY cool! Don't envy the code reviewers here though!

@joshmanders

This comment has been minimized.

Copy link

joshmanders commented Jan 19, 2016

This is by far one of the coolest things I've seen happen to Node.js since the convergence of io.js and Node.js.

@MikeFielden

This comment has been minimized.

Copy link

MikeFielden commented Jan 19, 2016

Legit. Thanks, cant wait until this happens 👍

@unicodeveloper

This comment has been minimized.

Copy link

unicodeveloper commented Jan 19, 2016

The review will really take a lot of time and scrutiny! 🍴

@SilverIgniter

This comment has been minimized.

Copy link

SilverIgniter commented Jan 19, 2016

+1️⃣ guys, really good job! 💥

@@ -10,6 +10,8 @@
'component%': 'static_library', # NB. these names match with what V8 expects
'msvs_multi_core_compile': '0', # we do enable multicore compiles, but not using the V8 way
'python%': 'python',
'node_engine%': 'v8',

This comment has been minimized.

Copy link
@robcolburn

robcolburn Jan 19, 2016

Nit: is node_engine established elsewhere? Maybe js_engine? This line and everywhere else for consistency

This comment has been minimized.

Copy link
@indutny

indutny Jan 19, 2016

Member

We usually just prefix node stuff with node_ in here.

This comment has been minimized.

Copy link
@robcolburn

robcolburn Jan 19, 2016

cool, cool

@ingIor

This comment has been minimized.

Copy link

ingIor commented Jan 19, 2016

This is incredibly cool, I have a request.

Please do not post comments on the GH issue unless you have something important to add. These issues gain a lot of attention and it makes it incredibly hard for collaborators to communicate.

Locking the issue to collaborators means other people from the outside who have a significant ***** contribution or want to help can't do that.

Comments like +1 -1 and such create a significant amount of noise.

Support open source, keep the discussion clean.

EDIT(@trevnorris): Possible accidental use of inappropriate word? Has been removed.

@trevnorris

This comment has been minimized.

Copy link
Contributor

trevnorris commented Jan 19, 2016

@kunalspathak What happens when v8 needs to be updated and the API doesn't match the Chakra shim? Will we need to hold off until the shim also has an update?

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Jan 19, 2016

@kunalspathak ... thank you for submitting this. As you can imagine, it's going to be a big review and could take some time to settle out.

@trevnorris ... I have separately reached out to each of the V8 and Chakra teams and invited both to sit down face to face to work through the API/ABI impact of this change and figure out how we can make the ABI layer more robust. I'm working out the logistics for that face to face now and want to make sure to extend the invite to all the @nodejs/ctc members as well. There are a ton of questions this brings up and I think sitting down for an afternoon to hash things out would be quite productive.

@benjamingr
benjamingr reviewed Jan 19, 2016
View changes
configure Outdated
@@ -1101,6 +1112,10 @@ def configure_intl(o):
pprint.pformat(icu_config, indent=2) + '\n')
return # end of configure_intl

def configure_engine(o):
engine = options.engine or 'v8'

This comment has been minimized.

Copy link
@benjamingr

benjamingr Jan 19, 2016

Member

Prefer function default parameter values def configure_engine(o = 'v8'):

This comment has been minimized.

Copy link
@ChALkeR

ChALkeR Jan 19, 2016

Member

But it's not a function parameter.

This comment has been minimized.

Copy link
@pritambaral

pritambaral Jan 19, 2016

Then default options.engine at nodejs/node-chakracore@12cc5d9#diff-e2d5a00791bce9a01f99bc6fd613a39dR380

    dest='engine',
+   default='v8',
    help='Use specified JS engine (default is V8)')

This comment has been minimized.

Copy link
@kunalspathak

kunalspathak Jan 19, 2016

Author Contributor

@pritambaral , @benjamingr - Default is v8 engine as seen in vcbuild.bat

This comment has been minimized.

Copy link
@pritambaral

pritambaral Jan 19, 2016

@kunalspathak Yes, but the line commented-upon here is in configure and not vcbuild.bat. It would be better if the or-set in this line were replaced with the very first assignment of options.engine, which is what I think @benjamingr intended.

This comment has been minimized.

Copy link
@kunalspathak

kunalspathak Jan 19, 2016

Author Contributor

Fair point.

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Jan 19, 2016

Marking it as semver-major for now (just to be safe), although semver-minor might be appropriate once we get further along in the review. Very happy to see this tho.

@@ -0,0 +1,54 @@
## Contributing Code

This comment has been minimized.

Copy link
@benjamingr

benjamingr Jan 19, 2016

Member

Is there any need for these files here?
Why isn't ChakraCore brought as a dependency?

This comment has been minimized.

Copy link
@MrRio

MrRio Jan 19, 2016

It's because this is the shim, the Core is brought in as a dep, but this is closely linked to node.

This comment has been minimized.

Copy link
@kunalspathak

kunalspathak Jan 19, 2016

Author Contributor

Thats right. The sources in core folder are from Microsoft/ChakraCore. It is just easier to update the chakracore sources in node deps when they are aligned with their origin in github counterpart.

This comment has been minimized.

Copy link
@panuhorsmalahti

panuhorsmalahti Jan 19, 2016

Why not use a git submodule?

This comment has been minimized.

Copy link
@jianchun

jianchun Jan 19, 2016

Why not use a git submodule?

This follows the convention of existing deps -- source copy instead of submodule. It allows more flexibility such as making changes here before upstream when necessary (Not familiar with submodule, don't know if that's also supported).

This comment has been minimized.

Copy link
@MylesBorins

MylesBorins Jan 19, 2016

Member

@panuhorsmalahti node currently does not use submodules for dependencies. It may be a good idea to document how dependencies are updated.

Currently for dependencies such as v8, libuv, or npm we take version updates as a single commit updating the state of the current directory, generally from a collaborator of the project we are downstream from. In between large updates we will sometimes backport changes as individual patch's.

This comment has been minimized.

Copy link
@panuhorsmalahti

panuhorsmalahti Jan 20, 2016

@jianchun: okay, I didn't know about that convention. I disagree with it, but I guess changing that convention is another discussion. To answer your question, if the upstream needs to be modified without changing e.g. the master branch, a new branch should be created upstream.

This comment has been minimized.

Copy link
@sam-github

sam-github Jan 21, 2016

Member

@panuhorsmalahti git submodules don't allow the thirdparty source to be patched, or work with thirdparty deps that are not available via git.

@Qix-

This comment has been minimized.

Copy link

Qix- commented Jan 19, 2016

Maybe this is a stupid question, but what is the advantage to this PR? To use Chakra instead of V8, right? Is that a runtime specification or a build specification?

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Jan 19, 2016

Am I correct that this currently works only on Windows platform? Initial Linux support is still on the roadmap of ChakraCore: https://github.com/Microsoft/ChakraCore/wiki/Roadmap. And that is decribed as «make it link, make it run, no JIT».

@sethgaurav

This comment has been minimized.

Copy link

sethgaurav commented Jan 19, 2016

Yes, its currently Windows only. For cross-platform support, the key target for next six months on the roadmap is to get the interpreter & runtime working. JIT would come after that (don't read it as no JIT forever - it was just a breakdown of what we need enable and its ordering for next 6 mos).

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Jan 19, 2016

Will ChakraCore be treated as a first-class citizen? That is:

  1. Will there be binary builds with ChakraCore for each Node.js release provided from the nodejs.org site?
  2. Would it be guaranteed that ChakraCore must work with every next Node.js version, or would it be ok if v6.2.0 builds with ChakraCore, v7.0.0 breaks with ChakraCore (let's say the shim update didn't get in in due time), and v7.1.0 works again with ChakraCore?
  3. What if v7.3.0 breaks again with ChakraCore in a semver-minor version, because of some additive change that didn't get yet ported to the shim?
@mikeal

This comment has been minimized.

Copy link
Contributor

mikeal commented Jan 19, 2016

@ChALkeR I think those questions can't be answered until after this PR lands. We have support in Node.js for architectures that we don't have in CI and pretty much all the ones we do have were added some time after support for the architecture was landed. I don't think anyone wants to add another vector to CI until we know there is some stability here and we're only going to figure that out after it lands.

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Jan 19, 2016

@ChALkeR ... at least in my opinion, it depends on what is meant by "first-class citizen"

  1. For the very near term, likely not. Having officially supported builds of Node+ChakraCore is likely quite some time off and wouldn't be able to happen at least until the cross-platform work is done. Similar to how we (IBM) handled our Node-on-PPC and Node-on-AIX builds, it would be more likely that Msft would host their own downloads with builds of Node+ChakraCore (please correct me if I'm wrong @orangemocha or @kunalspathak), at least until a decision is made to officially provide Chakra support.

2 and 3: I think it's too early to determine that. Given the dependency on the V8 APIs currently, and given how quickly those could change, there will need to be dedicated effort by Microsoft and other interested collaborators to ensure that the Chakra shim continues to work. It is entirely possible that semver-major update of V8 could land that could break these. Until we're sure that the shim is stable, we likely won't want to commit to ensuring that things would continue to work.

+1 to what @mikeal just said as I was typing this :-)

What I'd personally like to see is getting this landed but clearly indicating that it is experimental and unsupported for the time being.

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Jan 19, 2016

@mikeal @jasnell You mean that the initial answer is «no» to all questions (i.e.: no guarantees at all for the time being), with a possibility to change in the future? That seems legit.

@mikeal

This comment has been minimized.

Copy link
Contributor

mikeal commented Jan 19, 2016

Yup, this is sort of a classic "let's put this behind a flag and see what happens" situation :)

@Fishrock123

This comment has been minimized.

Copy link
Member

Fishrock123 commented Jan 19, 2016

Yup, this is sort of a classic "let's put this behind a flag and see what happens" situation :)

As a note, this is not a runtime feature, so it is explicitly behind a build-time flag.

@vkurchatkin

This comment has been minimized.

Copy link
Member

vkurchatkin commented Jan 19, 2016

It seems like a nightmare to support.

Imagine the following: someone make a PR that uses v8 API. CI breaks on Chakra build, because shim doesn't support said API. What we are going to do? wait until Chakra implements them? abandon the PR?

What if next week someone PRs support for JSC or SpiderMonkey?

@orangemocha

This comment has been minimized.

Copy link
Contributor

orangemocha commented Feb 11, 2016

And here is node-chakracore: https://github.com/nodejs/node-chakracore ! Hoping to get lots of contributions 😄

MylesBorins added a commit that referenced this pull request Feb 17, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (#4765) for obvious reason.

PR-URL: #4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
MylesBorins added a commit that referenced this pull request Feb 18, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (#4765) for obvious reason.

PR-URL: #4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
@kunalspathak

This comment has been minimized.

Copy link
Contributor Author

kunalspathak commented Feb 19, 2016

As @orangemocha mentioned these changes are ported to new repo https://github.com/nodejs/node-chakracore chakracore-master branch. Weekly merge from nodejs.node master and chakrashim changes will happen in this branch of new repo. Feel free to contribute and providing feedback. Thanks everyone for the support!

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Feb 19, 2016

@orangemocha Does this still need ctc-agenda?

@orangemocha orangemocha removed the ctc-agenda label Feb 19, 2016
@orangemocha

This comment has been minimized.

Copy link
Contributor

orangemocha commented Feb 19, 2016

Removed ctc-agenda. We'll ping the CTC periodically to provide updates and get feedback on the direction of the project. Thank you.

MylesBorins added a commit that referenced this pull request Mar 2, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (#4765) for obvious reason.

PR-URL: #4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
@itsjavi

This comment has been minimized.

Copy link

itsjavi commented Mar 23, 2016

Abstracting Node from the underlying engine sounds good, but we should avoid incompatibilities, otherwise we may start to see packages only working for a specific engine that will not be reusable in the other, so here comes the potential fragmentation.
Please, let's not bring cross-platform headache to Node, make the differences minimal, defining a clear and common API, that's all I would ask.
Great job btw.

@isiahmeadows

This comment has been minimized.

Copy link

isiahmeadows commented Mar 26, 2016

@mjolnic I agree, but I will note that Browserify/Webpack isn't helping that too much. I do feel that NAN might help alleviate much of that concern, though, but it'll have to go through a major version change to account for all the additional abstraction boilerplate it'll need.

I'm also staring intently on WebAssembly, which will also be runnable on future versions of Node, with Chakra and V8 both supporting it, and that may have an even more significant impact on native addons (more than even supporting Chakra IMHO, because of the language difference).

@jfbastien

This comment has been minimized.

Copy link

jfbastien commented Mar 26, 2016

From Node's POV WebAssembly will look like an opaque blob of JS. You can call into it from JS, it can call out to JS, but the API surface is the same as an opaque JS blob.

Eventually WebAssembly may gain access to extra APIs, but the MVP won't have anything special. If / when WebAssembly gains new APIs I think they should be designed with Node in mind, not just browsers.

scovetta pushed a commit to scovetta/node that referenced this pull request Apr 2, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (nodejs#4765) for obvious reason.

PR-URL: nodejs#4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
@htoooth htoooth mentioned this pull request Apr 18, 2016
kunalspathak added a commit to kunalspathak/node-gyp that referenced this pull request Dec 12, 2016
Microsoft's chakracore engine is dependent on Windows SDK, and build tools
should know the version installed on user machine. This change adds those
dependencies in node-gyp tools. Below is the summary:

* Configure msvs_windows_target_platform_version to use the right
   Windows SDK.
* Configure msvs_use_library_dependency_inputs to export symbols
   correctly (otherwise functions not used by node.exe but might be
   needed by native addon modules could be optimized away by linker).

 These changes were originally made in nodejs/node#4765, but as @shigeki
 mentioned, it was more sensible to send these changes as PR to node-gyp
 repo.
kunalspathak added a commit to kunalspathak/node-gyp that referenced this pull request Jan 19, 2017
Microsoft's chakracore engine is dependent on Windows SDK, and build tools
should know the version installed on user machine. This change adds those
dependencies in node-gyp tools. Below is the summary:

* Configure msvs_windows_target_platform_version to use the right
   Windows SDK.
* Configure msvs_use_library_dependency_inputs to export symbols
   correctly (otherwise functions not used by node.exe but might be
   needed by native addon modules could be optimized away by linker).

 These changes were originally made in nodejs/node#4765, but as @shigeki
 mentioned, it was more sensible to send these changes as PR to node-gyp
 repo.
bnoordhuis added a commit to nodejs/node-gyp that referenced this pull request Feb 15, 2017
Microsoft's chakracore engine is dependent on Windows SDK, and build
tools should know the version installed on user machine. This change
adds those dependencies in node-gyp tools. Below is the summary:

* Configure msvs_windows_target_platform_version to use the right
  Windows SDK.

* Configure msvs_use_library_dependency_inputs to export symbols
  correctly (otherwise functions not used by node.exe but might be
  needed by native addon modules could be optimized away by linker).

These changes were originally made in nodejs/node#4765, but as @shigeki
mentioned, it was more sensible to send these changes as PR to node-gyp
repo.

PR-URL: #873
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
@omeid

This comment has been minimized.

Copy link

omeid commented Oct 21, 2017

Just echoing @davidmurdoch,

What are the benefits of adding this? this question is specially important when you consider the complexity that is introduced not only to the core of node.js but also the code that will be written against node, having to worry about the version of node.js is enough, don't need to compound that into a matrix of backends and versions.

@kunalspathak Can you please give some context?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.