Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Node.js to run with Microsoft's ChakraCore engine #4765

Closed
wants to merge 9 commits into from

Conversation

kunalspathak
Copy link
Member

@kunalspathak kunalspathak commented Jan 19, 2016

(Note from the CTC (Fishrock123): This thread is expected to garner a lot of attention. Comments that are not productive to discussing the technical aspects may be removed.)


Enable Node.js to optionally build and run on Microsoft's ChakraCore JavaScript engine.

We implemented a V8 API shim (aka chakrashim) which takes advantage of ChakraCore runtime hosting APIs (JSRT). This shim implements most essential V8 APIs so that the underlying JavaScript engine changes are transparent to Node.js and native addon modules written for V8.

Here is the summary of commits :

  • chakrashim source code
  • ChakraCore v1.1.0.1 source code
  • build script changes to build Node.js with ChakraCore on Windows OS for x86, x64 and ARM
  • node-gyp changes to build Node.js with ChakraCore on Windows OS for x86, x64 and ARM 1
  • gyp changes to to build chakrashim and ChakraCore on Windows OS for x86, x64 and ARM 2
  • node source code changes needed to run on ChakraCore
  • unit test updated for ChakraCore

1 node-gyp changes won't directly be submitted via PR, but will be first submitted to upstream branch as pointed out in nodejs/node-gyp#873. They are included in PR so reviewers can build and test Node.js with chakracore. Node-gyp changes are divided in two commits, first is based on changes needed to build node.js with chakracore for x86 and x64 for Windows OS and other based on changes need to support building it for ARM architecture for Windows OS.

2 gyp changes are also divided in two commits. One for x86/x64 and other for ARM.

ChakraShim implements essential V8 APIs needed by Node.js on top of the
Chakra runtime hosting API (JSRT). This enables Node.js and other native
addon modules written for V8 to build and run with the Chakra JavaScript
engine.
Source code of [chakracore](https://github.com/Microsoft/ChakraCore.git)
that lights up Node.js for Chakra.
Building Node.js with Chakra produces chakracore.dll along with other binaries
that is needed by node.exe to function.
@Fishrock123 Fishrock123 added vm discuss labels Jan 19, 2016
@Fishrock123
Copy link
Member

@Fishrock123 Fishrock123 commented Jan 19, 2016

@kunalspathak Thanks for the PR! This has a very large impact so please expect review to take a considerable amount of time. :)

@defnorep
Copy link

@defnorep defnorep commented Jan 19, 2016

VERY cool! Don't envy the code reviewers here though!

@joshmanders
Copy link

@joshmanders joshmanders commented Jan 19, 2016

This is by far one of the coolest things I've seen happen to Node.js since the convergence of io.js and Node.js.

@MikeFielden
Copy link

@MikeFielden MikeFielden commented Jan 19, 2016

Legit. Thanks, cant wait until this happens 👍

@unicodeveloper
Copy link

@unicodeveloper unicodeveloper commented Jan 19, 2016

The review will really take a lot of time and scrutiny! 🍴

@SilverIgniter
Copy link

@SilverIgniter SilverIgniter commented Jan 19, 2016

+1️⃣ guys, really good job! 💥

@@ -10,6 +10,8 @@
'component%': 'static_library', # NB. these names match with what V8 expects
'msvs_multi_core_compile': '0', # we do enable multicore compiles, but not using the V8 way
'python%': 'python',
'node_engine%': 'v8',

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: is node_engine established elsewhere? Maybe js_engine? This line and everywhere else for consistency

Copy link
Member

@indutny indutny Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually just prefix node stuff with node_ in here.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, cool

@ingIor
Copy link

@ingIor ingIor commented Jan 19, 2016

This is incredibly cool, I have a request.

Please do not post comments on the GH issue unless you have something important to add. These issues gain a lot of attention and it makes it incredibly hard for collaborators to communicate.

Locking the issue to collaborators means other people from the outside who have a significant ***** contribution or want to help can't do that.

Comments like +1 -1 and such create a significant amount of noise.

Support open source, keep the discussion clean.

EDIT(@trevnorris): Possible accidental use of inappropriate word? Has been removed.

@trevnorris
Copy link
Contributor

@trevnorris trevnorris commented Jan 19, 2016

@kunalspathak What happens when v8 needs to be updated and the API doesn't match the Chakra shim? Will we need to hold off until the shim also has an update?

@jasnell
Copy link
Member

@jasnell jasnell commented Jan 19, 2016

@kunalspathak ... thank you for submitting this. As you can imagine, it's going to be a big review and could take some time to settle out.

@trevnorris ... I have separately reached out to each of the V8 and Chakra teams and invited both to sit down face to face to work through the API/ABI impact of this change and figure out how we can make the ABI layer more robust. I'm working out the logistics for that face to face now and want to make sure to extend the invite to all the @nodejs/ctc members as well. There are a ton of questions this brings up and I think sitting down for an afternoon to hash things out would be quite productive.

@jasnell jasnell added semver-major wip labels Jan 19, 2016
@@ -1101,6 +1112,10 @@ def configure_intl(o):
pprint.pformat(icu_config, indent=2) + '\n')
return # end of configure_intl

def configure_engine(o):
engine = options.engine or 'v8'
Copy link
Member

@benjamingr benjamingr Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer function default parameter values def configure_engine(o = 'v8'):

Copy link
Member

@ChALkeR ChALkeR Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But it's not a function parameter.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then default options.engine at nodejs/node-chakracore@12cc5d9#diff-e2d5a00791bce9a01f99bc6fd613a39dR380

    dest='engine',
+   default='v8',
    help='Use specified JS engine (default is V8)')

Copy link
Member Author

@kunalspathak kunalspathak Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pritambaral , @benjamingr - Default is v8 engine as seen in vcbuild.bat

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kunalspathak Yes, but the line commented-upon here is in configure and not vcbuild.bat. It would be better if the or-set in this line were replaced with the very first assignment of options.engine, which is what I think @benjamingr intended.

Copy link
Member Author

@kunalspathak kunalspathak Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point.

@jasnell
Copy link
Member

@jasnell jasnell commented Jan 19, 2016

Marking it as semver-major for now (just to be safe), although semver-minor might be appropriate once we get further along in the review. Very happy to see this tho.

@@ -0,0 +1,54 @@
## Contributing Code
Copy link
Member

@benjamingr benjamingr Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any need for these files here?
Why isn't ChakraCore brought as a dependency?

Copy link

@MrRio MrRio Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's because this is the shim, the Core is brought in as a dep, but this is closely linked to node.

Copy link
Member Author

@kunalspathak kunalspathak Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thats right. The sources in core folder are from Microsoft/ChakraCore. It is just easier to update the chakracore sources in node deps when they are aligned with their origin in github counterpart.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use a git submodule?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use a git submodule?

This follows the convention of existing deps -- source copy instead of submodule. It allows more flexibility such as making changes here before upstream when necessary (Not familiar with submodule, don't know if that's also supported).

Copy link
Member

@MylesBorins MylesBorins Jan 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@panuhorsmalahti node currently does not use submodules for dependencies. It may be a good idea to document how dependencies are updated.

Currently for dependencies such as v8, libuv, or npm we take version updates as a single commit updating the state of the current directory, generally from a collaborator of the project we are downstream from. In between large updates we will sometimes backport changes as individual patch's.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jianchun: okay, I didn't know about that convention. I disagree with it, but I guess changing that convention is another discussion. To answer your question, if the upstream needs to be modified without changing e.g. the master branch, a new branch should be created upstream.

Copy link
Contributor

@sam-github sam-github Jan 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@panuhorsmalahti git submodules don't allow the thirdparty source to be patched, or work with thirdparty deps that are not available via git.

@Qix-
Copy link

@Qix- Qix- commented Jan 19, 2016

Maybe this is a stupid question, but what is the advantage to this PR? To use Chakra instead of V8, right? Is that a runtime specification or a build specification?

@ChALkeR
Copy link
Member

@ChALkeR ChALkeR commented Jan 19, 2016

Am I correct that this currently works only on Windows platform? Initial Linux support is still on the roadmap of ChakraCore: https://github.com/Microsoft/ChakraCore/wiki/Roadmap. And that is decribed as «make it link, make it run, no JIT».

@sethgaurav
Copy link

@sethgaurav sethgaurav commented Jan 19, 2016

Yes, its currently Windows only. For cross-platform support, the key target for next six months on the roadmap is to get the interpreter & runtime working. JIT would come after that (don't read it as no JIT forever - it was just a breakdown of what we need enable and its ordering for next 6 mos).

@ChALkeR
Copy link
Member

@ChALkeR ChALkeR commented Jan 19, 2016

Will ChakraCore be treated as a first-class citizen? That is:

  1. Will there be binary builds with ChakraCore for each Node.js release provided from the nodejs.org site?
  2. Would it be guaranteed that ChakraCore must work with every next Node.js version, or would it be ok if v6.2.0 builds with ChakraCore, v7.0.0 breaks with ChakraCore (let's say the shim update didn't get in in due time), and v7.1.0 works again with ChakraCore?
  3. What if v7.3.0 breaks again with ChakraCore in a semver-minor version, because of some additive change that didn't get yet ported to the shim?

@mikeal
Copy link
Contributor

@mikeal mikeal commented Jan 19, 2016

@ChALkeR I think those questions can't be answered until after this PR lands. We have support in Node.js for architectures that we don't have in CI and pretty much all the ones we do have were added some time after support for the architecture was landed. I don't think anyone wants to add another vector to CI until we know there is some stability here and we're only going to figure that out after it lands.

@jasnell
Copy link
Member

@jasnell jasnell commented Jan 19, 2016

@ChALkeR ... at least in my opinion, it depends on what is meant by "first-class citizen"

  1. For the very near term, likely not. Having officially supported builds of Node+ChakraCore is likely quite some time off and wouldn't be able to happen at least until the cross-platform work is done. Similar to how we (IBM) handled our Node-on-PPC and Node-on-AIX builds, it would be more likely that Msft would host their own downloads with builds of Node+ChakraCore (please correct me if I'm wrong @orangemocha or @kunalspathak), at least until a decision is made to officially provide Chakra support.

2 and 3: I think it's too early to determine that. Given the dependency on the V8 APIs currently, and given how quickly those could change, there will need to be dedicated effort by Microsoft and other interested collaborators to ensure that the Chakra shim continues to work. It is entirely possible that semver-major update of V8 could land that could break these. Until we're sure that the shim is stable, we likely won't want to commit to ensuring that things would continue to work.

+1 to what @mikeal just said as I was typing this :-)

What I'd personally like to see is getting this landed but clearly indicating that it is experimental and unsupported for the time being.

@ChALkeR
Copy link
Member

@ChALkeR ChALkeR commented Jan 19, 2016

@mikeal @jasnell You mean that the initial answer is «no» to all questions (i.e.: no guarantees at all for the time being), with a possibility to change in the future? That seems legit.

@mikeal
Copy link
Contributor

@mikeal mikeal commented Jan 19, 2016

Yup, this is sort of a classic "let's put this behind a flag and see what happens" situation :)

@Fishrock123
Copy link
Member

@Fishrock123 Fishrock123 commented Jan 19, 2016

Yup, this is sort of a classic "let's put this behind a flag and see what happens" situation :)

As a note, this is not a runtime feature, so it is explicitly behind a build-time flag.

@vkurchatkin
Copy link
Contributor

@vkurchatkin vkurchatkin commented Jan 19, 2016

It seems like a nightmare to support.

Imagine the following: someone make a PR that uses v8 API. CI breaks on Chakra build, because shim doesn't support said API. What we are going to do? wait until Chakra implements them? abandon the PR?

What if next week someone PRs support for JSC or SpiderMonkey?

@orangemocha
Copy link
Contributor

@orangemocha orangemocha commented Feb 11, 2016

And here is node-chakracore: https://github.com/nodejs/node-chakracore ! Hoping to get lots of contributions 😄

MylesBorins pushed a commit that referenced this issue Feb 17, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (#4765) for obvious reason.

PR-URL: #4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
MylesBorins pushed a commit that referenced this issue Feb 18, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (#4765) for obvious reason.

PR-URL: #4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
@kunalspathak
Copy link
Member Author

@kunalspathak kunalspathak commented Feb 19, 2016

As @orangemocha mentioned these changes are ported to new repo https://github.com/nodejs/node-chakracore chakracore-master branch. Weekly merge from nodejs.node master and chakrashim changes will happen in this branch of new repo. Feel free to contribute and providing feedback. Thanks everyone for the support!

@ChALkeR
Copy link
Member

@ChALkeR ChALkeR commented Feb 19, 2016

@orangemocha Does this still need ctc-agenda?

@orangemocha
Copy link
Contributor

@orangemocha orangemocha commented Feb 19, 2016

Removed ctc-agenda. We'll ping the CTC periodically to provide updates and get feedback on the direction of the project. Thank you.

MylesBorins pushed a commit that referenced this issue Mar 2, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (#4765) for obvious reason.

PR-URL: #4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
@itsjavi
Copy link

@itsjavi itsjavi commented Mar 23, 2016

Abstracting Node from the underlying engine sounds good, but we should avoid incompatibilities, otherwise we may start to see packages only working for a specific engine that will not be reusable in the other, so here comes the potential fragmentation.
Please, let's not bring cross-platform headache to Node, make the differences minimal, defining a clear and common API, that's all I would ask.
Great job btw.

@dead-claudia
Copy link

@dead-claudia dead-claudia commented Mar 26, 2016

@mjolnic I agree, but I will note that Browserify/Webpack isn't helping that too much. I do feel that NAN might help alleviate much of that concern, though, but it'll have to go through a major version change to account for all the additional abstraction boilerplate it'll need.

I'm also staring intently on WebAssembly, which will also be runnable on future versions of Node, with Chakra and V8 both supporting it, and that may have an even more significant impact on native addons (more than even supporting Chakra IMHO, because of the language difference).

@jfbastien
Copy link

@jfbastien jfbastien commented Mar 26, 2016

From Node's POV WebAssembly will look like an opaque blob of JS. You can call into it from JS, it can call out to JS, but the API surface is the same as an opaque JS blob.

Eventually WebAssembly may gain access to extra APIs, but the MVP won't have anything special. If / when WebAssembly gains new APIs I think they should be designed with Node in mind, not just browsers.

scovetta pushed a commit to scovetta/node that referenced this issue Apr 2, 2016
Testing this wasn't really useful, besides Object.observe is going to be
deprecated.

Also this test fails with Chakra (nodejs#4765) for obvious reason.

PR-URL: nodejs#4769
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Brian White <mscdex@mscdex.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Roman Reiss <me@silverwind.io>
kunalspathak added a commit to kunalspathak/node-gyp that referenced this issue Dec 12, 2016
Microsoft's chakracore engine is dependent on Windows SDK, and build tools
should know the version installed on user machine. This change adds those
dependencies in node-gyp tools. Below is the summary:

* Configure msvs_windows_target_platform_version to use the right
   Windows SDK.
* Configure msvs_use_library_dependency_inputs to export symbols
   correctly (otherwise functions not used by node.exe but might be
   needed by native addon modules could be optimized away by linker).

 These changes were originally made in nodejs/node#4765, but as @shigeki
 mentioned, it was more sensible to send these changes as PR to node-gyp
 repo.
kunalspathak added a commit to kunalspathak/node-gyp that referenced this issue Jan 19, 2017
Microsoft's chakracore engine is dependent on Windows SDK, and build tools
should know the version installed on user machine. This change adds those
dependencies in node-gyp tools. Below is the summary:

* Configure msvs_windows_target_platform_version to use the right
   Windows SDK.
* Configure msvs_use_library_dependency_inputs to export symbols
   correctly (otherwise functions not used by node.exe but might be
   needed by native addon modules could be optimized away by linker).

 These changes were originally made in nodejs/node#4765, but as @shigeki
 mentioned, it was more sensible to send these changes as PR to node-gyp
 repo.
bnoordhuis pushed a commit to nodejs/node-gyp that referenced this issue Feb 15, 2017
Microsoft's chakracore engine is dependent on Windows SDK, and build
tools should know the version installed on user machine. This change
adds those dependencies in node-gyp tools. Below is the summary:

* Configure msvs_windows_target_platform_version to use the right
  Windows SDK.

* Configure msvs_use_library_dependency_inputs to export symbols
  correctly (otherwise functions not used by node.exe but might be
  needed by native addon modules could be optimized away by linker).

These changes were originally made in nodejs/node#4765, but as @shigeki
mentioned, it was more sensible to send these changes as PR to node-gyp
repo.

PR-URL: #873
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
@omeid
Copy link

@omeid omeid commented Oct 21, 2017

Just echoing @davidmurdoch,

What are the benefits of adding this? this question is specially important when you consider the complexity that is introduced not only to the core of node.js but also the code that will be written against node, having to worry about the version of node.js is enough, don't need to compound that into a matrix of backends and versions.

@kunalspathak Can you please give some context?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discuss semver-major vm windows wip
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet