Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native node modules from extensions #658

Open
aaronpowell opened this Issue Nov 25, 2015 · 23 comments

Comments

Projects
None yet
10 participants
@aaronpowell
Copy link

commented Nov 25, 2015

I’m looking to build an extension but one of the packages I need to depend on is a native module. When I do an npm install it installs the package just fine, I can launch a node shell and interact with it, etc. But when I go to use the package from within my plugin I get an error as it’s trying to load the ia32 build of the native module not the x64 which was compiled by node-gyp.

A quick bit of debugging indicates that the problem stems by process.arch returning ia32 when running in VS Code when my machine is a x64 machine (Win10 x64).

So is there some way to either:

  • Have VS Code run an x64 process
  • Have VS Code’s ia32 process do my npm restore (and subsequent compile with node-gyp)

Otherwise I fear that my extension might be dead in the water 😦

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Nov 26, 2015

I'm doing some more investigation on this to see if I can unblock myself by getting an ia32 version of the module loaded, so I installed the x86 version of node.js, installed the native module (which in turned got the right binary).

Now I get the following error:

native-module-error

Are native modules not supported in extensions?

@joaomoreno

This comment has been minimized.

Copy link
Member

commented Nov 26, 2015

The good: Nothing really.
The bad: Code extensions unfortunately don't support native extensions.
The ugly: There is a workaround. Bear with me:

  • Your native packages must be compiled against the exact same version of node and v8 that Code is. Specifically:
    • npm_config_disturl=https://atom.io/download/atom-shell
    • npm_config_target= whatever value is there on electronVersion
    • npm_config_arch= whatever the bit architecture you need
    • HOME= something other that ~, like ~/.electron-gyp, so node-gyp and npm don't confuse themselves.
  • The extension package must contain the binaries for all platforms Code supports: win32-ia32, darwin, linux-ia32 and linux-x64. It would then dynamically require the correct binary for the running platform at runtime. bindings might help you there.

@joaomoreno joaomoreno removed their assignment Nov 26, 2015

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Nov 26, 2015

Yeah I just came across this walkthrough which somewhat outlines the "ugly" side of things 😛.

Native modules are a pain, but I might keep at it and see where I can get to (or see if the native module can be reimplemented in "plain ol' JavaScript", but somehow I don't expect it to be possible.

@joaomoreno

This comment has been minimized.

Copy link
Member

commented Nov 26, 2015

We use the npm way in our build scripts, btw.

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Nov 30, 2015

I've been playing around with the easy way and got this far:

  • electron-rebuild generated a binary output
  • My extension still failed as the path was not matching the generated output
  • Modified the folder of the generated output and it worked

So - I now have a "working" native node package running in a VSCode extension. But I was curious as to why it wanted a folder of node-v46... to load the binary. It seems that the logic to load the binary uses the node-gyp build/versioning lookup which might be somewhat broken in VSCode. The failure is that internally it does some checks to whether it's node, node-wekbit or electron, but process.versions.electron is undefined in the VSCode plugin. My guess is that because the extension is running out-of-process it's not getting that passed through, which in turn makes it a little unhappy.

Anyway I can at least get the extension launched, so chalk that up to success number 1 😁.

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Dec 1, 2015

I've hit another problem that I'm not sure if it's related to it being a native module or if it's another bug, the native module is being reported as not found by VSCode's editor and the TypeScript compiler (when running the package step).

If I run from the debugger it works just fine, the native module is loaded, executed, etc, so I know it's there, but tsc doesn't seem to locate it. I think it's a TypeScript issue than a VSCode issue, but I can't find any info on it. Is it known at the VSCode end?

@joaomoreno

This comment has been minimized.

Copy link
Member

commented Dec 1, 2015

It seems that you just need TypeScript typings to go along your native module. Try placing a file in your extension similar to this:

mylib.d.ts

declare module 'mylib' {
    export function foo(arg: number): void;
}

Then, both VSCode and TypeScript would drop the errors when you'd import the module:

import * as mylib from 'mylib';
@aaronpowell

This comment has been minimized.

Copy link
Author

commented Dec 2, 2015

ah, I didn't realise you need to have everything with a .d.ts. All sorted and now it's on the marketplace - https://marketplace.visualstudio.com/items/aaronpowell.vscode-espruino (although I'm not sure it'll be installable for anyone else yet).

Is there any way (easy or ugly) to work out the version of Electron when installing a plugin?

@joaomoreno

This comment has been minimized.

Copy link
Member

commented Dec 2, 2015

You must always compile it against the version of Electron that VSCode depends on, considering the VSCode version you're targeting. So, for VSCode 0.10.3 it is Electron 0.34.1.

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Dec 2, 2015

Yeah I found that out, I'm just trying to avoid shipping a precompiled version of serialport and instead compile it as an install step, but I need to work out what version of VSCode is running (and that tells me what version of Electron)

@joaomoreno

This comment has been minimized.

Copy link
Member

commented Dec 3, 2015

Very tricky and it's exactly the reason why we haven't tackled this yet: every user that would install your extension would need the build tooling installed in their system (VS, gcc, etc.).

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Dec 3, 2015

Yeah that's a fair point, I guess shipping the compiled module in my extension is probably a good idea (or at least the binary) and get it done for each platform.

I might be able to make an assumption of the build tools (given that it's work working with microprocessors) but it's solid assumption.

@mateodelnorte

This comment has been minimized.

Copy link

commented Jul 28, 2016

Is there a mechanism for downloading precompiled binaries as part of the build process, such that extension creators could use a tool to pre-compile an array binaries for different platform and node version targets - and the build process would bring down the appropriate binary and link it?

@joaomoreno

This comment has been minimized.

Copy link
Member

commented Jul 28, 2016

node-pre-gyp exists for that purpose when dealing with node modules. But upon installation, we don't call npm install on an extension.

@Tyriar

This comment has been minimized.

Copy link
Member

commented Aug 1, 2016

So Atom's apm 1.10 beta seems to ship with node and npm now, likely to avoid these exact issues http://blog.atom.io/2016/08/01/atom-1-9-and-1-10-beta.html

@joaomoreno

This comment has been minimized.

Copy link
Member

commented Aug 2, 2016

Yep... it's starting to feel inevitable by now.

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Aug 4, 2016

I haven't done much with this for a while but my current thinking is that instead of trying to load the native module within the electron shell I think it'll be better to use spawn and uxe the native module from the default node.js install (which obviously requires there to be a node.js install).

I'm thinking this because it seems that the electron-rebuild module no longer works on Windows or with the version of electron being used.

@RLovelett

This comment has been minimized.

Copy link

commented Aug 18, 2016

Is there anyway to see the error messages upon faulty loading?

I am trying to make a language server that makes use of a native module. I've followed the steps outlined here and here and have arrived at a module that can be loaded from the extension. So far so good.

However, when my extension starts the server that tries to reference the same native module I get "The Swift server crashed 5 times in the last 3 minutes. The server will not be restarted."

This does not exactly give me a lot to go on to try and resolve where it is going wrong. And the process is dead before I have a chance to attach a debugger to it. If I could just get some messaging about what/why the server is dying that may help me crack the case.

@aaronpowell

This comment has been minimized.

Copy link
Author

commented Aug 18, 2016

@RLovelett the problem is now that the version of Electron used in VSCode is a bit out of date (0.37.6 in the insider build of 2016-08-17 that I'm running). It seems that electron-rebuild no longer works with that version (the problem is with nslog dependency) and I'm not sure if it's being maintained.

This is why, for me, I'm thinking of running node in a separate process and passing messages back and forth so I can use a "standard" version of node, rather than the forked version used by electron. Not sure if that'd work in your scenario though.

@TomasHubelbauer

This comment has been minimized.

Copy link
Contributor

commented May 27, 2018

Is this thread still up to date or have things changed since 2016? I see the mention of Atom shipping with Node and NPM, has VS Code started doing the same or is depending on native modules still cumbersome?

@ncannasse

This comment has been minimized.

Copy link

commented Nov 18, 2018

I'm maintaining a debugger which requires several native libraries. It requires me to push a new version everytime VSCode updates its Electron runtime (which 1.29 did again, from 2.0.5 to 2.0.12, which was preventing my precompiled extensions to load).

This is far from ideal, and while I understand this only concern a few extensions, maybe there should be a standard way to trigger an installation script so we can deal with it ourselves?

@octref octref referenced this issue Nov 20, 2018

Open

Document using native modules #2211

0 of 3 tasks complete
@octref

This comment has been minimized.

Copy link
Member

commented Nov 20, 2018

While I can't promise adding any support to make it easier to use native modules, we can add some documentations for the current state: Microsoft/vscode-docs#2211

@akbyrd

This comment has been minimized.

Copy link

commented Jan 6, 2019

Documentation would be great. This is definitely a nasty corner case to figure out for newcomers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.