Compiling JavaScript With Node.JS Source #1301

JacobTDC · 2018-05-27T19:34:11Z

How would I go about compiling Node.JS code using the source files and GCC/G++? I've figured out how to use tools/js2c.py to get C/C++, but how do I include all relevant header files to GCC without doing it manually?

Makefile:

default: js2c compile

compile: config
  @echo Compiling...
  @g++ main.cpp '-DNODE_WANT_INTERNALS=1' -I"./node/src" -I"./node/deps/v8/include" -I"./node/deps/uv/include" -I"./node/tools/msvs/genfiles" -I"./node/deps/cares/include" -std=gnu++1y -stdlib=libc++

config:
  @./tools/configure.sh $(nodeV)

js2c: config
  @echo Converting JavaScript to C++...
  @python2 ./node/tools/js2c.py main.cpp main.js

tools/configure.sh:

#!/bin/bash

echo "Checking for Node.js source..."
if [ ! -d "node" ]; then
  echo "Node.js source not found."
  if [ -n "$1" ]; then
    nodeV="v${1/#v}"
    echo "Getting Node.js $nodeV"
    nodeTab=$(wget "https://nodejs.org/dist/index.tab" -qO -)
    if [ -z "$(echo $nodeTab|grep "$nodeV\s")" ]; then
      echo "Node.js version \"$nodeV\" does not exist!"
      exit 1
    fi
    nodeUrl="$nodeV/node-$nodeV.tar.gz"
  else
    echo "No version specified, getting latest."
    nodeUrl="node-latest.tar.gz"
  fi
  tmpDir=$(mktemp -d "./tmp-XXXXXX")
  trap "{ rm -rf $tmpDir; }" EXIT
  echo "Downloading..."
  wget "https://nodejs.org/dist/$nodeUrl" --show-progress --progress=bar -qO -|tar xpz -C $tmpDir
  mv $tmpDir/* node
  rm -rf $tmpDir
else
  echo "Node.js source found."
fi

The text was updated successfully, but these errors were encountered:

gireeshpunathil · 2018-05-28T05:24:52Z

if you are looking for direct building Node.js from source, concrete steps are mentioned in https://github.com/nodejs/node/blob/master/BUILDING.md
if you are wanting some customization to the build, ./configure --help provides the necessary options
if you are wanting to build some custom C++ application that makes use of the object codes of the node.js's javascript source, you can declare the symbols you want as externs and link the source with node_javascript.o. I am not sure the thousands of symbols in the object file is declared in any header for consumption - probably such an app development model is not expected, I may be wrong.

let me know which is your case.

JacobTDC · 2018-05-28T16:50:24Z

I'm trying to build/compile standalone JavaScript apps with only official Node.JS source/resources. I have two reasons to not use 3rd party apps. One, they typically don't work on aarch64/Android, and two, I just want to learn.

gireeshpunathil · 2018-05-29T05:17:20Z

fair points, thanks. Still trying to figure out the exact requirement:

are you trying to build Node.js from source? basically building node executable?
are you trying to build custom JS code into C++ object code using js2c.py ?
are you trying to build a custom node executable with custom JS code in it?

gireeshpunathil · 2018-05-29T05:17:57Z

/cc @nodejs/build @nodejs/core

benjamingr · 2018-05-29T06:53:56Z

I don’t think tagging 86 people (core) in a help request waiting for author input is great use of core’s time.

gireeshpunathil · 2018-05-29T07:07:40Z

@benjamingr - point taken. Apologies to 86 people whose time was potentially wasted.

JacobTDC · 2018-05-29T12:58:21Z

@gireeshpunathil option 2, using js2c.py, and then compile that C++ code. I can't figure out how to compile it. If it's impossible, that's no big deal, I'm just doing this for fun/educational purposes.

devsnek · 2018-05-29T13:05:02Z

@benjamingr @gireeshpunathil I would argue those teams exist to be pinged.

@JacobTDC the output of js2c.py is just the source of the JavaScript but in char array form. it's interpreted as a string by node.js at runtime and evaluated. it cannot run by itself.

benjamingr · 2018-05-29T13:28:19Z

@devsnek those teams exist to be pinged at core issues relating to collaborators.

This is a help request from a new user - important, but not important enough to encourage collaborator overcommitment IMO. I think this is a problem we have and should get better at openjs-foundation/summit#64 . Sorry for the interruption :)

JacobTDC · 2018-05-29T15:10:04Z

@devsnek, yes, but by including the Node.JS and V8 source, shouldn't it be possible to compile that into a standalone?

devsnek · 2018-05-29T15:13:20Z

@JacobTDC yes, that's what the final binary node is. i'm not quite sure what your question is now

JacobTDC · 2018-05-29T15:29:39Z

@devsnek, see EDIT above. That should clarify things.

gireeshpunathil · 2018-05-30T07:38:14Z

@JacobTDC - can you try this?

g++  '-DNODE_WANT_INTERNALS=1'  -I../src  -I../deps/v8/include  -std=gnu++1y -stdlib=libc++  -c main.o main.cc

this is not backed up by any documentation, instead through following Node's build procedure for the JS files, and reducing the command line to a bare minimum that is required for most common cases.

richardlau · 2018-05-30T10:18:00Z

Have you considered building Node.js from source using --link-module and then manually looking at the generated makefile after the configure step?

JacobTDC · 2018-05-30T13:47:22Z

@gireeshpunathil I'm getting a new error I haven't seen before, but all the other errors are gone:

In file included from main.cpp:3:
In file included from ./node/src/node_javascript.h:27:
In file included from ./node/src/node_internals.h:28:
./node/src/node_mutex.h:5:10: fatal error: 'uv.h' file not found
#include "uv.h"
         ^~~~~~
1 error generated.
make: *** [Makefile:3: compile] Error 1

JacobTDC · 2018-05-30T13:52:41Z

I included does/uv/include to get this error:

main.cpp:43:10: error: use of undeclared identifier
      'internal_bootstrap_loaders_value'
  return internal_bootstrap_loaders_value.ToStringChecked(env->isolate());
         ^
main.cpp:47:10: error: use of undeclared identifier
      'internal_bootstrap_node_value'
  return internal_bootstrap_node_value.ToStringChecked(env->isolate());
         ^
2 errors generated.
make: *** [Makefile:3: compile] Error 1

devsnek · 2018-05-30T13:54:51Z

node depends on some specific files already existing (internal/bootstrap/node.js and internal/bootstrap/loaders.js)

JacobTDC · 2018-05-30T19:02:23Z

Also, @gireeshpunathil, where exactly did you find that?

JacobTDC · 2018-05-30T19:07:48Z

@devsnek, how do I include those two files?

gireeshpunathil · 2018-05-31T04:59:22Z

my approach was simple:

build node (./configure && make)
touch one of the '.js' files (say ./lib/net.js)
build again this time with verbose on (make -C out BUILDTYPE=Release V=1)

the commands generated and output are precisely the ones that are required for compiling and linking the JS files.

internal_bootstrap_loaders_value is a static structure inside the generated node_javascript.cc . I am assuming that you will eventually be linking with node_javascript.o as well. If so, you could declare internal_bootstrap_loaders_value as extern in your source, and the linker will find it when you link the objects together.

JacobTDC · 2018-05-31T13:26:21Z

@gireeshpunathil, I'm still new to C++ and compilation; can I get an English translation for that last part? XD
By that I mean, can you walk me through the process of including those files?

gireeshpunathil · 2018-05-31T13:54:58Z

sure - can you tell me how are you producing your main.cpp ? i.e., the arguments you passed to tools/js2c.py ?

JacobTDC · 2018-05-31T15:31:08Z

I haven't passed any args except main.js and main.cpp, @gireeshpunathil. Sorry it took me two hours to respond...
I actually have a makefile for this to make things easier:

default: js2c compile

compile: config
  @echo Compiling...
  @g++ main.cpp '-DNODE_WANT_INTERNALS=1' -I"./node/src" -I"./node/deps/v8/include" -I"./node/deps/uv/include" -I"./node/tools/msvs/genfiles" -I"./node/deps/cares/include" -std=gnu++1y -stdlib=libc++

config:
  @./tools/configure.sh $(nodeV)

js2c: config
  @echo Converting JavaScript to C++...
  @python2 ./node/tools/js2c.py main.cpp main.js

Where ./tools/configure.sh just makes sure Node.js source is downloaded.

My ./tools/configure.sh looks like this:

#!/bin/bash

echo "Checking for Node.js source..."
if [ ! -d "node" ]; then
  echo "Node.js source not found."
  if [ -n "$1" ]; then
    nodeV="v${1/#v}"
    echo "Getting Node.js $nodeV"
    nodeTab=$(wget "https://nodejs.org/dist/index.tab" -qO -)
    if [ -z "$(echo $nodeTab|grep "$nodeV\s")" ]; then
      echo "Node.js version \"$nodeV\" does not exist!"
      exit 1
    fi
    nodeUrl="$nodeV/node-$nodeV.tar.gz"
  else
    echo "No version specified, getting latest."
    nodeUrl="node-latest.tar.gz"
  fi
  tmpDir=$(mktemp -d "./tmp-XXXXXX")
  trap "{ rm -rf $tmpDir; }" EXIT
  echo "Downloading..."
  wget "https://nodejs.org/dist/$nodeUrl" --show-progress --progress=bar -qO -|tar xpz -C $tmpDir
  mv $tmpDir/* node
  rm -rf $tmpDir
else
  echo "Node.js source found."
fi

gireeshpunathil · 2018-05-31T15:50:54Z

@JacobTDC - the missing symbol error (internal_bootstrap_loaders_value) will be resolved if you add lib/internal/bootstrap/loaders.js along with the main.js in your js2c: config target.

However, I fear you may get new errors based on additional symbols being missed.

So 2 options:

try the above option, if it goes well fair enough. for every missing symbol, identify the JS module by replacing their underscore with a slash, and strip the last token, and add it to the list for js2c.py
Follow @richardlau 's suggestion:

$ ./configure --link-module main.js
$ make

I did not try this myself (will try), but from the documentation this looks like the right path.

JacobTDC · 2018-05-31T17:33:51Z

@gireeshpunathil it worked (though I did have to change return internal_bootstrap_loaders_value to return node_lib_internal_bootstrap_loaders_value in main.cpp), but now I get the error undefined reference to main from the linker. I know this is fixable with C++, but I'm not sure exactly how. I'm also not too excited to build ALL OF NODE FROM SOURCE on my phone, and you might guess why. XD

JacobTDC · 2018-05-31T18:10:11Z

Actually, I'm getting a lot of undefined reference errors from the linker.

gireeshpunathil · 2018-06-01T05:03:38Z

@JacobTDC - good to know you are progressing.
which of my suggested method you followed? 1 or 2? I guess 1?

I'm also not too excited to build ALL OF NODE FROM SOURCE on my phone and you might guess why.

No, I am unable to guess. Why? memory? lack of ABI support? or something else?

But nevertheless, that question hints at your intention as to convert JS code into C++ and run them standalone executable? as opposed to having a Node.js runtime? That will be tough, if not impossible:

the generated C++ source is still a piece of code that makes use of V8 APIs heavilly, so it needs a v8 instance to run the code, not directly on the C++ runtime
arbitrary JS code that makes use of Node.js API abstractions cannot be built and run without Node runtime.

JacobTDC · 2018-06-01T14:09:26Z

@gireeshpunathil, I waited for an hour, and still didn't fully build Node. My phone doesn't have enough processing power to build ALL of Node in a practical amount of time.
Also, Node compiles the JavaScript at runtime, so it should be possible to compile it using Node.js source by including the necessary components from V8 and Node.js.

JacobTDC · 2018-06-01T14:19:05Z

I used option 1, BTW.

gireeshpunathil · 2018-06-01T14:32:36Z

@JacobTDC - thanks

so it should be possible to compile it using Node.js source by including the necessary components from V8 and Node.js.

agreed, that is what we are attempting. but not having the infrastructure to do that is where we are.

if you just want to run a piece of pure JS code, you could use v8 JS engine to run it
if you want to run piece of JS code that uses Node.js APIs, you could use Node.js binary installation.
if you want to compile your JS code into C++ objects and bundle into an executable, you need entire Node.js source (that contain v8 source too) to be available for building, and the computation power to support the building. Depending on your use case however, we could customize the number of files that come up for compilation. For example, have a look at ./configure --help and see what optional things exists and what you wont be needing in the compilation unit. The 2 heavyweight things I can think of are icu and ssl . If you are not looking for wider ICU coverage, apply --with-intl=none . If you can share the opensll library from the system, apply --without-ssl . This may make your device bit more fast.

JacobTDC · 2018-06-01T15:14:48Z

@gireeshpunathil, using g++ -c main.cpp [includes] I got it to output main.o. Where do I go from here?

gireeshpunathil · 2018-06-01T15:26:59Z

you need to link it with node.o (+ all other object files from node and v8) to make an executable. If you do make V=1 on a regular system (such as linux) the linking command will be printed towards the end. Pay attention to the command that has -o node as the target, that will be your command.

JacobTDC · 2018-06-01T16:12:38Z

@gireeshpunathil Where is node.o? I can't find it...

JacobTDC · 2018-06-01T16:14:23Z

Wait, NVM, just realized. Your response just clicked for me...

JacobTDC · 2018-06-01T16:26:00Z

I assume node.o is the Node.js executable?

gireeshpunathil · 2018-06-02T11:35:18Z

@JacobTDC -

if you don't have node.o that means you haven't built any node sources.
if you don't build node sources, you have to manage the v8 orchestration by yourself!
adding a simple main method in your .cpp file will satisfy the linker
but that will not take you anywhere - from the entry point, you need path that leading to the v8 engine, and some code to manage the lifecycle of your applications (that is exactly node)

At one point I assumed that you were trying to build a virtual machine by hand, but from further interactions including the last one, I realized that you are experimenting and trying to learn. That is absolutely fine, but the direction you are wanting to take is neither supported, recommended nor tested. To add to that, you seem to be wanting to have this on a platform that inhibit these experiments.

I recommend that

do your experiments in a regular systems (such as linux on x64)
first build and use node from source (https://github.com/nodejs/node/blob/master/BUILDING.md)
understand (and try) different configure options (./configure --help)
understand snapshot, and building JS into C++ (follow the build command output from V=1)
with this level of understanding, try experimenting with that on the mobile.

hope this helps.

JacobTDC · 2018-06-02T13:32:25Z

Believe it or not, my PHONE has more processing power than my computer (XD), and I just like to experiment on it. However, @gireeshpunathil, I will try using a Linux VM. Maybe that will work better.
My phone is also my only non-virtual Linux machine.

gireeshpunathil · 2018-06-08T08:15:34Z

@JacobTDC - wonderful, thanks. Let me know if you have got all the info that you wanted.

gireeshpunathil · 2018-06-19T08:36:01Z

@JacobTDC - how is it going? are we all well and close this?

JacobTDC · 2018-06-19T14:10:43Z

@gireeshpunathil I'm closing the issue now. Sorry, I forgot to close it... XD

JacobTDC · 2018-06-22T15:39:58Z

I know I just closed this, but for anyone interested, I found patches that fix Node for android here: https://github.com/termux/termux-packages/tree/master/packages/nodejs

JacobTDC closed this as completed Jun 19, 2018

Compiling JavaScript With Node.JS Source #1301

Compiling JavaScript With Node.JS Source #1301

Comments

JacobTDC commented May 27, 2018 • edited

gireeshpunathil commented May 28, 2018

JacobTDC commented May 28, 2018 • edited

gireeshpunathil commented May 29, 2018

gireeshpunathil commented May 29, 2018

benjamingr commented May 29, 2018

gireeshpunathil commented May 29, 2018

JacobTDC commented May 29, 2018

devsnek commented May 29, 2018

benjamingr commented May 29, 2018 • edited

JacobTDC commented May 29, 2018

devsnek commented May 29, 2018

JacobTDC commented May 29, 2018 • edited

gireeshpunathil commented May 30, 2018

richardlau commented May 30, 2018

JacobTDC commented May 30, 2018 • edited

JacobTDC commented May 30, 2018

devsnek commented May 30, 2018

JacobTDC commented May 30, 2018

JacobTDC commented May 30, 2018

gireeshpunathil commented May 31, 2018

JacobTDC commented May 31, 2018 • edited

gireeshpunathil commented May 31, 2018

JacobTDC commented May 31, 2018 • edited

gireeshpunathil commented May 31, 2018

JacobTDC commented May 31, 2018 • edited

JacobTDC commented May 31, 2018

gireeshpunathil commented Jun 1, 2018

JacobTDC commented Jun 1, 2018 • edited

JacobTDC commented Jun 1, 2018

gireeshpunathil commented Jun 1, 2018

JacobTDC commented Jun 1, 2018

gireeshpunathil commented Jun 1, 2018

JacobTDC commented Jun 1, 2018

JacobTDC commented Jun 1, 2018

JacobTDC commented Jun 1, 2018

gireeshpunathil commented Jun 2, 2018

JacobTDC commented Jun 2, 2018 • edited

gireeshpunathil commented Jun 8, 2018

gireeshpunathil commented Jun 19, 2018

JacobTDC commented Jun 19, 2018

JacobTDC commented Jun 22, 2018 • edited

JacobTDC commented May 27, 2018 •

edited

JacobTDC commented May 28, 2018 •

edited

benjamingr commented May 29, 2018 •

edited

JacobTDC commented May 29, 2018 •

edited

JacobTDC commented May 30, 2018 •

edited

JacobTDC commented May 31, 2018 •

edited

JacobTDC commented May 31, 2018 •

edited

JacobTDC commented May 31, 2018 •

edited

JacobTDC commented Jun 1, 2018 •

edited

JacobTDC commented Jun 2, 2018 •

edited

JacobTDC commented Jun 22, 2018 •

edited