Skip to content

Commit

Permalink
src: add initial support for single executable applications
Browse files Browse the repository at this point in the history
Compile a JavaScript file into a single executable application:

```console
$ echo 'console.log(`Hello, ${process.argv[2]}!`);' > hello.js

$ cp $(command -v node) hello

$ npx postject hello NODE_JS_CODE hello.js \
    --sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2

$ npx postject hello NODE_JS_CODE hello.js \
    --sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2 \
    --macho-segment-name NODE_JS

$ ./hello world
Hello, world!
```

Signed-off-by: Darshan Sen <raisinten@gmail.com>
PR-URL: #45038
Backport-PR-URL: #47495
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
  • Loading branch information
RaisinTen authored and danielleadams committed Apr 11, 2023
1 parent 956f786 commit 8ea8354
Show file tree
Hide file tree
Showing 13 changed files with 506 additions and 1 deletion.
10 changes: 10 additions & 0 deletions configure.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,12 @@
default=None,
help='use on deprecated SunOS systems that do not support ifaddrs.h')

parser.add_argument('--disable-single-executable-application',
action='store_true',
dest='disable_single_executable_application',
default=None,
help='Disable Single Executable Application support.')

parser.add_argument("--fully-static",
action="store_true",
dest="fully_static",
Expand Down Expand Up @@ -1402,6 +1408,10 @@ def configure_node(o):
if options.no_ifaddrs:
o['defines'] += ['SUNOS_NO_IFADDRS']

o['variables']['single_executable_application'] = b(not options.disable_single_executable_application)
if options.disable_single_executable_application:
o['defines'] += ['DISABLE_SINGLE_EXECUTABLE_APPLICATION']

# By default, enable ETW on Windows.
if flavor == 'win':
o['variables']['node_use_etw'] = b(not options.without_etw)
Expand Down
1 change: 1 addition & 0 deletions doc/api/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@
* [Readline](readline.md)
* [REPL](repl.md)
* [Report](report.md)
* [Single executable applications](single-executable-applications.md)
* [Stream](stream.md)
* [String decoder](string_decoder.md)
* [Test runner](test.md)
Expand Down
140 changes: 140 additions & 0 deletions doc/api/single-executable-applications.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
# Single executable applications

<!--introduced_in=REPLACEME-->

> Stability: 1 - Experimental: This feature is being designed and will change.
<!-- source_link=lib/internal/main/single_executable_application.js -->

This feature allows the distribution of a Node.js application conveniently to a
system that does not have Node.js installed.

Node.js supports the creation of [single executable applications][] by allowing
the injection of a JavaScript file into the `node` binary. During start up, the
program checks if anything has been injected. If the script is found, it
executes its contents. Otherwise Node.js operates as it normally does.

The single executable application feature only supports running a single
embedded [CommonJS][] file.

A bundled JavaScript file can be turned into a single executable application
with any tool which can inject resources into the `node` binary.

Here are the steps for creating a single executable application using one such
tool, [postject][]:

1. Create a JavaScript file:
```console
$ echo 'console.log(`Hello, ${process.argv[2]}!`);' > hello.js
```

2. Create a copy of the `node` executable and name it according to your needs:
```console
$ cp $(command -v node) hello
```

3. Inject the JavaScript file into the copied binary by running `postject` with
the following options:

* `hello` - The name of the copy of the `node` executable created in step 2.
* `NODE_JS_CODE` - The name of the resource / note / section in the binary
where the contents of the JavaScript file will be stored.
* `hello.js` - The name of the JavaScript file created in step 1.
* `--sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2` - The
[fuse][] used by the Node.js project to detect if a file has been injected.
* `--macho-segment-name NODE_JS` (only needed on macOS) - The name of the
segment in the binary where the contents of the JavaScript file will be
stored.

To summarize, here is the required command for each platform:

* On systems other than macOS:
```console
$ npx postject hello NODE_JS_CODE hello.js \
--sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2
```

* On macOS:
```console
$ npx postject hello NODE_JS_CODE hello.js \
--sentinel-fuse NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2 \
--macho-segment-name NODE_JS
```

4. Run the binary:
```console
$ ./hello world
Hello, world!
```

## Notes

### `require(id)` in the injected module is not file based

`require()` in the injected module is not the same as the [`require()`][]
available to modules that are not injected. It also does not have any of the
properties that non-injected [`require()`][] has except [`require.main`][]. It
can only be used to load built-in modules. Attempting to load a module that can
only be found in the file system will throw an error.

Instead of relying on a file based `require()`, users can bundle their
application into a standalone JavaScript file to inject into the executable.
This also ensures a more deterministic dependency graph.

However, if a file based `require()` is still needed, that can also be achieved:

```js
const { createRequire } = require('node:module');
require = createRequire(__filename);
```

### `__filename` and `module.filename` in the injected module

The values of `__filename` and `module.filename` in the injected module are
equal to [`process.execPath`][].

### `__dirname` in the injected module

The value of `__dirname` in the injected module is equal to the directory name
of [`process.execPath`][].

### Single executable application creation process

A tool aiming to create a single executable Node.js application must
inject the contents of a JavaScript file into:

* a resource named `NODE_JS_CODE` if the `node` binary is a [PE][] file
* a section named `NODE_JS_CODE` in the `NODE_JS` segment if the `node` binary
is a [Mach-O][] file
* a note named `NODE_JS_CODE` if the `node` binary is an [ELF][] file

Search the binary for the
`NODE_JS_FUSE_fce680ab2cc467b6e072b8b5df1996b2:0` [fuse][] string and flip the
last character to `1` to indicate that a resource has been injected.

### Platform support

Single-executable support is tested regularly on CI only on the following
platforms:

* Windows
* macOS
* Linux (AMD64 only)

This is due to a lack of better tools to generate single-executables that can be
used to test this feature on other platforms.

Suggestions for other resource injection tools/workflows are welcomed. Please
start a discussion at <https://github.com/nodejs/single-executable/discussions>
to help us document them.

[CommonJS]: modules.md#modules-commonjs-modules
[ELF]: https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
[Mach-O]: https://en.wikipedia.org/wiki/Mach-O
[PE]: https://en.wikipedia.org/wiki/Portable_Executable
[`process.execPath`]: process.md#processexecpath
[`require()`]: modules.md#requireid
[`require.main`]: modules.md#accessing-the-main-module
[fuse]: https://www.electronjs.org/docs/latest/tutorial/fuses
[postject]: https://github.com/nodejs/postject
[single executable applications]: https://github.com/nodejs/single-executable
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Maintaining Single Executable Applications support

Support for [single executable applications][] is one of the key technical
priorities identified for the success of Node.js.

## High level strategy

From the [Next-10 discussions][] there are 2 approaches the project believes are
important to support:

### Compile with Node.js into executable

This is the approach followed by [boxednode][].

No additional code within the Node.js project is needed to support the
option of compiling a bundled application along with Node.js into a single
executable application.

### Bundle into existing Node.js executable

This is the approach followed by [pkg][].

The project does not plan to provide the complete solution but instead the key
elements which are required in the Node.js executable in order to enable
bundling with the pre-built Node.js binaries. This includes:

* Looking for a segment within the executable that holds bundled code.
* Running the bundled code when such a segment is found.

It is left up to external tools/solutions to:

* Bundle code into a single script.
* Generate a command line with appropriate options.
* Add a segment to an existing Node.js executable which contains
the command line and appropriate headers.
* Re-generate or removing signatures on the resulting executable
* Provide a virtual file system, and hooking it in if needed to
support native modules or reading file contents.

However, the project also maintains a separate tool, [postject][], for injecting
arbitrary read-only resources into the binary such as those needed for bundling
the application into the runtime.

## Planning

Planning for this feature takes place in the [single-executable repository][].

## Upcoming features

Currently, only running a single embedded CommonJS file is supported but support
for the following features are in the list of work we'd like to get to:

* Running an embedded ESM file.
* Running an archive of multiple files.
* Embedding [Node.js CLI options][] into the binary.
* [XCOFF][] executable format.
* Run tests on Linux architectures/distributions other than AMD64 Ubuntu.

## Disabling single executable application support

To disable single executable application support, build Node.js with the
`--disable-single-executable-application` configuration option.

## Implementation

When built with single executable application support, the Node.js process uses
[`postject-api.h`][] to check if the `NODE_JS_CODE` section exists in the
binary. If it is found, it passes the buffer to
[`single_executable_application.js`][], which executes the contents of the
embedded script.

[Next-10 discussions]: https://github.com/nodejs/next-10/blob/main/meetings/summit-nov-2021.md#single-executable-applications
[Node.js CLI options]: https://nodejs.org/api/cli.html
[XCOFF]: https://www.ibm.com/docs/en/aix/7.2?topic=formats-xcoff-object-file-format
[`postject-api.h`]: https://github.com/nodejs/node/blob/71951a0e86da9253d7c422fa2520ee9143e557fa/test/fixtures/postject-copy/node_modules/postject/dist/postject-api.h
[`single_executable_application.js`]: https://github.com/nodejs/node/blob/main/lib/internal/main/single_executable_application.js
[boxednode]: https://github.com/mongodb-js/boxednode
[pkg]: https://github.com/vercel/pkg
[postject]: https://github.com/nodejs/postject
[single executable applications]: https://github.com/nodejs/node/blob/main/doc/contributing/technical-priorities.md#single-executable-applications
[single-executable repository]: https://github.com/nodejs/single-executable
55 changes: 55 additions & 0 deletions lib/internal/main/single_executable_application.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
'use strict';
const {
prepareMainThreadExecution,
markBootstrapComplete,
} = require('internal/process/pre_execution');
const { getSingleExecutableCode } = internalBinding('sea');
const { emitExperimentalWarning } = require('internal/util');
const { Module, wrapSafe } = require('internal/modules/cjs/loader');
const { codes: { ERR_UNKNOWN_BUILTIN_MODULE } } = require('internal/errors');

prepareMainThreadExecution(false, true);
markBootstrapComplete();

emitExperimentalWarning('Single executable application');

// This is roughly the same as:
//
// const mod = new Module(filename);
// mod._compile(contents, filename);
//
// but the code has been duplicated because currently there is no way to set the
// value of require.main to module.
//
// TODO(RaisinTen): Find a way to deduplicate this.

const filename = process.execPath;
const contents = getSingleExecutableCode();
const compiledWrapper = wrapSafe(filename, contents);

const customModule = new Module(filename, null);
customModule.filename = filename;
customModule.paths = Module._nodeModulePaths(customModule.path);

const customExports = customModule.exports;

function customRequire(path) {
if (!Module.isBuiltin(path)) {
throw new ERR_UNKNOWN_BUILTIN_MODULE(path);
}

return require(path);
}

customRequire.main = customModule;

const customFilename = customModule.filename;

const customDirname = customModule.path;

compiledWrapper(
customExports,
customRequire,
customModule,
customFilename,
customDirname);
7 changes: 6 additions & 1 deletion node.gyp
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,8 @@

'include_dirs': [
'src',
'deps/v8/include'
'deps/v8/include',
'deps/postject'
],

'sources': [
Expand Down Expand Up @@ -458,6 +459,7 @@

'include_dirs': [
'src',
'deps/postject',
'<(SHARED_INTERMEDIATE_DIR)' # for node_natives.h
],
'dependencies': [
Expand Down Expand Up @@ -531,6 +533,7 @@
'src/node_report.cc',
'src/node_report_module.cc',
'src/node_report_utils.cc',
'src/node_sea.cc',
'src/node_serdes.cc',
'src/node_shadow_realm.cc',
'src/node_snapshotable.cc',
Expand Down Expand Up @@ -641,6 +644,7 @@
'src/node_report.h',
'src/node_revert.h',
'src/node_root_certs.h',
'src/node_sea.h',
'src/node_shadow_realm.h',
'src/node_snapshotable.h',
'src/node_snapshot_builder.h',
Expand Down Expand Up @@ -683,6 +687,7 @@
'src/util-inl.h',
# Dependency headers
'deps/v8/include/v8.h',
'deps/postject/postject-api.h'
# javascript files to make for an even more pleasant IDE experience
'<@(library_files)',
'<@(deps_files)',
Expand Down
18 changes: 18 additions & 0 deletions src/node.cc
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
#include "node_realm-inl.h"
#include "node_report.h"
#include "node_revert.h"
#include "node_sea.h"
#include "node_snapshot_builder.h"
#include "node_v8_platform-inl.h"
#include "node_version.h"
Expand Down Expand Up @@ -126,6 +127,7 @@
#include <cstring>

#include <string>
#include <tuple>
#include <vector>

namespace node {
Expand Down Expand Up @@ -321,6 +323,18 @@ MaybeLocal<Value> StartExecution(Environment* env, StartExecutionCallback cb) {
first_argv = env->argv()[1];
}

#ifndef DISABLE_SINGLE_EXECUTABLE_APPLICATION
if (sea::IsSingleExecutable()) {
// TODO(addaleax): Find a way to reuse:
//
// LoadEnvironment(Environment*, const char*)
//
// instead and not add yet another main entry point here because this
// already duplicates existing code.
return StartExecution(env, "internal/main/single_executable_application");
}
#endif

if (first_argv == "inspect") {
return StartExecution(env, "internal/main/inspect");
}
Expand Down Expand Up @@ -1187,6 +1201,10 @@ int LoadSnapshotDataAndRun(const SnapshotData** snapshot_data_ptr,
}

int Start(int argc, char** argv) {
#ifndef DISABLE_SINGLE_EXECUTABLE_APPLICATION
std::tie(argc, argv) = sea::FixupArgsForSEA(argc, argv);
#endif

CHECK_GT(argc, 0);

// Hack around with the argv pointer. Used for process.title = "blah".
Expand Down
Loading

0 comments on commit 8ea8354

Please sign in to comment.