New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot compile es6 module source #6303

Open
Mann90 opened this Issue Nov 23, 2017 · 59 comments

Comments

Projects
None yet
5 participants
@Mann90

Mann90 commented Nov 23, 2017

NWJS Version : v0.26.6
Operating System : macOS 10.13.1 (HighSierra)

Now we already can use es6 module features in v0.26.6 (Chrome 62). One example is:

`
-------------lib.js--------------

// lib.js
class TestModule {
foo() {
console.log('---foo----');
}
}

export {TestModule};

-----------testmodule.js----------------
// testmodule.js
import {TestModule} from "./lib.js";

var f = new TestModule();
f.foo();

-----------index.html----------------

<body>
    <script type="module" src="testmodule.js"></script>
    Hello.
</body>

----------package.json-----------
{
"name": "helloworld",
"main": "index.html",
"dependencies": {}
}
`

The above example can work properly. However, nwjc tool cannot compile them to binary code. The error message is:
`
$/Applications/nwjs/nwjc lib.js lib.bin

Failure compiling 'lib.js' (see above)
`

What's the correct procedure to use es6 module with nwjc? This is important for large scale application.

Thanks.

@Christywl

This comment has been minimized.

Contributor

Christywl commented Nov 24, 2017

I can reproduce this issue on Linux with nwjs-sdk-v0.26.6.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Nov 24, 2017

I just pushed a fix for nwjc to nw26 branch. Added a --nw-module command line switch which should be used when a module is being compiled.

We need a way to load compiled module binary into the application in the next step. Please propose.

@rogerwang rogerwang added the test-todo label Nov 24, 2017

@Mann90

This comment has been minimized.

Mann90 commented Nov 24, 2017

We need a way to load compiled module binary into the application in the next step. Please propose.

You mean another new API similar to evalNWBin? A quick-and-dirty way is make a evalNWBinModule?

@rogerwang

This comment has been minimized.

Member

rogerwang commented Nov 24, 2017

I'm not sure yet. testmodule.js wants to load lib.js, which is obviously not there when there is only binary.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Nov 24, 2017

@Mann90

This comment has been minimized.

Mann90 commented Dec 5, 2017

How long could it be taken to support module completely in V8 binary? Are there any technical difficulties? Thank you.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 5, 2017

@SMotaal

This comment has been minimized.

SMotaal commented Dec 21, 2017

Is there any progress on this?

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 21, 2017

@SMotaal as said in the last comment we are waiting for proposal from application developers. So if you want it to be implemented soon, please take some time to submit a proposal first.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 22, 2017

@rogerwang I was keeping a very close eye on the work the NodeJS team was doing to adopt V8's actual Modules using their own ModuleWrap (C++) objects together with their new internal/loader (JS) API. They made a very solid effort in keeping close to the standards to the point that they used file:// urls and are even working on porting Blobs and createObjectURL to make sure that they never expose the C++ implementation to the wild, ie no unchecked injection potential.

The bottomline is that it takes a hefty learning curve to understand how the new internal/loader works, especially when it was a moving target with --experimental-modules in versions 8 and 9 and not to mention the sometimes seemingly abrupt design choices which were driven by very lengthy and disjointed discussion threads.

IMHO I don't think that dealing with module loading should be an ad hoc effort if the goal is to get the community involved, it takes dedication and structure.

I am willing to be part of this effort, but let's not loose sight that like many application developers, I have a very limited understanding of the internals of NWJS, even when I spent weeks crawling over your various repositories and discussion threads, without knowing your workflows and with no clear context, it all seems like magic sometimes. I am only trying to rely an outsider's perspective and I am certain of your tremendous efforts behind the project and especially in involving the community in your efforts.

That said... The most important aspect behind Node's new loader is that we finally have specifications and native V8 implementations. In essence, you use a resolve(specifier, referrer) to map import paths to absolute urls and format (module/script) and an opt-in async dynamicInstantiate that can declare exports for a given url's namespace with getter callbacks for those declared names (called once during Module::instantiate).

So this is just a conversation starter, I am hoping we can keep this conversation going.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 22, 2017

I want to share how my thinking has evolved about modules with the evolution of native implementations (compared to before):

  1. Shall never down-transpile modules, should not even transpile (aside from esnext or non-es code with clear user opt-in). So all the existing function-wrapped module/bundling ecosystems are outdated in my books.

  2. Bundling should recreate [sic] modules (if only as blobs) with url mapping at the loader level without any string manipulation.

  3. Transpiler functions should not supersede the loader by overloading it, they can use it's hook methods and defer control to the loader. They should function deterministically and be agnostic to the environment (apart from contextual aspects for asset handling and resource optimization). They can chain into the resolve(specifier, referrer, defaultResolve) as "custom loaders" to perform the necessary functionality that would resolve a url and a format that is known to the loader. So they might need to verify a few aspects with the loader.

  4. Context-specific async injectors or factories for specific "custom formats" should only be allowed to act on completely resolved urls with the only purpose of linkage (eg: document.createElement('link') ... {href, type} …, or fetch.then(…)) ensuring proper exception handling to fulfill all upstream import(…).then() chains.

  5. This one was counter-intuitive at first, but a URL (including it's ?query) is only instantiated once, which is an acquired taste that grows on you when you realize how far this simple compromise can go in ensuring clear and predictable behaviour. So those getters I mentioned above are evaluated on instantiate and the returned values are locked in for the life span of the process / page.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 23, 2017

Thanks @SMotaal . Chromium engine had its module system implemented and it's working in NW. We could just reuse it for module binaries. It's just that a module would load its dependency by referring to it's JS filename (see testmodule.js importing lib.js in OP's example).
Regarding NW.js binary module, it might not be feasible to let developers change the source code to refer to the binary's filename (importing lib.bin instead of lib.js in testmodule.js), because that would require changing all the source code when the binary modules are to be built. I believe that would break developers' work flow.

I am thinking to extend the API Window.evalNWBin() which NW used to load compiled JS binary to something like :
evalNWBin(null, 'lib.bin', 'lib.js');

This creates a mapping between the paths of the module source and binary. After that, each time when Blink engine wants to load module lib.js, it will try loading lib.bin if the source code is missing.

I'm not sure if this would work the best way for JS application developers because my first language is C++. So please advise.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 23, 2017

So I just want to clarify first that you are only referring to modules imported from inside a <script type="module" … and not in node's context (with --experimental-modules flag)...

The reason I ask is that so far in all my tests, I only had success using import … inside <script type="module" … but node contexts throw the same error that you get if you run node without explicitly opting for --experimental-modules.

quick side note… I am yet to uncover a way to pass --experimental-modules to node contexts other than to force execArgs to child_process but this creates yet another degree of separation between the contexts and may only be ideal for worker-like functionality.

So I concluded from my tests that nwjs still did not address this huge transition that allows native ES modules in nodejs which I believe is planned for primetime (without flags) in the first public release of v10.0 (stable).

Is that a correct assumption?

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 24, 2017

<script type="module"> was shipped in Chrome 61 and JavaScript module import() was just shipped in Chrome 63. Both should work fine in latest NW 0.27, which is based on Chrome 63.

Chrome stable often has newer V8 version than Node.js latest stable so it has more features. In NW (thus the Node.js code in NW), we are always using Chrome's V8 version. So you should be able to use all those features by default without any flags.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 27, 2017

Thanks for the overview @rogerwang... Trust me, I've been keeping a very close eye on all the developments regarding native implementations for ES modules, especially in the V8 realm.

I apologize for the delay, but I used the last few days catching up on the internals of NW, rebuilding your latest nw28 mac builds and playing around with node's ESMLoader subsystem in separate and mixed contexts to be able to elaborate on the issues that I am trying to outline.

So now that I got the ESMLoader to work in node contexts, I must point out the following:

  1. Node's and Chrome's internal code base has and continues to be (for obvious historically necessities) script-based using non-standard JS modules (which are actually simply javascript code encapsulated in functions).

    The now "legacy" ways of dealing with fake JS modules in Chrome, Node, NW, Electron, everyone, are not interchangeable with ES modules because they all evaluate the code inside a function, and for all intents and purposes, import and export are top-level only.

    Instead of the idea of a "loader", ES modules require a "resolver" that maps every imported "specifier" to a URL that when accessed will pass all the security conditions necessary (like CORS and MIME) outside of the scope of the module subsystem. And the wammy, just for kicks, every unqiue URL get's evaluated only once in a given execution context and only after all it's imports have been resolved.

  2. Node and Chrome both use v8::Module (excuse my C) when a standard ES module is used but each use their own specifier-to-url mapping subsystem (which are "only partly conceptually" similar to what goes on in node-nw/lib/module) but are completely independent from their previously exisitng fake-JS-module subsystems.

    Chrome (for obvious reasons) sticks very close to the loader specs you mentioned above and since the formalization of the "custom loader" aspects did not land in the first round and the spec is now in upstream limbo, they did not implement (or at least don't expose) a way for us developers to be able to map "at the module level" things like import myapp from 'app:/…' to specific urls. It is possible however to intercept requests and play around with those (but that complicates the simplicity of ES module specifiers).

    Node on the other hand decided to hide this ugliness behind a subsystem of their own making and expose a new parallel but independent hook-based mechanism (like the one they used for "fake" Common JS modules) which for all it's challenges complies with at least one of the most fundamental aspect of ES modules, that a module is evaluated only once (in the process's sole context), and never evaluated until all it's dependencies are resolved. Their poorly named --loader is fundamentally a --resolver and/or --loader because the resolver aspect is far more important for native ES module consumption than the non-standard perks that you can bake into those custom loaders.

  3. Although there is no doubt that both Chrome's and Node's ES module related extension subsystems will coexist in NW, but they do not yet cooperate. For valid ES modules to cooperate, they must resolve equivalently across all contexts types, but load equivocally based on the context, so that importing node's process yields a wrapped module around a specific node-context bound process module exports when imported in a browser context or any other context, and this is where things start getting tricky.

IMHO, as a JavaScript developer using an application development platform that can use web technologies "outside of the constraints imposed by the web", Node's ESM subsystem provides the necessary mechanisms to use ES modules efficiently and effectively.

The same resolution mechanism should be followed for non-web-only uses:

When <script type="module" src="{{specifier}}"> or <script type="module"> import x from "{{specifier}}" or <script type="module"> import("{{specifier}}").then(…) or [console] > import("{{specifier}}").then(…) where document.baseURL (or the baseUrl of the ESMLoader of the node-context of the background-page) is the referrer, then URL[content-type='text/javascript'] is the result of calling the custom loader's resolve(specifier, referrer), which is hooked into the ESMLoader as a custom resolve hook.

When import x from "{{specifier}}" or import("{{specifier}}").then(…) statements where import.meta.url is the referrer (ie inside a module), then URL[content-type='text/javascript'] is the result of calling the custom loader's `resolve(specifier, referrer)… etc.

Node's implementation uses special protocols for resolved URL, like "node:process" which resolves to the builtin "dynamically instantiated" node module that exposes as "constant snapshot" of the CommonJS exports yielded by require('process') and they are naturally "context-bound". While context's may be a little tricky to figure out, still NW can also use bin:{{filename[.bin]}} as a resolved URL since it can check before hand if it should use the bin version or the file://…{{filename[.js|.mjs|.m.js]}}.

So when I import process from "process" node's default resolver calls my custom resolver and then it is "process", so node's default resolver now maps that to the special protocol "node:process" and handles that for me. The same can be accomplished for relative paths and bin-files... etc.

Chrome's implementation requires [content-type] for ES Modules and it does not allow except the "web standards" protocols which makes sense for web content but not for NW applications, so I gather there needs to be some work there.

https://github.com/nwjs/chromium.src/blob/45be0261c5a04cc38dead115740a8a6f7e6958a7/third_party/WebKit/Source/core/loader/modulescript/DocumentModuleScriptFetcher.cpp#L39-L51

Now here is the punchline:

If we are still talking real ES modules, not fake ones or "rollups" as bundles, then this is not a simple thing to do, because evalESMFromBin() (if it does not hook into "legacy" eval() or function-wrapped module loading mechanisms) must be a special variant of the native implementation of the import().then() which would revive a snapshot of those previously resolved modules modules and all their dependencies as if they had just been resolved from disk, and do so while also checking if one of those resolutions should be substituted when the file is found at the respective resolved path (for optional overloading).

FYI: I did not even bother looking into Electron since it really caters to Atom and VSCode and both already locked themselves with huge fake-JS module code bases and their own loaders, so they are in their own happy land.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 27, 2017

In NW applications the files are with chrome-extension:// protocol, where the URL response has an expected content-type so Chrome's implementation should work well.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 27, 2017

btw, are you saying that Chrome doesn't support "real ES modules"? I think it supports well and my idea of extending evalNWBin is going to reuse/hook into its implementation.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 27, 2017

I do apologize for the very long post and multiple edits.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 27, 2017

btw, are you saying that Chrome doesn't support "real ES modules"? I think it supports well and my idea of extending evalNWBin is going to reuse/hook into its implementation.

Chrome uses real ES modules, without doubt, however, all their existing Javascript code (ie if you look in their own extension:// files in the inspector) still uses "legacy" function-wrapped modules because ES modules require a lot of rewiring and it is not a simple switch.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 27, 2017

I see, but that's only for their UI of the devtools or other web-ui for settings, etc. It has nothing to do with the code from the web page or NW application.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 27, 2017

In NW applications the files are with chrome-extension:// protocol, where the URL response has an expected content-type so Chrome's implementation should work well.

Yes, but here is a tricky thing, if we don't use a custom resolver on the node end then node will only load ESM modules from files with the .mjs (tested with a modified lib/bootstrap_node.js that enabled ESMLoader if process.__nw by default since there is no way to configure NW to pass --experimental-module which node looks for in process.binding('config')).

So when I managed to load ./module.mjs in node, and tried to load chrome-extension:…/module.mjs, Chrome complained that the URL is not a valid content-type as per spec, which I have no means of dealing with as a javascript developer (it's hard coded in the lines mentioned above).

@SMotaal

This comment has been minimized.

SMotaal commented Dec 27, 2017

I see, but that's only for their UI of the devtools or other web-ui for settings, etc. It has nothing to do with the code from the web page or NW application.

True, but I figure, they did not really deal with non-web use cases themselves, so they did not do much of the work that the Node folks had to do (and here I was with the rest of the community wondering why they were so slow).

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 27, 2017

So when I managed to load ./module.mjs in node, and tried to load chrome-extension:…/module.mjs, Chrome complained that the URL is not a valid content-type as per spec, which I have no means of dealing with as a javascript developer (it's hard coded in the lines mentioned above).

Are you hitting this bug? https://bugs.chromium.org/p/chromium/issues/detail?id=797712 Otherwise it should work.

@SMotaal

This comment has been minimized.

SMotaal commented Dec 27, 2017

The nice thing is that v8::Module is the lowest common denominator. However, I should point out that when node tried to load a module for a URL that was already loaded in chrome (hence the v8::Module already existed for this url) I think it might have caused because of ESMLoader subsystem works with certain expectations of internalized side-effects that were not wired correctly for that instance of the v8::Module.

@rogerwang

This comment has been minimized.

Member

rogerwang commented Dec 27, 2017

Yes, but here is a tricky thing, if we don't use a custom resolver on the node end then node will only load ESM modules from files with the .mjs

Finally I see. You want to make Node's module support and Chrome's working together ... I think it should be a separate issue. What OP is requesting is about supporting this NW feature with the module feature of Chrome.

@rogerwang rogerwang removed the test-todo label Jan 4, 2018

rogerwang added a commit that referenced this issue Jan 4, 2018

@rogerwang

This comment has been minimized.

Member

rogerwang commented Jan 4, 2018

This is fixed in nw28 branch and will be available in the next nightly build.

@rogerwang rogerwang closed this Jan 4, 2018

@rogerwang

This comment has been minimized.

Member

rogerwang commented Jan 4, 2018

This is done for now by adding evalNWBinModule() as proposed. Please see the docs change for usage. The API is subject to change in future versions.

@rogerwang

This comment has been minimized.

@Mann90

This comment has been minimized.

Mann90 commented Jan 26, 2018

Awesome!!

But there are two issues found:

  1. If import * from './lib.js' is not there, the nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js'); doesn't load and run the script.

For example, if there is a top-level statement console.log('lib loaded.'); in the lib.bin (lib.js), then that statement doesn't run actually.

  1. Why does nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js'); take the third argument? Could we simplify it to nw.Window.get().evalNWBinModule(null, 'lib.bin'');. (And we assume that foo.bin must be generated from foo.js.)

Or even further, we could unify evalNWBinModule and evalNWBin, let NW to check if the binary is a compiled by --nw-module flag or not. Is it possible?

Another option is to add a new boolean argument to evalNWBin to indicate if the lib.bin is a module or not.

@ghost

This comment has been minimized.

ghost commented Jan 30, 2018

If I understand correctly the documentation, forgive me to not have yet tested the correctness of my understanding, when using compiled modules, we have to explicitely preload all the binaries via the new api before importing them into the application code.

So we have to programmatically create a map of all needed modules, then import all of them via evalNWBinModule before even starting the application code:

<script>
/* 
  index.html 
*/
/* here we start with preloading all the binary modules 
for them to be available to import in code*/
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');
nw.Window.get().evalNWBinModule(null, 'dep.bin', 'dep.js');

import('./lib.js')
</script>
/* lib.js */
import('./dep.js');

For large applications with lots of modules, it may be inconvenient to have to preload all the binaries beforehand.

Ideally, we would just have to require or import a .js or rather a .esm file (like nodejs modules extension) into the application code, then nwjs would check if a binary exists for that file on the same path but with the .bin extension, or on a different path to resolve and with a different extension set as parameters into global preferences. If the binary exists on disk then it would be loaded under the hood through evalNWBinModule and automatically executed.

<script>
/* 
  index.html 
*/
// here we start with defining global preferences for loading binary modules
nw.binary.check = true; // whether nwjs will check for the existence of a binary file on disk
nw.binary.path = "__dirname\\bin"; // default base path to resolve for binary modules
nw.binary.extension = "bin"; // default binary extension to check on disk
nw.module.extension = "esm" // default extension for javascript modules
/* under the hood nwjs checks for binary file on disk, if present, it
loads and evaluates it, or default to import non binary js file */
import('./lib.esm'); 
</script>
/* 
lib.esm 
*/
import('./dep.esm');
@ghost

This comment has been minimized.

ghost commented Jan 30, 2018

After testing the function with nwjc, it seems that nested imports are not supported... as nwjc --nw-module lib.js lib.bin rises an error when compiling a script containing a dynamic import:

console.log("lib.js started");

import('./dep.js');

That makes it impossible to use in the perspective of an application made of many modules requiring each others and which would have been compiled into binaries.

@ghost

This comment has been minimized.

ghost commented Jan 30, 2018

This makes me think that it would be nice to have a loader module in NW.js which would control the way files are loaded via a require or an import call.

This could be a generalization of the binary loader for javascript files, based on extensions.

When we require or import a .js or an .esm file, nwjs would search for a binary file like described in my previous comment according to different parameters defined within the API, for example via functions.

But we could potentially use that same API to create any kind of loaders in nwjs. This could be for example a loader for webassembly files as well, or a loader for files to be transpiled at runtime by exposing hooks into the require and import mechanisms available in node and chromium.

Loader and transform functions would be defined within the API like evalNWBinModule or evalWasmModule which would be called when a require or import call is made with particular file extensions.

nw.loader['js'] = {
   onFetch:findPath, // function to call in order to overwrite the location of the required/importee, returns the path
   onLoad:loadBinaryFile, // function to call with path of importee as argument, returns the source
   onExecute:nw.Window.get().evalNWBinModule //function to call with source of importee as argument
};

nw.loader['json'] = {
   onExecute(src){
      return JSON.parse(src);
  }
};
@SMotaal

This comment has been minimized.

SMotaal commented Jan 30, 2018

I've already been working on a parallel module loading system based on node's new loader system which hooks into v8's dynamic import and import meta callbacks effectively creating a parallel module system that can be customized indefinitely. While I don't have any working knowledge of evalNW* functions, I am almost certain that it would be possible to support. (FYI: this only applies to scopes where import is supported, i.e. does not work for things like service workers and cannot be used to alter the behaviour of calls to global importScript functions)

My efforts are completely solo at the moment though, if anyone is interested in collaborating to refactor this into an open source project I am very interested. I have managed to limit the required patching to only require building libnode.dylib and made sure that the amount of C++ changes made are limited to a handful SLOC's aside from pulling one or two files from pull requests from the node repo (expected to land in v10).

@rogerwang

This comment has been minimized.

Member

rogerwang commented Jan 31, 2018

reopen to track supporting nested imports.

@rogerwang rogerwang reopened this Jan 31, 2018

@ghost

This comment has been minimized.

ghost commented Jan 31, 2018

For now, I may not use it correctly, but I'm not succeeding to make evalNWBinModule work, even when there is no nested imports, whether it is with dynamic or static import

<!-- index.html -->
<script>
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');

import('./lib.js')
</script>

or

<!-- index.html -->
<script>
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');
</script>
<script type='module'>
import './lib.js'
</script>
/* test/lib.js -> compiled to lib.bin */
console.log('lib.js started as binary...');
@ghost

This comment has been minimized.

ghost commented Jan 31, 2018

@ smotaal, can't help on C++ personally, but that sounds very neat: the ability to hook into v8's dynamic import callback for loading any kind of customized resources into javascript (including binary snapshots) would be very useful in a project like NW.js.

But how would that relate to nested imports of compiled modules ? My knowledge about snapshots is limited, but as far as I understand them, we can not have a 'require' or 'import' into the top-level scope of the module because the files are not loaded into the heap when nwjc creates the snapshot for individual files.

And nwjc seems to complain about dynamic imports even when they are not in top-level scope (within a function definition for example).

So, the only way to support dynamic nested imports presently seems to have a function into the global scope which would serve as a wrapper for dynamic imports, and within the module the wrapper call would have to not be on the top-level scope:

<!-- index.html -->
<script>
nw.getModule = (path) => {import(path)}
</script>
<script type='module'>
import {useDep} from './lib.js'
</script>
/* test/lib.js -> compiled to lib.bin */
export let useDep = () =>{ nw.getModule('./dep.js')}
@SMotaal

This comment has been minimized.

SMotaal commented Jan 31, 2018

I just want to point out a difference between static and dynamic imports in V8's implementation:

The following is static:

<script type=module>
import './lib.js'  // expected to load .bin (no .js)
</script>
// lib.bin
import {a} from 'lib-a.js' // also expected to load .bin (no .js)
// lib-a.bin
import {b} from 'lib-b.js' // this one expected to load .js

However, you might not get the same if (the following is dynamic):

// lib.bin
import('lib-a.js').then(({a}) => {})

Here is where V8's isolate->SetHostImportModuleDynamicallyCallback plays part:

https://v8docs.nodesource.com/node-8.9/d5/dda/classv8_1_1_isolate.html#a2b1e8fe982f0b2a8c61d9413cf94c87c

And this is where the discrepancy might be coming from.

My approach basically gives node's loader full control from the get-go, it takes charge of all static imports (I even wired it to <script module>…</script> tags) and that loader does all nested static imports.

At the same time, I make it register itself as the isolate's DynamicImportCallback. So effectively it hijacks it and processes it using the same logic, ie able to remap it with the same single logic ensuring consistency.

One issue with node's loader is that it was designed for a single "main" context, so it essentially crashed when it tried to access a module across nw frames and that required either redesigning it (no thanks) or finding ways to limit the scope of module-mapping to the contexts for which they are intended (which is 99% javascript code at the moment).

@ab38, like you, I never found my footing in C++ (it's like advanced calculus, which is cool, but no thanks, not my thing) and I basically geek out on finding the path of least resistance and most hackability (ie performance-oriented javascript).

@SMotaal

This comment has been minimized.

SMotaal commented Jan 31, 2018

So it comes down to this:

The previous mind-set that worked for require-oriented modules is not suited for es modules, and though it may be possible to patch the system enough to get it function for certain scenarios, it is like trying to fit the squares through the round holes, they only need to be big enough at the corners to pass (if all you care about is to make them pass). Instead, I believe that the benefits of es modules are far to important to be ignored when they are not handled correctly, not through square holes.

Node's loader provides that, and the best thing is that it is already wired for "require" loading as a bonus.

I have not yet looked into how to add nw's own features into node's loader, my focus was to make it function as a loader that can work in chromium, and it does. It can import('./lib.m.js').then(…), import process from 'process', it can even import x from 'package/src/index.ts' and compile that on demand. It is slowly but carefully adding support for things like import.meta - it works as advertised in upstream PRs - tested. It already has support for node native modules, json, and if you have a require statement, it handles it using the existing module wiring, ie honours any custom wiring like nw.js.

@ghost

This comment has been minimized.

ghost commented Jan 31, 2018

What I'm wondering, actually, is the following: is it possible to make nested imports (either static either dynamic modules) to work with binary compiled modules via nwjc ?

So that a whole NW.js application made up of separate es modules can be compiled into individual binary module files which would work together when the main module is loaded into NW.js.

Ideally, as a developper, and as stated by @rogerwang, there should not be any change to make into the application code in order to support it.

So, one's best guess would be that NW.js somehow would hook into the import internal mechanism of either node or chromium. And you seem to suggest, @SMotaal , that the new node's loader would be the more pertinent to use internally by NW.js to achieve that goal.

Now, how would that connect to binary modules requiring or importing each other ? Can nwjc compile es modules with top-level imports or dynamic imports into binaries which would then be imported at runtime by the application, each binary module being also able to import the others like non compiled es modules?

My point is that perhaps to protect the javascript source code in the context of separate modules is not possible with nwjc snapshots, but I may be wrong, not knowing the internals of how a snapshot is made. But to use a customized node's loader as you pointed out, @SMotaal, could also allow to use alternative encryption tools to protect the source code without using the nwjc snapshots (if they do not work in the context of modules).

@ghost

This comment has been minimized.

ghost commented Jan 31, 2018

Again, as just an end-user of NW.js developing js applications, what would be ideal is that NW.js itself loads the compiled binary when an import call (dynamic or static) is made into the application code (if that is at all possible in the context of modules).

nw.loader['js'].enable();
import('./main.js'); // nwjs takes care of loading the binary if present and all its dependencies.

And as a generalized utility, this could be also very useful to have an api to also define custom loaders depending on files extension.

In definitive the 'js' extension would be associated with a predefined loader used by NW.js in charge of loading the individual binaries and connecting them together at runtime. There could be a simple commandline switch activating the loader associating the import call of js files with modules precompiled with nwjc.

@SMotaal

This comment has been minimized.

SMotaal commented Jan 31, 2018

From my perspective, a wholistic loader system is essential to developer platforms like nw. In my own opinion, browser implementations are mostly preferred, except when it comes to module loading that is adequate for a desktop application due to security restrictions intended for the web, so in this case, node's loader is the way to go.

The problem is that hooks for the dynamic import and import meta callbacks are not context-scoped, they are isolate-scoped, so you cannot pick and choose, if nw decides to move forward by depending on them for one purpose (i.e. to catch evalNW's) then anyone needing to hook to them for a different purpose will either override nw's hooks or if nw is aggressive, may end up with a mess where some hooks work and others don't depending on their luck.

As a developer, I want to decide on the loader that would be hooked. I want to bring my own — did I mention I have an awesome one already working for me :) — my only problem is that I am not equipped (or honestly maybe due to my limited understanding of snapshots not confident enough to be motivated) to explore adding support for snapshots. If someone knows enough and wants to collaborate, and maybe in the process can helping me understand it better from their perspective, it might turn out to be easier to solve than I imagine.

But, please, don't restrict such powerful new additions if you end up using them internally without providing means for developers to properly utilize them if they so choose.

@SMotaal

This comment has been minimized.

SMotaal commented Jan 31, 2018

@ab38 You can most certainly add an encryption layer, just keep in mind that unlike snapshots which are consumed in a post-source state (in theory they are never reversed to source) encrypted modules must be decrypted by the loader, so they are simply hard to hack but not unhackable.

@ghost

This comment has been minimized.

ghost commented Jan 31, 2018

@SMotaal , thanks for the interesting comments.

Another way to look at javascript source code protection could be to use a custom encryption layer associated with a custom loader, both residing into a single snapshot and chosen by the developer, and hooked via a nwjs api to the import call within the application code. The nwjs api would then create at runtime and in-memory individual file-based snapshots or one multi-files snapshot which would then be injected into the v8's runtime engine, and modified at runtime, all from within a single snapshot.

So the snapshot (the nwjs api part of it) would just be in charge to transform decrypted javascript code (coming from disk or elsewhere depending on the developer's logic) into a runtime in-memory snapshot.

Schematically, the single snapshot would take care of the following process:

-> connect to import hook via a nwjs api
-> load encrypted js via custom logic
-> decrypt js via custom logic
-> nwjs api creates a runtime snapshot for the file
-> nwjs api injects it into v8 at runtime

In other words, one snapshot to rule them all... :)

@ghost

This comment has been minimized.

ghost commented Feb 1, 2018

Proposal for javascript source code protection compatible with ecmascript modules in NW.js

Prerequisites

  • The Source Code Protection (SCP) logic should be compatible with ecmascript modules.
  • It should not interfer with the application code.
  • It should be optional and customizable
  • It should take advantage of V8's snapshot mechanism
  • It should be backward compatible with older NW.js versions

Rationale

  • In order to maximize source code protection (SCP), developers should be able to use their own encryption logic, because if snapshots can be reversed one day, all nwjs applications would be vulnerable at the same time.
  • In order to be compatible with ecmascript modules without interfering with the application code, the SCP logic would hook into the import calls of the V8 engine (both static and dynamic).
  • By hooking into the import call, it could be customizable for different types of files depending on their extensions.
  • The underlying mechanism would offer a loader utility mechanism to developers, which would serve beyond the mere SCP logic.

Description

In a startup snapshot, optional custom developer logic is created for dealing with the import calls of particular file extensions. The developer logic is in charge to optionally resolve, fetch, load and decrypt js source code. Then the js source code is transformed into a snapshot via the nwjs api, and executed into the V8 engine.

API

/* 
	js content of startup.bin (nwjc dev/startup.js dist/startup.bin) 
*/
function init(){
  nw.enableLoader( 'js', {
  	onResolve(importee, importer){
                /*
  		  optional custom developer logic modifying the path (importee) of the import call
  		      importee: path of the import call, ie: import('importee')
  		      importer: path of the module importing the importee
  		      return importee by default
               */
  	},
  	onFetch(importee){
                /*
  		   optional custom developer logic for fetching the source code
  		       return source code or encrypted source code
                */
  	},
  	onLoad(source){
                 /*
  		    optional custom developer logic after fetching the source code
  		    custom decryption logic goes here (local or remote through authorization,
                    code-signing verification (including the startup.bin), etc.)
  		        return source code or decrypted source code
                 */
  	},
  	async onExecute(source){
  		 /* 
                    optional custom developer logic for executing the source code
  		    snapshot protection logic goes here via nwjs api 
                 */
  		  let snapshot = await nw.createSnapshot(source, true); //nw.createSnapshot(source, asModule)
  		  nw.evalNWBinModule(null, snapshot);
               /* or 
                  let snapshot = await nw.createSnapshot(source);
                  nw.evalNWBin(null, snapshot, true) //nw.evalNWBin(frame, snapshot, asModule)
               */
  	}
  }); // enable 'js' loader to hook into import calls
}
init();
@ghost

This comment has been minimized.

ghost commented Feb 1, 2018

The issue that I pointed out in #6303 (comment) is caused by reloading the page with "Ctrl + R" which I use to do to test my application code, instead of restarting the nwjs process.

When one reloads the page, evalNWBinModule doesn't work anymore in my tests, one has to restart the nwjs process altogether. Otherwise it seems to work fine as long as there is no nested import into the binary module.

@Mann90

This comment has been minimized.

Mann90 commented Feb 1, 2018

My two cents: It'd be better if we can simplify the APIs to be consistent, such as:

  • Merge evalNWBin and evalNWModule. Let evalNWBin take an optional boolean argument "asModule"
  • Remove --nw-module argument from "nwjc". Let "nwjc" just compile "foo.js" to "foo.bin", and then let evalNWBin to decide load it as a module or not.

Thus, it could be more consistent to the HTML semantics: One js file could be loaded as a module or not.

  • <script>: To load a js file.
  • <script type="module">: To load a js file as a module.
@alxproject

This comment has been minimized.

alxproject commented Sep 7, 2018

using latest nightly 0.33.1-sdk show error with dynamic import

$ ~/nwjs-sdk-v0.33.1-linux-x64/nwjc --nw-module libimp.js libimp.bin
Unhandle exception: Uncaught SyntaxError: Unexpected token import @  return import(lib);[2]
Failure compiling 'libimp.js' (see above)

libimp.js

export function importar(lib) {
	return import(lib);
}

compiling any other library that not contains dynamic import works perfectly

@rogerwang

This comment has been minimized.

Member

rogerwang commented Sep 7, 2018

@alxproject these flags of nwjc seems working for me: --harmony-shipping. Please try it on your side and I'll make it default.

@alxproject

This comment has been minimized.

alxproject commented Sep 8, 2018

@rogerwang Hello, thanks for your answer
I try this:
nwjs 0.32.4-sdk
compiling with --nw-module works correctly if the library not contains dynamic imports
compiling with --nw-module and --harmony-shipping works correctly if the library not contains dynamic imports
compiling with --nw-module failed to compile library whit dynamic import
compiling with --nw-module and --harmony-shipping works correctly with dynamic import

nwjs 0.33.0-sdk
compiling with --nw-module works correctly but the binary file generatet not work when is imported
compiling with --nw-module and --harmony-shipping works correctly but the binary file generatet not work when is imported
compiling with --nw-module failed to compile library whit dynamic import
compiling with --nw-module and --harmony-shipping works correctly with dynamic import but the binary file generatet not work when is imported

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment