Cannot compile es6 module source #6303
Comments
|
I can reproduce this issue on Linux with nwjs-sdk-v0.26.6. |
|
I just pushed a fix for We need a way to load compiled module binary into the application in the next step. Please propose. |
You mean another new API similar to evalNWBin? A quick-and-dirty way is make a evalNWBinModule? |
|
I'm not sure yet. |
|
Any proposal should take this into consideration: https://html.spec.whatwg.org/multipage/webappapis.html#integration-with-the-javascript-module-system |
|
How long could it be taken to support module completely in V8 binary? Are there any technical difficulties? Thank you. |
|
It should not be difficult to implement. But we're waiting for API
proposal, which would work well with work flow on the application
developer's side.
…On Dec 5, 2017 10:47 PM, "Mann90" ***@***.***> wrote:
How long could it be taken to support module completely in V8 binary? Are
there any technical difficulties? Thank you.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#6303 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAKGGU5MzizJxTjudRBrj8ag66I7nihNks5s9VdpgaJpZM4Qo-3j>
.
|
|
Is there any progress on this? |
|
@SMotaal as said in the last comment we are waiting for proposal from application developers. So if you want it to be implemented soon, please take some time to submit a proposal first. |
|
@rogerwang I was keeping a very close eye on the work the NodeJS team was doing to adopt V8's actual Modules using their own ModuleWrap (C++) objects together with their new internal/loader (JS) API. They made a very solid effort in keeping close to the standards to the point that they used The bottomline is that it takes a hefty learning curve to understand how the new internal/loader works, especially when it was a moving target with IMHO I don't think that dealing with module loading should be an ad hoc effort if the goal is to get the community involved, it takes dedication and structure. I am willing to be part of this effort, but let's not loose sight that like many application developers, I have a very limited understanding of the internals of NWJS, even when I spent weeks crawling over your various repositories and discussion threads, without knowing your workflows and with no clear context, it all seems like magic sometimes. I am only trying to rely an outsider's perspective and I am certain of your tremendous efforts behind the project and especially in involving the community in your efforts. That said... The most important aspect behind Node's new loader is that we finally have specifications and native V8 implementations. In essence, you use a So this is just a conversation starter, I am hoping we can keep this conversation going. |
|
I want to share how my thinking has evolved about modules with the evolution of native implementations (compared to before):
|
|
Thanks @SMotaal . Chromium engine had its module system implemented and it's working in NW. We could just reuse it for module binaries. It's just that a module would load its dependency by referring to it's JS filename (see I am thinking to extend the API This creates a mapping between the paths of the module source and binary. After that, each time when Blink engine wants to load module I'm not sure if this would work the best way for JS application developers because my first language is C++. So please advise. |
|
So I just want to clarify first that you are only referring to modules imported from inside a The reason I ask is that so far in all my tests, I only had success using
So I concluded from my tests that nwjs still did not address this huge transition that allows native ES modules in nodejs which I believe is planned for primetime (without flags) in the first public release of v10.0 (stable). Is that a correct assumption? |
|
Chrome stable often has newer V8 version than Node.js latest stable so it has more features. In NW (thus the Node.js code in NW), we are always using Chrome's V8 version. So you should be able to use all those features by default without any flags. |
|
Thanks for the overview @rogerwang... Trust me, I've been keeping a very close eye on all the developments regarding native implementations for ES modules, especially in the V8 realm. I apologize for the delay, but I used the last few days catching up on the internals of NW, rebuilding your latest nw28 mac builds and playing around with node's ESMLoader subsystem in separate and mixed contexts to be able to elaborate on the issues that I am trying to outline. So now that I got the ESMLoader to work in node contexts, I must point out the following:
IMHO, as a JavaScript developer using an application development platform that can use web technologies "outside of the constraints imposed by the web", Node's ESM subsystem provides the necessary mechanisms to use ES modules efficiently and effectively. The same resolution mechanism should be followed for non-web-only uses: When When Node's implementation uses special protocols for resolved URL, like "node:process" which resolves to the builtin "dynamically instantiated" node module that exposes as "constant snapshot" of the CommonJS exports yielded by So when I Chrome's implementation requires [content-type] for ES Modules and it does not allow except the "web standards" protocols which makes sense for web content but not for NW applications, so I gather there needs to be some work there. Now here is the punchline: If we are still talking real ES modules, not fake ones or "rollups" as bundles, then this is not a simple thing to do, because evalESMFromBin() (if it does not hook into "legacy" eval() or function-wrapped module loading mechanisms) must be a special variant of the native implementation of the import().then() which would revive a snapshot of those previously resolved modules modules and all their dependencies as if they had just been resolved from disk, and do so while also checking if one of those resolutions should be substituted when the file is found at the respective resolved path (for optional overloading). FYI: I did not even bother looking into Electron since it really caters to Atom and VSCode and both already locked themselves with huge fake-JS module code bases and their own loaders, so they are in their own happy land. |
|
In NW applications the files are with |
|
btw, are you saying that Chrome doesn't support "real ES modules"? I think it supports well and my idea of extending |
|
I do apologize for the very long post and multiple edits. |
Chrome uses real ES modules, without doubt, however, all their existing Javascript code (ie if you look in their own |
|
I see, but that's only for their UI of the devtools or other web-ui for settings, etc. It has nothing to do with the code from the web page or NW application. |
Yes, but here is a tricky thing, if we don't use a custom resolver on the node end then node will only load ESM modules from files with the So when I managed to load |
True, but I figure, they did not really deal with non-web use cases themselves, so they did not do much of the work that the Node folks had to do (and here I was with the rest of the community wondering why they were so slow). |
Are you hitting this bug? https://bugs.chromium.org/p/chromium/issues/detail?id=797712 Otherwise it should work. |
|
The nice thing is that v8::Module is the lowest common denominator. However, I should point out that when node tried to load a module for a URL that was already loaded in chrome (hence the v8::Module already existed for this url) I think it might have caused because of ESMLoader subsystem works with certain expectations of internalized side-effects that were not wired correctly for that instance of the v8::Module. |
Finally I see. You want to make Node's module support and Chrome's working together ... I think it should be a separate issue. What OP is requesting is about supporting this NW feature with the module feature of Chrome. |
|
If I understand correctly the documentation, forgive me to not have yet tested the correctness of my understanding, when using compiled modules, we have to explicitely preload all the binaries via the new api before importing them into the application code. So we have to programmatically create a map of all needed modules, then import all of them via <script>
/*
index.html
*/
/* here we start with preloading all the binary modules
for them to be available to import in code*/
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');
nw.Window.get().evalNWBinModule(null, 'dep.bin', 'dep.js');
import('./lib.js')
</script>/* lib.js */
import('./dep.js');For large applications with lots of modules, it may be inconvenient to have to preload all the binaries beforehand. Ideally, we would just have to require or import a <script>
/*
index.html
*/
// here we start with defining global preferences for loading binary modules
nw.binary.check = true; // whether nwjs will check for the existence of a binary file on disk
nw.binary.path = "__dirname\\bin"; // default base path to resolve for binary modules
nw.binary.extension = "bin"; // default binary extension to check on disk
nw.module.extension = "esm" // default extension for javascript modules
/* under the hood nwjs checks for binary file on disk, if present, it
loads and evaluates it, or default to import non binary js file */
import('./lib.esm');
</script>/*
lib.esm
*/
import('./dep.esm'); |
|
After testing the function with nwjc, it seems that nested imports are not supported... as console.log("lib.js started");
import('./dep.js');That makes it impossible to use in the perspective of an application made of many modules requiring each others and which would have been compiled into binaries. |
|
This makes me think that it would be nice to have a loader module in NW.js which would control the way files are loaded via a This could be a generalization of the binary loader for javascript files, based on extensions. When we require or import a But we could potentially use that same API to create any kind of loaders in nwjs. This could be for example a loader for webassembly files as well, or a loader for files to be transpiled at runtime by exposing hooks into the require and import mechanisms available in node and chromium. Loader and transform functions would be defined within the API like nw.loader['js'] = {
onFetch:findPath, // function to call in order to overwrite the location of the required/importee, returns the path
onLoad:loadBinaryFile, // function to call with path of importee as argument, returns the source
onExecute:nw.Window.get().evalNWBinModule //function to call with source of importee as argument
};
nw.loader['json'] = {
onExecute(src){
return JSON.parse(src);
}
}; |
|
I've already been working on a parallel module loading system based on node's new loader system which hooks into v8's dynamic import and import meta callbacks effectively creating a parallel module system that can be customized indefinitely. While I don't have any working knowledge of evalNW* functions, I am almost certain that it would be possible to support. (FYI: this only applies to scopes where import is supported, i.e. does not work for things like service workers and cannot be used to alter the behaviour of calls to global importScript functions) My efforts are completely solo at the moment though, if anyone is interested in collaborating to refactor this into an open source project I am very interested. I have managed to limit the required patching to only require building libnode.dylib and made sure that the amount of C++ changes made are limited to a handful SLOC's aside from pulling one or two files from pull requests from the node repo (expected to land in v10). |
|
reopen to track supporting nested imports. |
|
For now, I may not use it correctly, but I'm not succeeding to make <!-- index.html -->
<script>
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');
import('./lib.js')
</script>or <!-- index.html -->
<script>
nw.Window.get().evalNWBinModule(null, 'lib.bin', 'lib.js');
</script>
<script type='module'>
import './lib.js'
</script>/* test/lib.js -> compiled to lib.bin */
console.log('lib.js started as binary...'); |
|
@ smotaal, can't help on C++ personally, but that sounds very neat: the ability to hook into v8's dynamic import callback for loading any kind of customized resources into javascript (including binary snapshots) would be very useful in a project like NW.js. But how would that relate to nested imports of compiled modules ? My knowledge about snapshots is limited, but as far as I understand them, we can not have a 'require' or 'import' into the top-level scope of the module because the files are not loaded into the heap when nwjc creates the snapshot for individual files. And nwjc seems to complain about dynamic imports even when they are not in top-level scope (within a function definition for example). So, the only way to support dynamic nested imports presently seems to have a function into the global scope which would serve as a wrapper for dynamic imports, and within the module the wrapper call would have to not be on the top-level scope: <!-- index.html -->
<script>
nw.getModule = (path) => {import(path)}
</script>
<script type='module'>
import {useDep} from './lib.js'
</script>/* test/lib.js -> compiled to lib.bin */
export let useDep = () =>{ nw.getModule('./dep.js')} |
|
I just want to point out a difference between static and dynamic imports in V8's implementation: The following is static: <script type=module>
import './lib.js' // expected to load .bin (no .js)
</script>// lib.bin
import {a} from 'lib-a.js' // also expected to load .bin (no .js)// lib-a.bin
import {b} from 'lib-b.js' // this one expected to load .jsHowever, you might not get the same if (the following is dynamic): // lib.bin
import('lib-a.js').then(({a}) => {})Here is where V8's isolate->SetHostImportModuleDynamicallyCallback plays part: And this is where the discrepancy might be coming from. My approach basically gives node's loader full control from the get-go, it takes charge of all static imports (I even wired it to <script module>…</script> tags) and that loader does all nested static imports. At the same time, I make it register itself as the isolate's DynamicImportCallback. So effectively it hijacks it and processes it using the same logic, ie able to remap it with the same single logic ensuring consistency. One issue with node's loader is that it was designed for a single "main" context, so it essentially crashed when it tried to access a module across nw frames and that required either redesigning it (no thanks) or finding ways to limit the scope of module-mapping to the contexts for which they are intended (which is 99% javascript code at the moment). @ab38, like you, I never found my footing in C++ (it's like advanced calculus, which is cool, but no thanks, not my thing) and I basically geek out on finding the path of least resistance and most hackability (ie performance-oriented javascript). |
|
So it comes down to this: The previous mind-set that worked for require-oriented modules is not suited for es modules, and though it may be possible to patch the system enough to get it function for certain scenarios, it is like trying to fit the squares through the round holes, they only need to be big enough at the corners to pass (if all you care about is to make them pass). Instead, I believe that the benefits of es modules are far to important to be ignored when they are not handled correctly, not through square holes. Node's loader provides that, and the best thing is that it is already wired for "require" loading as a bonus. I have not yet looked into how to add nw's own features into node's loader, my focus was to make it function as a loader that can work in chromium, and it does. It can |
|
What I'm wondering, actually, is the following: is it possible to make nested imports (either static either dynamic modules) to work with binary compiled modules via nwjc ? So that a whole NW.js application made up of separate es modules can be compiled into individual binary module files which would work together when the main module is loaded into NW.js. Ideally, as a developper, and as stated by @rogerwang, there should not be any change to make into the application code in order to support it. So, one's best guess would be that NW.js somehow would hook into the Now, how would that connect to binary modules requiring or importing each other ? Can nwjc compile es modules with top-level imports or dynamic imports into binaries which would then be imported at runtime by the application, each binary module being also able to import the others like non compiled es modules? My point is that perhaps to protect the javascript source code in the context of separate modules is not possible with nwjc snapshots, but I may be wrong, not knowing the internals of how a snapshot is made. But to use a customized node's loader as you pointed out, @SMotaal, could also allow to use alternative encryption tools to protect the source code without using the nwjc snapshots (if they do not work in the context of modules). |
|
Again, as just an end-user of NW.js developing js applications, what would be ideal is that NW.js itself loads the compiled binary when an nw.loader['js'].enable();
import('./main.js'); // nwjs takes care of loading the binary if present and all its dependencies.And as a generalized utility, this could be also very useful to have an api to also define custom loaders depending on files extension. In definitive the 'js' extension would be associated with a predefined loader used by NW.js in charge of loading the individual binaries and connecting them together at runtime. There could be a simple commandline switch activating the loader associating the import call of js files with modules precompiled with nwjc. |
|
From my perspective, a wholistic loader system is essential to developer platforms like nw. In my own opinion, browser implementations are mostly preferred, except when it comes to module loading that is adequate for a desktop application due to security restrictions intended for the web, so in this case, node's loader is the way to go. The problem is that hooks for the dynamic import and import meta callbacks are not context-scoped, they are isolate-scoped, so you cannot pick and choose, if nw decides to move forward by depending on them for one purpose (i.e. to catch evalNW's) then anyone needing to hook to them for a different purpose will either override nw's hooks or if nw is aggressive, may end up with a mess where some hooks work and others don't depending on their luck. As a developer, I want to decide on the loader that would be hooked. I want to bring my own — did I mention I have an awesome one already working for me :) — my only problem is that I am not equipped (or honestly maybe due to my limited understanding of snapshots not confident enough to be motivated) to explore adding support for snapshots. If someone knows enough and wants to collaborate, and maybe in the process can helping me understand it better from their perspective, it might turn out to be easier to solve than I imagine. But, please, don't restrict such powerful new additions if you end up using them internally without providing means for developers to properly utilize them if they so choose. |
|
@ab38 You can most certainly add an encryption layer, just keep in mind that unlike snapshots which are consumed in a post-source state (in theory they are never reversed to source) encrypted modules must be decrypted by the loader, so they are simply hard to hack but not unhackable. |
|
@SMotaal , thanks for the interesting comments. Another way to look at javascript source code protection could be to use a custom encryption layer associated with a custom loader, both residing into a single snapshot and chosen by the developer, and hooked via a nwjs api to the So the snapshot (the nwjs api part of it) would just be in charge to transform decrypted javascript code (coming from disk or elsewhere depending on the developer's logic) into a runtime in-memory snapshot. Schematically, the single snapshot would take care of the following process: -> connect to In other words, one snapshot to rule them all... :) |
Proposal for javascript source code protection compatible with ecmascript modules in NW.jsPrerequisites
Rationale
DescriptionIn a startup snapshot, optional custom developer logic is created for dealing with the import calls of particular file extensions. The developer logic is in charge to optionally resolve, fetch, load and decrypt js source code. Then the js source code is transformed into a snapshot via the nwjs api, and executed into the V8 engine. API/*
js content of startup.bin (nwjc dev/startup.js dist/startup.bin)
*/
function init(){
nw.enableLoader( 'js', {
onResolve(importee, importer){
/*
optional custom developer logic modifying the path (importee) of the import call
importee: path of the import call, ie: import('importee')
importer: path of the module importing the importee
return importee by default
*/
},
onFetch(importee){
/*
optional custom developer logic for fetching the source code
return source code or encrypted source code
*/
},
onLoad(source){
/*
optional custom developer logic after fetching the source code
custom decryption logic goes here (local or remote through authorization,
code-signing verification (including the startup.bin), etc.)
return source code or decrypted source code
*/
},
async onExecute(source){
/*
optional custom developer logic for executing the source code
snapshot protection logic goes here via nwjs api
*/
let snapshot = await nw.createSnapshot(source, true); //nw.createSnapshot(source, asModule)
nw.evalNWBinModule(null, snapshot);
/* or
let snapshot = await nw.createSnapshot(source);
nw.evalNWBin(null, snapshot, true) //nw.evalNWBin(frame, snapshot, asModule)
*/
}
}); // enable 'js' loader to hook into import calls
}
init(); |
|
The issue that I pointed out in #6303 (comment) is caused by reloading the page with "Ctrl + R" which I use to do to test my application code, instead of restarting the nwjs process. When one reloads the page, |
|
My two cents: It'd be better if we can simplify the APIs to be consistent, such as:
Thus, it could be more consistent to the HTML semantics: One js file could be loaded as a module or not.
|
|
using latest nightly 0.33.1-sdk show error with dynamic import libimp.js compiling any other library that not contains dynamic import works perfectly |
|
@alxproject these flags of nwjc seems working for me: |
|
@rogerwang Hello, thanks for your answer nwjs 0.33.0-sdk |
|
Hello, evalNWBinModule doesn't work on v0.35.3. module.js: Compile: Run: The code was not executed at all. |
|
It is not fixed in v0.35.5. module.js: Compile: Load and run in index.html: The code was not executed at all. |
|
@lab4quant @rogerwang |
|
For people who are having problems loading the es6 modules, I leave here as I do in my app so that it can help them. Estructure: src mymodule_1.js | mymodule_1.bin export function person( name = "Manu", age = 37, height = 180 ) {
return {
name: name,
age: age,
height : height ,
}
}mymodule_2.js | mymodule_2.bin import { person as Person } from "./mymodule_1.js";
class ClassRoom {
constructor() {
this.students = [];
}
set_student( name, age, height ) {
this.stundents.push( Person( name, age, height ));
}
num_students() {
return this.stundents.length;
}
}
const classroom = new ClassRoom();
export { classroom };myscript.js | myscript.bin import { classroom as Classroom } from "./mymodule_2.js";
class App {
constructor() {
Classroom.set_student( "Manu", 37, 180 );
document.querySelector( "h1" ).innerHTML = "Num of students in classroom: " + Classroom.num_students();
}
}
const app = new App();
export { app };index.html <html>
<head></head>
<body>
<h1></h1>
<!-- Importing modules -->
<script>nw.Window.get().evalNWBinModule( null, './src/modulesES6/mymodule_1.bin', './mymodule_1.js' );</script>
<script>nw.Window.get().evalNWBinModule( null, './src/modulesES6/mymodule_2.bin', './mymodule_2.js' );</script>
<!-- Importing scripts -->
<script>nw.Window.get().evalNWBinModule( null, './src/scripts/myscript.bin', './myscript.js' );</script>
<script type="module">
import { app } from "./myscripts.js";
</script>
</body>
</html>Note that the import in ** myscript.js **, I do not put the relative path to the module that I am importing, I put the base path of the app. Where would you have to write You just have to adapt it to the structure of your app and you should not have problems to make it work. It may seem a bit cumbersome, but with a little skill, I have programmed a compiler that adjusts the import paths in the modules automatically when I compile the application, as well as changing me So with a single console command, the application is compiled, modifies imports in .js and scripts in html, packaged in exe and zip and uploaded to my repository on my server by FTP ready for users to download or when a user who already has it installed, receive automatic update and install. It is undoubtedly the ability of nw.js to convert .js files into .bin files that prompted me to switch from electron to nw.js. When I tried nw.js and saw its capabilities I will never go back to electron. Not only because of the protection of the scripts, but also because of the operation of the application's contexts, the possibility of including Polymer that didn't work in electron and the non-SDK flavor that doesn't allow inspecting the app's windows. Once you understand nw.js you can easily make node.js an automatic compiler with a little effort that even works better with electron builders. I hope to help. |
|
manujoz ... really thanks. it saved a lot of time. |
|
@manujoz this is exactly what i was looking for. Thanks a lot. The documentation doesn't really mention ESM anywhere, so i was starting to think that nw also doesn't support it. (i'll file an issue about that in the near future.) |
and others
NWJS Version : v0.26.6
Operating System : macOS 10.13.1 (HighSierra)
Now we already can use es6 module features in v0.26.6 (Chrome 62). One example is:
`
-------------lib.js--------------
// lib.js
class TestModule {
foo() {
console.log('---foo----');
}
}
export {TestModule};
-----------testmodule.js----------------
// testmodule.js
import {TestModule} from "./lib.js";
var f = new TestModule();
f.foo();
-----------index.html----------------
----------package.json-----------
{
"name": "helloworld",
"main": "index.html",
"dependencies": {}
}
`
The above example can work properly. However, nwjc tool cannot compile them to binary code. The error message is:
`
$/Applications/nwjs/nwjc lib.js lib.bin
Failure compiling 'lib.js' (see above)
`
What's the correct procedure to use es6 module with nwjc? This is important for large scale application.
Thanks.
The text was updated successfully, but these errors were encountered: