[RFC] Better support for js system operations in the stdlib

[alehander92]

The Nim JS backend is very useful for targetting node-based code (e.g. Electron). However one still needs to do a lot of system stuff there: IO, path handling etc.

In my projects I have implemented parts of Nim's system stdlib modules for the nodejs backend and I want to include those (and kickstart an effort to map even more stuff if somebody else works with this backend) upstream.

This is my plan (however it has a lot of ?, so please help me improve it):

1 Resolve the jsffi vs system problem:
In my implementations of some primitives, I reuse some node importc functions which often deal with JsObject and other stuff from jsffi. However I can't depend on jsffi in system:

I can make jsffi implicitly part of system for nodejs (which probably would've made sense in the beginning, but now it might be breaking code), or I can make those functions require jsffi for the nodejs backend (which might be not compatible with other code, still you can use when defined(nodejs): import jsffi and write cli tools that work with both backends)

2 Include most functions in when defined(nodejs) in their original modules

3 Add some tests (I am not sure I found tests for IO etc, where are they)

4 Add async versions for some functions: they might be unique for the nodejs backend, but they are very useful and having the same signature will be convenient (or maybe we will have async readFile for C one day?)

5 Expose some node types/functions in something similar to jscore

[dom96]

I'm afraid I must be honest here and repeat the sentiments I sort of shared on IRC. Compiling Nim to nodejs is wrong.

The main reason Nim supports nodejs is to test our JS backend (which is essential for running in the browser). But I want to stop this trend of compiling Nim to JS for the node target, doing so is absurd.

If you're using node because you need Electron then there is a great alternative for you: libcef. We even have a wrapper: https://github.com/jangko/nimCEF. It's time somebody wrote an Electron-like library for Nim. You'll get better performance, and support for all of Nim's standard library. This is IMO a better use of time than wrapping nodejs.

I guess right now I can't stop you. But I'm sorry to say, I strongly believe any other node-specific modules should go into a Nimble package. I'm happy to accept JS code that works in the browser and nodejs, but nothing more that is node-specific.

[alehander92]

Honestly, I thought about libcef and I just don't think that would be very revolutionary.

It would be very nice to have a Nim electron-like library, but the real win would be to somehow optimize the UI part: and I don't see how libcef would be much more efficient than Electron as UI.
Native Nim would be still usable only for the index processes which rarely are the bottleneck.

Now, if you imagine Nim compiling to wasm and using not the DOM, but some kind of optimized webgl-based UI, that would be very cool, but that's a way bigger project than just replicating a subset of Electron with native Nim support. Also it's ecosystem would be lacking if it can't support none of the existing web js libs.

Such a library based on Nim would be amazing, but right now we already have full working Nim support for Electron. Yes, it does require Node for its index process, but wrapping some Node functions is way simpler (and you can reuse the whole JS ecosystem in the UI). There are very good and real reasons to target Node.

Now, I think having this kind of support in the stdlib would be cool, I even thought about mapping some of that to the browser too (so e.g. you can do io with local storage etc).
If there is a good way to distribute that as a Nimble package, I guess that can work too, but compiling Nim to nodejs is not "wrong" and "bad", it's pragmatic today (I'd be happy to see the day when it's not needed anymore).

[alehander92]

I think attacking Nim/Electron combo which works right now with almost the same ease and tooling support as Typescript/Electron in order to help a hypothetical Nim-based Electron equivalent with very unclear benefits is just not very good for Nim: let's have both of them.

EDIT: in a lot of the desktop browser apps big part of the code will be UI-based, so it will compile to JS in the near future anyway: having Nim as an equal alternative to Typescript/ClojureScript/StuffToWasm etc there is also pretty important even if the JS backend is originally just an addition

[alehander92]

tl;dr (about Electron vs libcef)

• We can already write desktop browser-based apps fully in Nim: with Electron
• Having native Nim for the main process is nice, but the the bottleneck of all those apps is usually the UI and browser.
• I can't see why such a new framework would be a killer app, if it doesn't have something radically different in the UI implementation: maybe some kind of very modular browser which doesn't pack things you don't need for an app. Including a whole browser is what people hate and love about Electron, and if the main process is native doesn't seem to matter so much

[yglukhov]

Just an idea. A while ago I ended up with a portable_gl module that wraps either nim-opengl or webgl depending on the target. You could implement portable_x that is a drop-in replacement for std/x that supports both native and nodejs targets. Of course the downside is it has to be injected into all your dependencies that use x, but it's a good start anyway.

[alehander92]

@yglukhov Yes, that's similar to the nimble package idea (I guess with the right settings one can make those x come from there only for nodejs)

I think this is more a discussion of if we need node-specific stuff in stdlib or not. What do you think?

[zah]

Well, obviously the benefit of using Electron is that you can leverage HTML as an UI system (e.g. base your UI on Karax).

If @alehander42's work can be pushed as a nimble package, this would be a nice option, but is this really possible? Imagine a program using a module such as os.nim. If you want to compile it for both Electron (nodejs) and the regular C target, you can try to alter the module search paths somehow to intercept the os.nim import and replace it with an alternative implementation, but you'll probably end up duplicating some of the definitions in there (the helpers that sit above the low-level cross-platform code). The only benefit of this seems to be that the stdlib won't be burdened to support the nodejs portions in the future.