split js_of_ocaml in sub libraries

[hhugo]

• js_of_ocaml.base (compatible with nodejs & co)
• js_of_ocaml.dom? (browser's api)
  depend on js_of_ocaml.core
• js_of_ocaml.lwt (lwt specific)
  depend on js_of_ocaml.core and js_of_ocaml.dom
• js_of_ocaml.async (see Initial support for async (WIP) #261)

[Drup]

I think it's a good idea. We should be careful with backward compat though.

[bobzhang]

I am really glad to see this change, also, I think it will be very helpful to cut the dependency of js_of_ocaml.core .
In my company, we have our own javascript engine which is not running under browser, it would be nice to see no dependency on lwt, thanks

[hhugo]

@bobzhang do you already use js_of_ocaml to run ocaml programs in a browser-less JavaScript context ? If so, may I ask what's your use case ?

[alainfrisch]

FWIW, my company would also be very interested if js_of_ocaml could be split into parts, with a kernel containing only the compiler and the js runtime support (no OCaml library at all). It would also be of great help if external dependencies for this kernel were reduced to a minimum:

• menhir dependency could be dropped by shipping generated parsers;
• avoid the use of cppo (unless I'm wrong, this is used in the compiler only to address the optional dependency on findlib, but this could be achieved without preprocessing, by linking optionally a module which defines a "hook" calling findlib)
• there is a single use of base64 encoding in the compiler itself; I don't know if this can be replaced by something simpler, but otherwise, re-implementing base64 encoding is not that hard
• findlib dependency is already optional

If I'm correct, this would leave only the dependency on cmdliner, which would already be quite good.

FWIW, any dependency bring difficulties for at least two reasons: (i) many packages tend to break under Windows, and anyway, OPAM doesn't work with native Windows ports yet, so installing these dependencies is still manual on Windows; (ii) the licenses of all dependencies need to be inspected closely for use in an industrial setting, and even if these licenses are certainly fine for our internal use and our end-user customers, some of our technology customers have stricter rules for the license on the code we deliver to them, including required dependencies.

[Drup]

I would like to point out that cppo and menhir, being build deps, are irrelevant to point (ii) (and to point (i) if you consider cross compil, but that's a mine field on it's own, I would be very surprised if they didn't build on windows anyway).

Splitting everything can be advantageous, but let's not overdo it, as it complicates maintenance and optional dependencies often cause packaging complications.

I'm very in favor of @hhugo's original plan, though.

[hhugo]

making camlp4 an optional dep would be nice. But I don't quite see a nice transition for this.
The js_of_ocaml libraries needs some kind of syntax extension and we cannot move to ppx if we want to keep compat with older OCamls.

[Drup]

I agree, and I don't see any either, except duplicating the codebase (urrrgh).

[alainfrisch]

I would like to point out that cppo and menhir, being build deps, are irrelevant to point (ii) (and to point (i) if
you consider cross compil, but that's a mine field on it's own, I would be very surprised if they didn't build
on windows anyway).

I don't see why build deps are irrelevant to point (ii) (except for binary packages of js_of_ocaml) : companies that need to build js_of_ocaml codebase need to install these dependencies as well, and so are bound by their license. For instance, if the license of one of these build deps prohibited use for commercial purposes, companies would not be allowed to use them, and so to build js_of_ocaml.

Splitting everything can be advantageous, but let's not overdo it, as it complicates maintenance and
optional dependencies often cause packaging complications.

My suggestions were not actually about splitting the package, but rather reducing dependencies for some parts of it. For instance, creating a custom implementation of the base64 encoding function, shipping pre-compiled Menhir parsers, and avoiding cppo in the compiler directory would already address most concerns with dependencies. Users could then quite easily decide to compile only the compiler subdirectory, and this would work e.g. even if lwt is not installed.

[Drup]

They do need to install them, but the final product doesn't depends on their license. Code generated by a tool is not considered bound by the license of said tool (otherwise binaries produced by caml would be bound by the Q public license !!). You can make money selling binaries produced by OCaml, you can't make money selling modified version of OCaml (except when member of the consortium as I'm sure you are very well aware :p). The same distinction applies to cppo and menhir and any build-time code generator.

[alainfrisch]

They do need to install them, but the final product doesn't depends on their license.

Yes, but when a company installs some software, they need to check that its license allows them to use that software. The fact that their license doesn't pollute their result means that js_of_ocaml are allowed to have e.g. the Menhir-generated code licensed as they see fit.

[agarwal]

Code generated by a tool is not considered bound by the license of said tool

This is not a generally true statement. A license can say anything. So no matter how you're using some code, there is the burden of checking its license.

[Drup]

Indeed, I was talking about most licenses around.

[bobzhang]

a sad truth is that opam is not always allowed for some companies. make js_of_ocaml self contained definitely could help more people evaluate such library

[bobzhang]

we can have a dev mode and release mode, when you release code, you can release generated code instead

[agarwal]

opam is not always allowed for some companies

Can you explain the restriction? I can understand that the default public opam repository isn't allowed, but I don't see why opam wouldn't be allowed. You can rather easily create a private opam repository, which seems very supportive of a company's needs. You can tightly control exactly what packages are available.

[bobzhang]

@agarwal the thing is we have our own in-house package system, people don't like to learn yet another package system(I can only play opam at home). for js_of_ocaml such a fundamental tool(another backend of ocaml), it would be nice that the dependency is minimal

[agarwal]

shipping pre-compiled Menhir parsers ... would already address most concerns with dependencies

Are you sure? The Menhir license could restrict you directly using Menhir yourself, or restrict you in using code generated by Menhir. Do you know which, or do you still have to check with your lawyers about this? If you do still have to check with your lawyers, then your point (ii) doesn't benefit. My point is, I also agree with @Drup about not overdoing it. Avoiding dependencies also causes headaches.

[alainfrisch]

Do you know which, or do you still have to check with your lawyers about this?

The burden is transferred to maintainers of js_of_ocaml. If they are allowed to redistribute the code generated by Menhir (and of course they do!) under their own license, users of js_of_ocaml simply don't need to deal with it. Apart from the legal issue, it'd also greatly simplify the technical aspect of installing js_of_ocaml (esp. for companies that cannot use OPAM either because of internal policy or because Windows is one of their supported build environments).

[agarwal]

The burden is transferred to maintainers of js_of_ocaml.

I don't think so. Some other organization may interpret a license incorrectly. That doesn't free your company from having to abide by the correct interpretation. If you're really concerned about obeying all the rules, your lawyers will still need to get involved. Sorry, I don't mean to drag this conversation on. I just feel people often exaggerate the complications of dependencies, and to me this leads to many problems too. There is no clear answer, and obviously your company may have concerns that I don't understand.

We should be less abstract. Do you have a problem using Menhir specifically? The licenses involved are the Q License and LGPL. I'm guessing it won't be hard for your lawyers to approve it. If it is hard, then I'm curious how you manage to use so many other software. (I don't know about using Menhir on Windows, so that may well be a valid point.)

[dbuenzli]

js_of_ocaml.dom? (browser's api) depend on js_of_ocaml.core

I think we need a good name for that. So that it be reused by package authors. On my side I have the following names:

• vg.htmlc, it's mostly okay I think, since I may want to distinguish the html canvas backend from an hypothetic webgl one. Maybe we are missing the fact that this is js (see below).
• useri.jsoo, it's not okay this is "browser api".
• mtime.jsoo, it's not okay this is "browser api", I may want to implement, say mtime.nodejs at some point.
• fut.jsoo, it's not okay this is "browser api".
• jsont.jsoo, it's okay JSON.parse is in ECMA.

Dumping ideas.

• PKG.jsoo (or PKG.js ? it seems other people are using that, e.g. in my ocamlfind list I have at least github.js, xtmpl.js). This means pure JavaScript (basically ECMA whatever version).
• PKG.browser, PKG.dom, PKG.js_www, whatever... something that runs in the browser.
• PKG.nodejs, specifically uses nodejs APIs.

Or maybe a better strategy would be PKG.js[_$API], the js indicating JavaScript backend, i.e. jsoo (I don't think there is more than one contender these days) and the [_$API] making precise where this can run.

In any case I think it's a good idea if the jsoo project tries to decide on a few conventions, adheres to it for its own libraries and documents them for others to use so that orientation is simplified for package users.

[alainfrisch]

Do you have a problem using Menhir specifically?

We have two kind concerns with any dependency:

• We don't currently use any package manager, all external dependencies are included in our main code base (so that we can go back to the exact same configuration as any point in the past with just an "svn up"), and we usually reimplement their build system from scratch (to integrate them in our main omake-based build system, and to make them work under all platforms we support). So adding any dependency takes some effort indeed. (Moreover, currently, we don't install ocamlbuild locally; and camlp4 -- and any packages that depend on it -- cannot be used, since it is not ported to our local fork of OCaml.) I'm just describing the current situation, I'm not claiming this is the right approach.
• We don't have any problem with our lawyers, but with those of our clients (including those of another participant to this discussion). When we deliver our code base in source form to technology clients, they often require us to list in the agreement all external packages on which depend to build our software, and they need to check the licenses of all these packages. Sometimes, they do some "open source due-diligence" to check that the code we deliver don't contain (copies of) public OO code. And so we cannot afford to add a new external package in the subset of the code we deliver for a given "live" project. Any external dependency then becomes a legal burden for us.

We are ok to pay the price for these technical and legal burdens; we are only kindly requesting that external dependencies are introduced sparingly and that small changes allowing to get rid of some of them be considered. Including Menhir-generated parsers in distributed packages (which is an easy way to formalize that they are covered by js_of_ocaml's license) is such a change. Similarly for using Makefile-based conditionals instead of relying on cppo for the compiler subpart (enabling/disabling findlib support).

[agarwal]

Thanks for the explanation. I see you do have substantially greater complications than usual.

[bobzhang]

we have exactly the same concerns as @alainfrisch , thanks

[jmineraud]

I was wondering it would be possible to split the library a little bit more.
For instance, in https://blog.codecentric.de/en/2014/06/project-nashorn-javascript-jvm-polyglott/ there is picture of the different blocks (see section "Nashorn is only a JavaScript Engine").

I am personally using XmlHttpRequest and loop events, but I would like to get rid of all other dependency (for example window.location).

[vasilisp]

Bringing this up for the optional-Camlp4 part. Are we finally ready to translate the lib to PPX and make the Camlp4 bits optional?

The OPAM package currently depends on 4.02.x, and so does Tyxml_js. Even if the rest theoretically runs with 4.01 and older, for most people JSOO practically requires 4.02 or newer.

(I volunteer to do the translation and build system updates, it shouldn't be a lot of work.)

[agarwal]

make the Camlp4 bits optional

Is continued support for camlp4 important? Why not get rid of it and reduce the maintenance burden.