Native windows support

[bschommer]

Currently ocamlbuild requires some either mingw or cygwin under windows to work, which is not really a good solution.

I think it is not that complicated to get a working version under windows and I already tried to patch it in the past to work for my local build. The main problem is the use of cp, rm, etc. in shell.ml which can be replaced by the windows pendants and the calls of Sys.command, etc.

[bobot]

This solution seems easier than reimplementing all these commands with Unix and it doesn't require to modify the scheduler.

Do you plan to always use the windows pendant on windows or to select the right version?

[bschommer]

I think it would be safe to use the windows pendants under windows instead of relying on an underlying bash. However the main problem was that most commands are executed through functions that call cmd.exe (Sys.command, Unix.execv, etc.) and would need to be replaced by create_process in order to avoid taking care of quoting everything correct.

[fdopen]

The major problem is

ocamlbuild/src/signatures.mli

Line 190 in 13feb3c

| Sh of string (** A bit of raw shell code, that will not be escaped. *)

, not external tools like mv and cp.

Many existing rules rely on the fact that Sh is a unix shell (e.g. cygwin's or msys's bash under Windows), not cmd.exe. (e.g.

ocamlbuild/src/ocaml_compiler.ml

Line 90 in 13feb3c

Cmd(S[A"touch"; P mli; Sh" ; if "; cmd; Sh" ; then "; rm; Sh" ; else ";

)

I've just recently tried to remove the indirection steps through cmd.exe and bash.exe as much as possible and logged external calls of dozens existing packages. (but just for performance reasons, so i didn't replace cygwin's mv, cp with windows equivalents)

[bschommer]

I was just referring to my experiments which I did in 2014 and at least for simple projects one could get a result by replacing mv, cp, etc. as well as the bash before each command but more complex rules resulted in problems with quoting.

[gasche]

I think any PR that minimizes the use of bash for Windows users would be interesting. It is good for performance reasons, as fdopen mentioned, and even if it does not completely solve the build-without-cygwin problem for all cases, it's a good first step in that direction.

[bluddy]

I would also like to see ocamlbuild work on native Windows. Additionally, it would be great if ocamlbuild had features that made it more like a full solution for building across platforms. What do I mean?

1. Command line args for debug/native/bytecode rather than odd suffixes.
2. Build the executable at any location rather than needing to symlink (a unix-ism)
3. Specifying targets so that a Makefile isn't needed in the common case (another unix-ism)

All of this should be doable without any need for OCaml plugin code, since that's just not something the average user wants to mess with.

[gasche]

You can already build a set of targets using the .itarget -> .otarget rule (see ocamlbuild -documentation). You can pass -tag debug to get debug build, but I'm sceptical that confusing native and bytecode executables under the same name would actually help.

If you don't want to wrap the ocamlbuild invocation using a script, you have to write OCaml plugin code. It should be better documented than it currently is (help improving the manual is welcome), and better APIs could be provided (they can also be distributed separately as plugins, at least at the beginning while their design stabilizes), but it is not fundamentally wrong to propose "average users" to use it when they want to do more than just building targets.

[bluddy]

The reason mixing native and debug is a good idea is that people don't expect their build tools to force the name of the build targets. If I want to differentiate my build names, let me do it as I will. Using a non-standard convention is really bad (IMO), unless you're always expecting me to wrap my calls with a shell script or symlink, both of which make using ocamlbuild both harder and less portable than it should be.

Diverging from common practice as far as the outside API is concerned is a really big problem. It certainly doesn't help that OCaml chose to have long arguments with a single dash (as in -documentation). No external generic user of a program that just happens to be written in OCaml expects the executable to have a .native or .byte suffix!

I personally want ocamlbuild to become the main go-to build tool for OCaml on any platform, including Windows. Look around OCaml projects though, and you see everybody still has a Makefile, sometimes from Oasis, and there's a need to deal with ocamlbuild's weird imposition of suffixes. Why should we have to wrap ocamlbuild's output so much? Why can't we get it to a point that it 'just works'?

The truth is, I don't care whether ocamlbuild is the default tool or ocp-build or something else is. I just want something that covers 90% of users' needs and can be made easy and cross-platform. On the first point, I think ocamlbuild qualifies, and the new Namespaces plugin makes it even more promising. The latter point is where ocamlbuild currently fails IMO. I would be happy to try and help deal with this issue if you tell me that the will is there to improve the current state of things.

[bschommer]

I fully agree with @bluddy that a unified build tool for all platforms would be nice, especially since gnu-make under windows is a pain to use.

[gasche]

My personal worry is fragility and complexity. OCamlbuild is already a complex tool. I'm all for adding things that respect the tool's internal structure and design and also help users. I'm ready to extend said structure and design if we get concrete proposals that strike a reasonable compromise in terms of simplicity and expressivity.

But I'm afraid that adding everything-in-the-kitchen-sink in the never-ending quest to do things "just right" by a magic combination of a myriad of additional command-line arguments or special-purposes config file is a recipe for disaster rather than convenience.

I agree the Namespace extension is very interesting work that goes in the right direction. I warmly welcome any pull request that gracefully extend OCamlbuild to solve user problems -- or that makes it easier to build solutions to these problems on top of it. But I don't think we should sacrifice internal consistency for the sake of convenience -- as the saying goes in a different context, this is a sure way to get none of them in the end.

To make things more concrete: the problem with proposals to build targets outside build is that mixing source artifacts and result-of-building artifacts in the same directory tree that ocamlbuild sees has no well-defined semantics -- currently ocamlbuild assumes that everything in its traversed tree (so in particular, not the build dir) is "source artifact", that should be considered as "the truth" by the tool and in particular is never rebuilt. If you start adding features to create build artifacts inside this traversed tree, either you expect users to follow a strict discipline of always explicitly telling ocamlbuild what is not a source artifact (and I think this is hopeless) or you expose yourself to really unexpected build behavior. If you have a good solution to solve this design issue (discussed in #17 ) I'm all ears, but I'm afraid this unresolved design tension makes your "just works" goal difficult to reach.

[bschommer]

Concerning the problems with cmd.exe etc: I would be willing to help and contribute workarounds. However I think the main problem is still that avoiding cmd.exe would result in needing the Unix module, since in my experience create_process is the only way to reliable execute any commands under linux/windows and from what I have seen in the source code this is not really desired.

[bschommer]

Just a short question: Would it be acceptable if one wants to use the unix module in more places. I would like to try changing the calls to external tools to use create_process. But I'm not sure if this would not break some assumptions made by ocamlbuild. Is there a general way the separation between the unix dependent parts and the one not depending on unix are separated?

[gasche]

I don't actually know about why the my_unix.ml mechanism is in place. I think that in the very old days (before it was first released, I think), ocamlbuild had plans to have extra features relying on sensibly more advanced operating system features (see misc/opentracer.ml relying on strace/ktrace for example). If you want to replace this abstraction by direct calls to the Unix modules, it's basically fine with me as long as we are careful not to introduce regressions.

ocaml-fileutils has reimplementations in pure OCaml of many posix filesystem utilities. I wonder whether using them directly (instead of constantly being in the path escaping business) couldn't aslo help. It is a bit easier to use third-party libraries now that ocamlbuild is released separately from the compiler, and we could take advantage of that -- but rather carefully, because ocamlbuild is on the dependency path of many packages and, besides the risks of cyclic dependencies, people could notice an increase in total build/install time for the package.

[bschommer]

Thanks a lot for this clarification. I will have a look into this.

[dbuenzli]

Note that IIRC @dra27 may have some patches that improve ocamlbuild on Windows. Also it seems silly not to use Unix since contrary to what its name suggest it has a lot of Windows compatibility (thanks to David 4.03 even brings you symlinks). Regarding using third party libraries for ocamlbuild I think it's better if it remains without dependencies.

[bschommer]

Using just unix should be fine. The good thing about for example the Unix create_process function is that it you do not need to quote the input which avoids the obvious (and often wrong) solution of putting just another layer of quotes around the arguments.

[bluddy]

@dbuenzli, I disagree. Unix is still far too sparse to allow us to replace all shell-isms. For example, it doesn't support mkdir -p. I think ideally, we want to deprecate and remove all the my_unix stuff, deprecate and remove the Sh constructor, and use (and enhance, where appropriate) FileUtils + Unix for process manipulation (though an easier library would help there as well).

Nothing wrong with dependencies in the OPAM era, IMO.

[dbuenzli]

For example, it doesn't support mkdir -p.

That doesn't look insurmountable.

Nothing wrong with dependencies in the OPAM era, IMO.

As @gasche noticed build systems hold a particular place in the OPAM dependency tree which makes this blanket rule not applicable here. If dependencies are being used I'd suggest to vendor them in (like OPAM itself does).

[bluddy]

As @gasche noticed build systems hold a particular place in the OPAM dependency tree which makes this blanket rule not applicable here. If dependencies are being used I'd suggest to vendor them in (like OPAM itself does).

Fair enough. Nevertheless, I think that we really want to provide enough of a wealth of utility functions that there's no need for resorting to the shell -- however we do it.

[gasche]

Moving towards deprecating the Sh constructor seems like a reasonable idea to me. A first step would be to get a list of the packages that currently use it, and a second step would be some sort of analysis of the need it fits for them and what they could do instead -- or new features in ocamlbuild that could help to transition away. @bluddy , would you be interested in giving this a start?

(One way to do this could be to download all opam packages that depend on ocamlbuild, look for those mentioning Sh in a myocamlbuild.ml file, and report the list of such packages. For the record, here is the opam script I used to download all opam packages, and analyse their unpacked archives to find out which had ocamlbuild dependencies. This may be a building block if someone decides to have a crack at it. Another, probably simpler, solution could be to use the extract_mini_repository.sh script provided by opam upstream, or build on it.)

[dra27]

I'm not sitting on any private patches at the moment (at least I don't think I am!) - but the changes in 4.03 to Unix.stat automatically improve ocamlbuild's handling of native symlinks (which is to say that they actually work).

I'd agree that the long-term aim should be the complete removal of shell commands from the core of ocamlbuild - ocaml-fileutils I think is abandoned or at least in need of more assistance (I think there was a comment on opam-devel of an experiment in using it for opam, and it had to be abandoned because of too many bugs/missing features).

[bluddy]

I probably won't get around to doing this for a couple of weeks -- I have my GBO test coming up.

[gasche]

Getting rid of Sh and unix userland utilities may become less important in the future:

Ars Technica: bash and Linux command line coming to Windows 10

[bschommer]

That sounds awesome.

But for the moment we are stuck with the current api. I did some experiments and at least for the tests shipped with ocamlbuild just replacing the call to Sys.command by create_process already worked (Minus the fact that we one needs to split the command string again).

[Chris00]

FYI, while it is good to be compatible with the older versions of Windows, this news may help for Windows 10.

[bschommer]

After reading a bit more about it, I doubt it since it seems to me that it is similar to the already existing SUA utils (until Winodws 8) which were always kind of broken.

[dbuenzli]

In the long term getting out of shell scripting seems a better idea anyways.

[dra27]

It is now available, if anyone is using the Insider program (I don't have time, or rather a properly configured VM, at the moment). It does potentially sound interesting - because they're doing it with a kernel layer rather than a user-space layer (as Cygwin does), they're able to implement, rather than emulate, fork which is the fundamental thing which makes Cygwin so slow. The fact that they're actually pulling in Ubuntu may make it better than the old Services-for-Unix - the new cuddly open-source Microsoft is tricky to work out 😄

But philosophically, if we want a build system which wraps shell calls, we have make and it does that quite well. In the perfect world, an intelligent system like ocamlbuild shouldn't need it (in my not-remotely-humble-opinion).

[msprotz]

I'm not sitting on any private patches at the moment (at least I don't think I am!) - but the changes in 4.03 to Unix.stat automatically improve ocamlbuild's handling of native symlinks (which is to say that they actually work).

Interesting... because on 4.02.3 basically ocamlbuild doesn't work until you first delete the foo.native symlink

[modlfo]

Any updates on this topic? My builds on windows take about 3 times more (on the same machine) compared to linux. So anything that could reduce my current build time (18 minutes) on windows is greatly appreciated.

[bschommer]

Since there seems to be an increasing interest in getting a working ocamlbuild for windows I would like to make a few suggestions:

First it would be a good idea to set up some continuous integration. Since ocamlbuild is open source it should be possible to also set up free travis, appveyor, etc.

As a next step one could remove the calls to cp, mv, rm, etc. as well as the my_unix module by implementations based on the Unix module and since ocamlbuild is no longer part of the ocaml compiler itself one even could use third party libraries for this part.

For removing the [Sh] construct there are several possibilities. One could either break the current API and just remove them. This would also open the possibility to remove or replace other parts of the API. The other possibility would be to implement a correct quoting function for cmd.exe.

[adrien-n]

A quick pot-pourri of hints and answers, some which pertain to comments from the beginning of the thread. In no particular order.

I don't think that the current reliance on external tools impacts the build times significantly because they are called infrequently afair.

Build times are bound to be longer on Windows because spawning processes takes much more time there and I/O is pretty slow. Raw computing performance is unchanged so the main culprit might be the number of times the compiler itself needs to be invoked.

SUA was based on an older netbsd and had fairly poor performance as I understand. The new Linux subsystem is not a solution either because most things there live in a separate namespace (e.g. filesystems: some aspects cannot be merged easily). It's still fairly new and I had troubles when running patch in the first build so I'm not holding my breath yet.

Avoid symlinks: they are particularly limited in the common case and only less limited (but still) when running with admin privs. MSYS' ln is actually cp. Cygwin provides (almost) conforming symlinks but they can only be used in the cygwin environment. Ocamlbuild doesn't make an extensive use of symlinks: just copy the output file, our machines definitely can handle that.

ocaml-fileutils needs some love to work well for these needs. I think it's where the efforts should work, rather than half-reimplementing it in various projects and while making them only half-working (the half that you care about but still half).

I haven't commented on anything gasche has said I think, because I +1 pretty much everything he said.

[agarwal]

To make things more concrete: the problem with proposals to build targets outside build is that mixing source artifacts and result-of-building artifacts in the same directory tree that ocamlbuild sees has no well-defined semantics -- currently ocamlbuild assumes that everything in its traversed tree (so in particular, not the build dir) is "source artifact", that should be considered as "the truth" by the tool and in particular is never rebuilt. If you start adding features to create build artifacts inside this traversed tree, either you expect users to follow a strict discipline of always explicitly telling ocamlbuild what is not a source artifact (and I think this is hopeless) or you expose yourself to really unexpected build behavior. If you have a good solution to solve this design issue (discussed in #17 ) I'm all ears, but I'm afraid this unresolved design tension makes your "just works" goal difficult to reach.

I didn't read this whole thread but want to comment on this. Allowing build artifacts in the source tree doesn't require users to specify what is not a source artifact, which I agree is hopeless. Rather, it can be accomplished by requiring users to specify what is a source artifact, which is attainable and I think even sensible (we do this in solvuu-build). However, I'm not sure this is compatible with ocamlbuild's current rule set, so maybe it is not practical here.

[bschommer]

Since dune seems to work fine under windows, also for my needs I think one could close this.