Use dune to build config.ml #167

TheLortex · 2019-02-20T17:36:31Z

The change is the following: to build config.cmxs functoria now symlinks config.ml in the _build directory, then it generates a dune configuration file to be able to build _build/_build/default/config.cmxs. It can also grab a dune.inc file in order to add options to the configuration build (necessary for tests).

Tests have been updated to support the modification. They all pass, and I also tested with some Mirage samples. It might be useful to test it on other setups to be sure it doesn't break anything !

(in the mirage/mirage#969 context)

To build config.cmxs functoria copies `config.ml` in the `_build` directory, then it generates a `dune` configuration file to be able to build `_build/_build/default/config.cmxs`. It can also grab a `dune.inc` file in order to add options to the configuration build (necessary for tests). Tests have been updated to support the modification.

TheLortex · 2019-02-20T17:51:41Z

By the way this enables the possibility for config.ml to depend on external libraries but I haven't implemented that yet.

app/functoria_app.ml

…them.

Also renamed dune.inc -> config.dune

Drup · 2019-02-21T19:47:01Z

I'll take a look at this when I have time. From a quick look, there are various hacks which I'm not fond of, I wouldn't mind input from the usual dune experts, @rgrinberg and @diml.

ghost · 2019-02-25T10:17:02Z

Just a few comments after having a quick look at the diff:

I can see a few references to _build, which means that things might not work as expected when the build directory is set to something else. esy does this for instance
it is usually not recommended to manually pass include directories. Indeed, the compiler might read files in these directories but dune won't know about it, which means that it can break incremental compilation
paths such as lib/.test_app.objs/byte are internal details of the build rules that are not documented. In particular, they are not covered by the semantic versioning of dune and we already changed them twice to accommodate various features, so I don't recommend making assumption about these paths

In general, if you are trying to do something that can't be done easily with the current feature set of dune, the best is to open a ticket on github.com/dune describing what you are trying to do so that we can discuss and see what can be done.

TheLortex · 2019-02-25T10:55:03Z

I agree with you on this. I translated to dune what was previously done with ocamlbuild to dune so these hacks are not of my creation.

The goal is to build from a config.ml the config.cmxs file, and then dynlink it with functoria. The way I do it is the following:

A _build directory is created, inside there is a generated dune file and config.ml is symlinked.
dune build --root _build _build/default/config.cmxs is executed to generate the dynlinkable config.
Therefore the dynlinkable file is _build/_build/default/config.cmxs.

The problem is that during tests, functoria has to launch dune inside a build context, as config.ml resides in _build/default/tests/app. In order to build config.cmxs I need to define a build context inside the test build context, which is therefore a bit hacky in order to get things right: the drawback is that inside the inner build context there's no access to outer libraries such as test_app, which forces me to use the include hack.

How could we reduce the complexity in that ?

rgrinberg · 2019-02-25T11:11:49Z

The above sounds like a cross compilation problem to me. You want to generate the config.cmxs in the host context, and then use it to build something in the target context.

ghost · 2019-02-25T11:22:04Z

Would it be easier to define the tests if we consider that functoria has been previously installed? If yes, then the (package functoria) dependency might be the answer: it gives you an environment in which it looks like functoria has been installed. In particular, you can call dune inside this environment. That's how we write all our integration tests in dune. Combined with cram testing, it's really convenient.

TheLortex · 2019-02-25T15:59:37Z

Thank you for your answers. I've been able to get rid of most hacks using the (package functoria) dependency at the cost of defining a functoria.test_app package.
Now the behavior is the following:

app/config.ml and app/dune.inc are symlinked in a app/build-config/ directory.
app/_build/default/build-config/config.cmxs is built using dune. In the test case, it's has access to functoria and test_app.
dynlinking is done as before.

I'll push the update soon.

EDIT: Tests seems to pass but it induced another bug in Mirage unikernels, I'm working on it.
EDIT2: This is fixed.

Now `config.ml` and `dune.inc` are copied in their own `build-config` directory, where `dune` is also generated. Tests don't rely anymore on "include hacks".

app/functoria_app.ml

emillon · 2019-02-28T13:31:40Z

tests/test_full.ml

-     "myocamlbuild.ml" (* FIXME: add a .mirage-ignore file to avoid this *) ]
+    ["main.ml"; "app.ml"; "config.dune"; ".mirage.config"; "dune"; "key_gen.ml";
+    "build-config"; "dune-project"; "_build"
+   (* FIXME: add a .mirage-ignore file to avoid this *) ]


As a reminder, don't forget this :)

- use dune lang 1.0 instead of 1.7 when building config.ml - reuse the build-config location variable

Drup · 2019-03-03T01:15:21Z

Hmm, is using symlinks a potential issue for portability ?

TheLortex · 2019-03-04T09:42:44Z

@Drup On Windows Unix.symlink is a privileged operation (according to OCaml's Unix module documentation) so that might be an issue. However there is already some code that uses Bos.OS.Path.symlink (https://github.com/mirage/functoria/blob/master/app/functoria_app.ml#L584) therefore we might ignore that for now. I don't see any other portability issue. What do you think ?

avsm · 2019-03-11T12:39:46Z

This looks good enough to merge and work on in master for a while...

samoht · 2019-03-17T10:07:03Z

Just a few remarks:

could you use a copy rule in the dune file instead of symlinks?
is config.dune mandatory? Can you describe how to use that file to add external dependencies? I think the current norm is to name these files dune.<smting> so dune.config would be better.

Once this is cleared out, I think we should merge this PR to unlock the dune+mirage port.

rgrinberg · 2019-03-17T14:01:06Z

Let me just note that using the symlink action in dune does not cause portability issues. Dune will in fact copy the file anyway on windows.

TheLortex · 2019-03-18T13:56:49Z

@samoht

* could you use a `copy` rule in the `dune` file instead of symlinks?

I encounter some problems with that because dune.config and config.ml can live outside the dune workspace when the -build-dir is option used.

* is `config.dune` mandatory? Can you describe how to use that file to add external dependencies? I think the current norm is to name these files `dune.<smting>` so `dune.config` would be better.

Let's use dune.config then. This file is optional, and if it exists it's included in the generated dune file, and a dependency to a dummy config_custom library is added. To add external dependencies, you just have to specify them in the dummy library specification, inside dune.config. This is done for tests, for example:

(library
    (name config_custom)
    (libraries functoria.test)
    (modules)
)

samoht · 2019-03-18T17:25:20Z

Looks good, thanks!

@TheLortex

CHANGES: * use `dune` to build `config.ml` (@TheLortex, mirage/functoria#167) * add the ability to use external libraries un `config.ml` via an optional `dune.config` file (@TheLortex, mirage/functoria#167) * Replace dynlink method by a 2-stage build (@TheLortex, mirage/functoria#167)

TheLortex added 2 commits February 20, 2019 17:58

Make dune silent

abc9531

hannesm reviewed Feb 20, 2019

View reviewed changes

app/functoria_app.ml Outdated Show resolved Hide resolved

TheLortex added 2 commits February 21, 2019 10:58

Make sure dune and dune-workspace don't already exist before writing …

89ad5e0

…them.

Add the ability to depend on external libraries.

b81e8d2

Also renamed dune.inc -> config.dune

Drup self-assigned this Feb 21, 2019

TheLortex force-pushed the use-dune branch from e438a7e to 6a228ab Compare February 25, 2019 17:21

Update config.ml build to reduce complexity

dd94b19

Now `config.ml` and `dune.inc` are copied in their own `build-config` directory, where `dune` is also generated. Tests don't rely anymore on "include hacks".

TheLortex force-pushed the use-dune branch from fc60242 to dd94b19 Compare February 26, 2019 10:28

emillon reviewed Feb 28, 2019

View reviewed changes

Small updates

dcb1f2c

- use dune lang 1.0 instead of 1.7 when building config.ml - reuse the build-config location variable

Rename config.dune to dune.config

e65f15d

samoht merged commit d917a5e into mirage:master Mar 18, 2019

hannesm mentioned this pull request May 9, 2019

functoria_app: app_info fix callout to opam list --installed #170

Merged

TheLortex deleted the use-dune branch July 8, 2019 13:53

samoht mentioned this pull request Jul 30, 2019

[new release] functoria-runtime and functoria (3.0.0) ocaml/opam-repository#14611

Merged

This was referenced Jul 31, 2019

Update names #174

Merged

MirageOS 4.0 draft release roadmap mirage/mirage#994

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use dune to build config.ml #167

Use dune to build config.ml #167

TheLortex commented Feb 20, 2019 •

edited

Loading

TheLortex commented Feb 20, 2019

Drup commented Feb 21, 2019 •

edited

Loading

ghost commented Feb 25, 2019

TheLortex commented Feb 25, 2019

rgrinberg commented Feb 25, 2019

ghost commented Feb 25, 2019

TheLortex commented Feb 25, 2019 •

edited

Loading

emillon Feb 28, 2019

Drup commented Mar 3, 2019

TheLortex commented Mar 4, 2019

avsm commented Mar 11, 2019

samoht commented Mar 17, 2019

rgrinberg commented Mar 17, 2019

TheLortex commented Mar 18, 2019

samoht commented Mar 18, 2019

Use dune to build config.ml #167

Use dune to build config.ml #167

Conversation

TheLortex commented Feb 20, 2019 • edited Loading

TheLortex commented Feb 20, 2019

Drup commented Feb 21, 2019 • edited Loading

ghost commented Feb 25, 2019

TheLortex commented Feb 25, 2019

rgrinberg commented Feb 25, 2019

ghost commented Feb 25, 2019

TheLortex commented Feb 25, 2019 • edited Loading

emillon Feb 28, 2019

Choose a reason for hiding this comment

Drup commented Mar 3, 2019

TheLortex commented Mar 4, 2019

avsm commented Mar 11, 2019

samoht commented Mar 17, 2019

rgrinberg commented Mar 17, 2019

TheLortex commented Mar 18, 2019

samoht commented Mar 18, 2019

TheLortex commented Feb 20, 2019 •

edited

Loading

Drup commented Feb 21, 2019 •

edited

Loading

TheLortex commented Feb 25, 2019 •

edited

Loading