Sometimes it can be useful to generate Dune rules that depend on the file system layout or on the content of configuration files. This often happens for integration tests.
In this document, we will see two ways to encode this behavior.
We suppose that we are testing an executable named tool
. There are some input files named *.input
, output files named *.output
, and we want to ensure that when running tool
on x.input
, the standard output corresponds to x.output
.
Note
This is the most common way to do this. It has a couple drawbacks listed below, but you should start with this pattern.
What we are going to do is:
- generate a
dune.inc
file; - include it in our main
dune
file; - commit the generated code in the source repository.
This creates a loop: a program (the "generator") looks at the file system and creates a dune.inc
file. Changes to the file system (for example, if a test is added) mean that a change in dune.inc
will be promoted. These generated rules are included in the main dune
file, so dune runtest
will run the tests. Finally, the generated file is part of the source repository, so it is not necessary to run several commands to run the test suite.
Let's expand a bit on how to achieve this.
Create a gen
subdirectory and create a gen.ml
file in it:
gen/gen.ml
let generate_rules base = Printf.printf {| (rule (with-stdout-to %s.gen (run %%{bin:tool} %s.input))) (rule (alias runtest) (action (diff %s.output %s.gen))) |} base base base base let () = Sys.readdir "." |> Array.to_list |> List.sort String.compare |> List.filter_map (Filename.chop_suffix_opt ~suffix:".input") |> List.iter generate_rules
Create a dune
file in that directory:
gen/dune
(executable (name gen))
This defines an executable that lists *.input
files in the current directory and outputs rules on its standard output.
Note
It is important to sort the input files to ensure that the output is independent from the order in which Sys.readdir
returns the files .
For each input file, we output two rules:
- The first one creates a
x.gen
file that corresponds to the actual output. - The second uses a
/reference/actions/diff
action to compare the actual output to the expected output. If it is different,dune runtest
will display the difference, which can be accepted bydune promote
.
Note
It is possible to have more complicated logic here. For example, to pass different arguments to tool
depending on the presence of a *.args
file. To do that, check if *.args
exists in generate_rules
and emit a different (run ...)
action.
Our main test dune
file contains the following:
dune
(include dune.inc) (rule (deps (source_tree .)) (with-stdout-to dune.inc.gen (run gen/gen.exe))) (rule (alias runtest) (action (diff dune.inc dune.inc.gen)))
In addition to including the contents of dune.inc
, we use the same pattern as before: dune.inc.gen
is the actual output of the generator, and dune.inc
is the expected output. At runtime, the generator will read the contents of the current directory (where the *.input
and *.output
files are located), so we record (source_tree .)
as a dependency to make it run again if a file is created, for example.
To make this work, we have a final step to do. We have to add the generated file to our source tree. But since it is generated, we will have to first create an empty file, run the test, and promote the result.
$ touch dune.inc
$ dune runtest
+ (rule
+ (with-stdout-to a.gen
+ (run %{bin:tool} a.input)))
+
+ (rule
+ (alias runtest)
+ (action
+ (diff a.output a.gen)))
$ dune promote dune.inc
$ git add dune.inc
Now, running dune runtest
will run the test suite.
This pattern is "correct": it will execute all tests and make sure the list of tests is up to date. But when adding a test, it is necessary to first run dune runtest
, promote the result, and then re-run dune runtest
to actually run the test (and possibly promote the result of the test itself).
There is a variant of this pattern which will promote the output automatically instead of using a manual promotion step. This variant can be used either for the test list or for the individual tests.
To use it in the test list, replace the dune
file by this version:
dune
(alternative version)(include dune.inc) (rule (mode promote) (alias runtest) (deps (source_tree .)) (with-stdout-to dune.inc (run gen/gen.exe)))
Using this version, dune runtest
will directly replace dune.inc
with an updated version.
Another caveat of this approach is that the generator needs to emit the same output on all systems. For example, if some tests should be skipped on Linux, the generator can not just filter the corresponding tests depending on Sys.os_type
. It has to consistently emit a (enabled_if)
field for the rules.
3.14
This technique relies on /reference/dune/dynamic_include
, which is more flexible than /reference/dune/include
. The difference is that the intermediate dune.inc
file does not need to be part of the source tree. It will only be generated by a rule and be present in the _build
directory.
At first it looks like it would be possible to reuse the same pattern as above: change include
to dynamic_include
and delete the dune.inc
file. However, it is not possible. The reason is that rules are loaded per directory, and there needs to be a strict order (no cycles) between directories for this to work.
So, instead we are going to:
- generate
dune.inc
in a subdirectory namedgenerate
, and - include these rules in a subdirectory named
run
.
These subdirectories do not need to be actual directories. They can be emulated through /reference/dune/subdir
.
To do this, we can create the following dune
file in the same directory as the *.input
and *.output
files.
dune
(executable (name gen)) (subdir run (dynamic_include ../generate/dune.inc)) (subdir generate (rule (deps (glob_files ../*.input)) (action (with-stdout-to dune.inc (run ../gen.exe)))))
Then create the following gen.ml
file. Note that here we can define it in the same directory.
gen.ml
let generate_rules base = Printf.printf {| (rule (with-stdout-to %s.gen (run %%{bin:tool} ../%s.input))) (rule (alias runtest) (action (diff ../%s.output %s.gen))) |} base base base base let () = Sys.readdir ".." |> Array.to_list |> List.sort String.compare |> List.filter_map (Filename.chop_suffix_opt ~suffix:".input") |> List.iter generate_rules
There are a few differences from the generator above because this one is going to be invoked from subdirectories, so it is necessary to refer to the ..
directory both in the input (which files to read) and in the output (how the rules are executed).
These two files are enough. dune runtest
is going to generate the rules and interpret them in a single command.
This approach is shorter, but it might be more difficult to debug because changes to the generated rules will not be visible. Also, it works in that case, but it is not possible to generate all kinds of stanzas with that pattern. See /reference/dune/dynamic_include
for more information about the limitations.