Added the direct dependencies between modules to the output of dune describe

[esope]

When the new --with-deps flag is given to the command dune describe workspace, the output now contains a new field module_deps for each module entry. This new field contains the list of modules of the current library/executable a module immediately depends on.

For example, on a simple library project with two files a.ml and b.ml such that a.ml refers to the module B, with the following dune file:

(library
 (name a))

the command dune describe workspace --with-deps prints:

((library                         
  ((name a)
   (uid 3210b99379777ec5c43564aae2d79282)
   (local true)
   (requires ())
   (source_dir _build/default)
   (modules
    (((name B)
      (impl (_build/default/b.ml))
      (intf ())
      (cmt (_build/default/.a.objs/byte/a__B.cmt))
      (cmti ())
      (module_deps ()))
     ((name A)
      (impl (_build/default/a.ml))
      (intf ())
      (cmt (_build/default/.a.objs/byte/a.cmt))
      (cmti ())
      (module_deps (B)))
     ((name A__)
      (impl (_build/default/a__.ml-gen))
      (intf ())
      (cmt (_build/default/.a.objs/byte/a__.cmt))
      (cmti ())
      (module_deps ()))))
   (include_dirs (_build/default/.a.objs/byte)))))

[esope]

This PR results from this discussion thread, and hopefully provides a solution to issue#3425.
CC @nojb @rgrinberg

[rgrinberg]

Do you think it's perhaps better to print these sets separately and let callers merge them if needed?

[esope]

Good idea. Implemented in the recently pushed changes.

[rgrinberg]

Could you add tests for this feature?

[esope]

@rgrinberg: with your recent commit b5ad3a39b8debb430654665b60433a3acf5b4410 refactor: implement progress with svar I encounter the following error

File "src/dune_engine/build_system.ml", line 214, characters 11-17:
File "src/dune_engine/build_system.ml", line 214, characters 11-17: Assertion
failed
Raised at Dune_engine__Build_system.State.start_rule_exn in file
  "src/dune_engine/build_system.ml", line 214, characters 11-23
Called from Dune_engine__Build_system.Exported.execute_rule_impl in file
  "src/dune_engine/build_system.ml", line 557, characters 51-76
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13
Re-raised at Stdune__Exn.raise_with_backtrace in file
  "otherlibs/stdune/exn.ml", line 36, characters 27-56
Called from Fiber.Scheduler.exec in file "src/fiber/fiber.ml", line 878,
  characters 10-13

I must not crash.  Uncertainty is the mind-killer. Exceptions are the
little-death that brings total obliteration.  I will fully express my cases. 
Execution will pass over me and through me.  And when it has gone past, I
will unwind the stack along its path.  Where the cases are handled there will
be nothing.  Only I will remain.

On line 214 of file src/dune_engine/build_system.ml, the variable current seems to be in the Initializing case.

My code used to work properly without these svar-related changes.

[esope]

Could you add tests for this feature?

This is pushed in the recent commits.

But it triggered another (minor) issue: I expanded the current test for dune describe so that it contains an example that uses a PPX rewriter. As a consequence, new dependencies to external libraries appear, and they refer to absolute path names (example here)... This will create spurious mistakes on every developers' machines, since these paths are likely to refer to their own home directories, and can even differ when tests are run with two different versions of the OCaml compiler...

Do you see an easy solution to this issue?

[rgrinberg]

@esope thanks for the report. I'll fix this issue shortly.

[rgrinberg]

Do you see an easy solution to this issue?

Honestly, there's no easy solution. The obvious thing to do is to add the paths to BUILD_PATH_PREFIX_MAP, but unfortunately the formatting is going to depend on the unfiltered length which still makes it not completely reproducible.

How about just writing a simple pre-processor that will redact the fields that aren't reproducible?

[esope]

How about just writing a simple pre-processor that will redact the fields that aren't reproducible?

I'll try this solution. I've also noticed that since the UIDs depend on the paths, they are subject to instability too...

[esope]

How about just writing a simple pre-processor that will redact the fields that aren't reproducible?

I'll try this solution. I've also noticed that since the UIDs depend on the paths, they are subject to instability too...

I'm trying to write a simple executable that reads the output of dune describe on stdin, parses it, sanitizes it to normalize the non-reproducible parts, and prints its result on stdout. This executable will be part of the test/blackbox-tests/test-cases/describe.t directory.

To do this, I need to parse the contents of stdin to produce something akin to a S-expression, for example using the functions of dune's dune_lang library. It seems I cannot access this library at all from the test/blackbox-tests/test-cases/describe.t directory. @rgrinberg: do you have any idea on how to to this?

An alternative path would be to add a flag to dune describe workspace, that would enable the sanitization before printing, but running dune's internal tests would be this flag's sole purpose...

[rgrinberg]

To do this, I need to parse the contents of stdin to produce something akin to a S-expression, for example using the functions of dune's dune_lang library. It seems I cannot access this library at all from the test/blackbox-tests/test-cases/describe.t directory. @rgrinberg: do you have any idea on how to to this?

The easiest way is to introduce a subcommand to dune_cmd. Have a look at how that executable is defined.

An alternative path would be to add a flag to dune describe workspace, that would enable the sanitization before printing, but running dune's internal tests would be this flag's sole purpose...

If you think this is simpler, it is fine by me.

[esope]

An alternative path would be to add a flag to dune describe workspace, that would enable the sanitization before printing, but running dune's internal tests would be this flag's sole purpose...

If you think this is simpler, it is fine by me.

Done in commit 72cdde1.

[rgrinberg]

CI isn't happy:

Run opam exec -- make dune.exe
[11](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:11)
ocamlc -output-complete-exe -w -24 -g -o .duneboot.exe -I boot unix.cma boot/libs.ml boot/duneboot.ml
[12](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:12)
./.duneboot.exe
[13](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:13)
cd _boot && /home/runner/.opam/4.12.1/bin/ocamldep.opt -modules -map main__.ml -map dune_rpc_impl.ml -map build_info__.ml -map dune_rules__.ml -map dune_engine__.ml -map async_inotify_for_dune__.ml -map ocaml_inotify.ml -map dune_rpc_private__.ml -map dune_meta_parser.ml -map dune_action_plugin__.ml -map dune_glob__.ml -map dune_re__.ml -map dune_cache.ml -map dune_cache_storage__.ml -map dune_util__.ml -map memo__.ml -map dag__.ml -map incremental_cycles__.ml -map dune_lang__.ml -map dune_graph__.ml -map stdune__.ml -args source_files
[14](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:14)

[15](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:15)
File "bin/describe.ml", line 838, characters 28-30:
[16](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:16)
Error: Syntax error: 'end' expected
[17](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:17)
File "bin/describe.ml", line 8[18](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:18), characters 17-23:
18
  This 'struct' might be unmatched
[19](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:19)
make: *** [Makefile:48: dune.exe] Error 2
[20](https://github.com/ocaml/dune/runs/5161075121?check_suite_focus=true#step:8:20)

[esope]

CI is much happier now.
I had to reimplement String.ends_with, that is only available in recent version of the standard library. @rgrinberg: I wrote the implementation in describe.ml, but do you think of a better module where this implementation should be?

[rgrinberg]

Merged but I simplified the sanitation quite a bit. It's not as robust, but good enough for our testing needs.