Ocamlbuild cannot resolve dependencies when there are no _tags file (but can with an empty _tags file)

[vicuna]

Original bug ID: 5851
Reporter: Matthieu Lemerre
Assigned to: @gasche
Status: closed (set by @xavierleroy on 2016-12-07T10:34:34Z)
Resolution: fixed
Priority: normal
Severity: minor
Version: 3.12.1
Target version: 4.02.0+dev
Fixed in version: 4.02.0+dev
Category: -for ocamlbuild use https://github.com/ocaml/ocamlbuild/issues

Bug description

There is a difference in behaviour between no _tags file and an empty one. Ocamlbuild succeed in building a simple project only if there is an empty _tags file in the directory.

Steps to reproduce

First create a project with the following files:

File: plu/bla.ml
#+begin_src ocaml
let bla = 3;;
#+end_src

File: plu/bli.ml
#+begin_src ocaml
let bli = Bla.bla;;
#+end_src

File: plu.mlpack
#+begin_example
plu/Bla
plu/Bli
#+end_example

File: use.ml
#+begin_src ocaml
Plu.Bla.bli
#+end_src

Try to compile:

#+begin_example
ocamlbuild use.byte
/usr/bin/ocamlc -c -o use.cmo use.ml

• /usr/bin/ocamlc -c -o use.cmo use.ml
  File "use.ml", line 1, characters 0-11:
  Error: Unbound module Plu
  Command exited with code 2.
  #+end_example

It seems that the ocamlbuild solver does
not know how to build the =.cmo= out of the =.mlpack=:

#+begin_example
ocamlbuild plu.cmo
Solver failed:
#+end_example

However, if you create (even an empty) =_tags= file at the root of the
project, then everything works:

#+begin_example
ocamlbuild use.byte
/usr/bin/ocamldep -modules plu/bla.ml > plu/bla.ml.depends
/usr/bin/ocamldep -modules plu/bli.ml > plu/bli.ml.depends
/usr/bin/ocamlc -c -I plu -o plu/bla.cmo plu/bla.ml
/usr/bin/ocamlc -c -I plu -o plu/bli.cmo plu/bli.ml
/usr/bin/ocamlc -pack plu/bla.cmo plu/bli.cmo -o plu.cmo
/usr/bin/ocamlc -c -o use.cmo use.ml
/usr/bin/ocamlc plu.cmo use.cmo -o use.byte
#+end_example

[vicuna]

Comment author: @xclerc

It is arguably a design choice. As of today, the subdirectories
of the current directory are marked as "traverse" iff there is
either a "_tags", or a "myocamlbuild.ml" file.

Of course, this default behavior could as well be changed.
I am pondering the implications of such a change over
existing projects. Any advice is welcome.

[vicuna]

Comment author: Matthieu Lemerre

I was not aware of this "traverse" tag, this solves my problem. But this behavior is quite surprising: adding _tags file seems unrelated to the notion of traversing subdirectories.

I am not sure of how "traverse" interacts with the rest of the system, or what is its purpose. Is this only to improve performance by avoiding directory traversal?

It seems that directories added by the -I flag are traversed (at least, "ocamlbuild -I plu use.byte" works in my case), so I think the .mlpack case is comparable: at least the directories present in the .mlpack files should be traversable (or a warning/error should be issued if a referenced directory is not traversable).

[vicuna]

Comment author: @xclerc

I was not aware of this "traverse" tag, this solves my problem. But this behavior is quite surprising:
adding _tags file seems unrelated to the notion of traversing subdirectories.

I do agree that is seems strange at first, and think it would be a good thing to
change that. I am just afraid of breaking something...

I am not sure of how "traverse" interacts with the rest of the system, or what is its purpose. Is this
only to improve performance by avoiding directory traversal?

That is also my understanding.

It seems that directories added by the -I flag are traversed (at least, "ocamlbuild -I plu use.byte"
works in my case), so I think the .mlpack case is comparable: at least the directories present in the
.mlpack files should be traversable (or a warning/error should be issued if a referenced directory is
not traversable).

OK, fixing the problem this way could indeed be better.

[vicuna]

Comment author: meyer

This needs to be triaged again.

[vicuna]

Comment author: william

from the documentation :

camlbuild used to traverse by default any subdirectory not explicitly excluded. This is no longer the case. Note that you can still have a fine grained control using your _tags file and the traverse tag.

There is no longer the true: traverse tag declaration by default. To make ocamlbuild recursive use one of these:
Give the -r flag to ocamlbuild.
Have a _tags or myocamlbuild.ml file in your top directory.

The reason is given in gallium website :
The problem with this default setting (true:traverse) is that when starting, ocamlbuild wants to read deeply the whole current directory. For instance running ocamlbuild -documentation in your home could take a while.

• There is the question of having traverse or not by default
• but the behaviour where creating a _tags file makes ocamlbuild recursive is bad IMHO

[vicuna]

Comment author: william

I see two options :

Either set the "traverse" by default, even if no _tags or myocamlbuild.ml file is present.

• it is already what happens for anyone having a _tags or myocamlbuild.ml file, that is to say 95% of people
• it works "out of the box"
• the drawback is that it is most likely ocamlbuild would not be setup in an optimal way, as user would not know that ocamlbuild would traverse all files

Either forbid the traverse by default, even if _tags or myocamlbuild.ml file is present. And add a message when ocamlbuild fails, along with the error message and the backtrace, that indicates which directories are traversed.

• it would break many user configurations
• it would require most people to add a "true:traverse" tag to their project
• but it would ensure people use ocamlbuild in a more optimal way

[vicuna]

Comment author: @gasche

I don't think it is reasonable to change the default behavior. What we can do, however, is try to detect this edge case and try to provide an explanation to the user.

I just implemented this idea is a temporary branch
https://github.com/gasche/ocaml/tree/ocamlbuild-messages
commit
gasche@1bf5fc5

The example given in this bug report behaves in the following way:

% ../ocamlbuild.byte use.byte
Warning: Failed to build the module Plu requested by ocamldep.
Hint: Recursive traversal of subdirectories is not enabled by default as this
does not look like an ocamlbuild project (no _tags or myocamlbuild.ml
file). If Plu is in a subdirectory, you should add the option -r or create
an empty _tags file.

• /home/gabriel/.opam/4.02.0local+git-4.02/bin/ocamlc -c -o use.cmo use.ml
  File "use.ml", line 1, characters 0-11:
  Error: Unbound module Plu
  Command exited with code 2.
  Compilation unsuccessful after building 2 targets (1 cached) in 00:00:00.

Do you think the error message is clear enough?

(I know hints can be annoying in case of false positives. But I think it is a reasonably safe heuristic to assume that when people run ocamlbuild from a directory without a _tags or myocamlbuild.ml, they can tolerate getting beginners-oriented hints.)

[vicuna]

Comment author: @gasche

"Fixed" in 4.02@15111. I changed the Hint message a bit from the above draft -- the new version is included at the end of this message.

It is only fixed insofar as users now get a better explanation of the problem. The initial expectation of the reporter (that we could detect this situation and fix it without the user intervention) is not realized, but I'll still mark the bug resolved, as I don't think it is reasonable to try to do much more.

Hint: Recursive traversal of subdirectories was not enabled for this build,
as the working directory does not look like an ocamlbuild project (no
'_tags' or 'myocamlbuild.ml' file). If you have modules in subdirectories,
you should add the option "-r" or create an empty '_tags' file.

To enable recursive traversal for some subdirectories only, you can use the
following '_tags' file:

  true: -traverse
  <dir1> or <dir2>: traverse

[vicuna]

Comment author: @protz

While trying to build camlp4 on windows, I got this error message printed about 30 times at the end of the ocamlbuild invocation. Can you add an extra check to make sure this message is printed at most once at the end of each ocamlbuild invocation?