Menhir support in jbuilder?

[grayswandyr]

I am trying to set up jbuilder to run Menhir in various ways on several files. Namely, I have two parsers: Parser.mly and SMV_trace_parser.mly. They are unrelated but located in the same directory. However, for SMV_trace_parser.mly, I rely on the Menhir option of defining the (parsing) tokens in another Menhir file called SMV_trace_tokens.mly.

Hence I added the following rules (using only the menhir rule did not work, even with flags added when necessary):

    (menhir
    ((flags (--infer --explain))
     (modules (Parser))))
     
    (rule
      ((targets (SMV_trace_tokens.ml SMV_trace_tokens.mli))
       (deps    (SMV_trace_tokens.mly))
       (action  (chdir ${ROOT} (run ${bin:menhir} --infer --explain --only-tokens ${<})))))
    	 
    (rule
      ((targets (SMV_trace_parser.ml SMV_trace_parser.mli))
       (deps    (SMV_trace_parser.mly SMV_trace_tokens.mli SMV_trace_tokens.ml))
       (action  (chdir ${ROOT} (run ${bin:menhir} --infer --explain --external-tokens SMV_trace_tokens ${<})))))

When running jbuilder, I get the following error:

    jbuilder build @DEFAULT @install
    Multiple rules generated for _build/default/src/SMV_trace_tokens.mly

[rgrinberg]

Would you mind putting up a reproducible example?

[grayswandyr]

So, I was trying to devise an MWE, but the error I get is not even the one cited above.

I'm used to Ocamlbuild which can deal very easily with grammars spanning many files and where some files only define (syntactic) tokens and others only use them (options --only-tokens and --external-tokens), all together the important --infer option.

So as a first example, I don't even see how to compile with jbuilder the demos/calc-two/ example from the Menhir distribution (Even compiling without Ocamlbuild is quite hard.)

[grayswandyr]

Hi, my project is now released publically... and I don't see at all how to compile it with dune. As I (and others elsewhere) said, Menhir has an --infer tag that uses ocamlc to infer types (and this tag is highly advised by Menhir's authors); besides tokens may be defined outside of the mly files (and I think it's quite common) as the SMV_xxxx files show.

[ghost]

I had a look at demos/calc-two and I was able to build it with the following jbuild file:

(jbuild_version 1)

(executable
 ((name calc)))

(ocamllex (lexer))

(menhir
 ((flags   (--only-tokens))
  (modules (tokens))))

(menhir
 ((merge_into algebraic)
  (flags      (--external-tokens Tokens))
  (modules    (tokens algebraic common))))

(menhir
 ((merge_into reverse)
  (flags      (--external-tokens Tokens))
  (modules    (tokens reverse common))))

Regarding the --infer flag, I initially made the assumption in jbuilder that we could always split OCaml compilation into the following phases:

1. pre-processing and other source code generation
2. dependency computation
3. compilation (including typing)

In fact, these 3 steps are always required in this order; to compile a .ml file we need to have compiled its dependencies first, and to know its dependencies we need the final source code.

However this doesn't work when step 1 needs to use the typer. This is the case of menhir --infer and a few ppx rewriters such as ppx_import. This obviously create a circular dependency. For the case of menhir, we can break the cycle with menhir --raw-depend ... which gives us step 2 before step 1 . This requires a bit of refactoring. For the case of ppx_import, I'm a bit puzzled how it works.

[grayswandyr]

Thanks I will have a look at this, then. I'd love to switch my project to Dune but my experiments until now have failed. I'll see if I can succeed now.
Cheers from Toulouse ;-)

[grayswandyr]

All right, so I can build my project (in this branch), but I'm not sure I got all your explanations (and I'm pretty sure my jbuild file is non-conventional: I had some trouble having ppx_deriving, ppx_blob and visitors all work; as well as disabling several flags at once), in particular regarding --infer: is it still possible to use this flag?

[ghost]

No, it's not possible to use --infer. Some work would need to be done on jbuilder to allow it. Do you have examples where using --infer is better for the user?

[grayswandyr]

Well I don't have an example at hand where --infer is essential but thanks to this flag, errors are traced back to the mly grammar file rather than the huge generated ml file. It's also used to deal with so called inline rules, in Menhir, that are quite common.

[ghost]

Ok. I noticed there is another ticket about --infer specifically (#305), we can continue the discussion there. I assume that except for --infer, this issue is sorted?

I had some trouble having ppx_deriving, ppx_blob and visitors all work; as well as disabling several flags at once

yes, it's a bit painful at the moment. Switching ppx_blob to ocaml-migrate-parsetree should allow to use a more standard (preprocess (pps (ppx_visitors ppx_blob))) field.

[grayswandyr]

Yeah sure thank you.