topkg publish dies with ocamlbuild plugins

[pqwy]

If there are magic command line arguments necessary to get myocamlbuild.ml to compile -- like it referencing extra modules given by -plugin-tag -- topkg publish will bomb out because doc generation doesn't reuse the build command line.

With the up-and-coming glorious era of OCamlbuild plugins published, and distributed, on a massive scale, this should be re-thought.

[dbuenzli]

Yes it's because of the ad-hoc support to publish docs (see #24). topkg publish doc dies because it tries to invoke ocamlbuild by itself rather than try to reuse the ocamlbuild invocation you may have specified in your pkg.ml file (since topkg is build system agnostic).

A work around it to simply replace topkg publish by topkg publish distrib and proceed with the release process.

[dbuenzli]

@pqwy mentions it should just be a matter of adding the -no-plugin line to that invocation.

[pqwy]

Yeah, #81 does it.

It's a bit hacky. But not terribly, given how doc building fits in with the rest.

[dbuenzli]

Fixed by 9582ee0

[samoht]

This fix seems to break doc generation when you have a custom myocamlbuild.ml right? I don't see how add -no-plugin is helping with the original problem.

[samoht]

See #93 for a continuation of this PR.

[dbuenzli]

Why doesn't @pqwy's solution work ? I don't consider this mirage/ocaml-github@65c1a17 to be a solution. topkg doc should build the docs and that's all.

[samoht]

Which solution?

I don't understand what this PR is solving. With that commit in, I get:

$ topkg build
[ ... all works ... ]
$ topkg doc
 topkg doc
ocamlfind ocamlc -c -g -principal -strict-sequence -package 'cohttp uri cohttp.lwt-core yojson atdgen stringext tls' -package lwt -w @5@8@10@11@12@14@23@24@26@29 -I lib -o lib/github_s.cmi lib/github_s.mli
Solver failed:
  Ocamlbuild knows of no rules that apply to a target named lib/Github_t.ml. This can happen if you ask Ocamlbuild to build a target with the wrong extension (e.g. .opt instead of .native) or if the source files live in directories that have not been specified as include directories.
Backtrace:
[...]
topkg: [ERROR] run ['ocamlbuild' '-classic-display' '-use-ocamlfind'
               '-no-links'
                    '-no-plugin' '-build-dir' '_build' '-docflags'
                    '-colorize-code,-charset,utf-8'
               'doc/api.docdir/index.html']: exited with 6

topkg doc works fine if I remove the -no-plugin flag, either by reverting that PR or by applying #93 + --build-flags ''.

[samoht]

It also works when using -just-plugin instead of -no-plugin. Not sure what this option does exactly. I'll try to see if I can repro @pqwy's case.

[samoht]

Hum ok I understand the initial problem better now. My issue si that I need to read (and compile) myocamlbuild.ml to create the implicit rules for the generated .mli files. A solution is to run the ocamlbuild command in the build directory instead of running it at the toplevel, so all the generated files are there (with -no-hygiene). Would it be something that you could consider including? I don't know how to fix that issue without passing build flags to ocamldoc.

[samoht]

Last edit: actually if _build/myocamlbuild exists, calling _build/myocamlbuild instead of ocamlbuild seems to work.

[pqwy]

We have a total, head-on collision of requirements here.

I require additional ocamlfind libs in my myocamlbuild.ml. The only way to add libs to its compilation is to pass extra arguments to ocamlfind (-plugin-tag, see the various discussions around why _tags doesn't come into play). Since the doc-command is ad-hoc, in that is does not derive from the build-command, it does not share the extra arguments, so ocamlbuild bombs out in the first step -- compiling its plugin. A way around that is to completely ignore myocamlbuild. -no-plugin does just that, and is an argument that ocamlbuild internally uses when invoking itself, as compiled from myocamlbuild, to stop the cycle.

Use case is anything that uses ocb-stubblr.

You have extra generated files, and the rules to build them are registered in your myocamlbuild.ml. Clearly, if one skips the plugin, the rules to materialize a bunch of files are missing.

Use case is really any custom rules for generating ml files, which are in turn input for the doc step.

This change completely and comprehensively disabled myocamlbuild.ml processing during doc generation.

A way out of the stalemate is to actually rethink how the doc-command should relate to the build-command, while keeping topkg's semi-ocamlbuild-agnostic structure. Whoever comes up with a good redesign of how topkg derives its doc-command gets to fix this, I guess. A telltale mark of a good redesign is that you don't have to repeat things in pkg.ml. Another one is that you don't have to pass empty arguments to anything.

[samoht]

@pqwy but can't you just build before building the docs? e.g. in your case, topkg build && topkg doc should just work right? The first invocation will build _build/myocamlbuil, the second one will just re-use that.

[dbuenzli]

So I discussed this with @pqwy in person and I'll simply try to reuse the build command specified in the pkg/pkg.ml file which should make it work for everyone.

[samoht]

Any update on this? It is unfortunately not possible to generate docs with 0.8.1 if using a myocamlbuid.ml file. Not sure what is the recommended way to workaround that: use odig and do some magic during topkg publish?

[dbuenzli]

@samoht @pqwy could you please confirm me that the new topkg doc works for both of you by trying with:

opam pin add --dev-repo cmdliner
opam pin add --dev-repo topkg 
opam pin add --dev-repo topkg-care

[samoht]

@dbuenzli I confirm that it works well on https://github.com/samoht/ocaml-github/tree/topkg. Thanks!