Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

odoc should drive itself #137

Open
aantron opened this issue Apr 25, 2018 · 8 comments

Comments

Projects
7 participants
@aantron
Copy link
Contributor

commented Apr 25, 2018

Building odoc's output requires calling the odoc binary many times, about twice per .cmti file in a given project, or in a set of installed packages:

odoc compile [OPTIONS] foo.cmti
odoc html [OPTIONS] foo.odoc
# For each .cmti file

In addition, the commands have to be issued in a correct order, determined by the OCaml dependencies between the .cmti files.

This has several negative effects:

  • Both odig and Dune currently have a bunch of logic for issuing these commands correctly. It is difficult to also add it to, say, bsb. Humans can't use odoc directly.
  • Adding a new output target, like LaTeX or Docusaurus, cannot be done only in odoc. We have to do non-trivial work in other projects that will drive odoc.
  • Exposing the structure of the output to the driving program makes it difficult to change this structure in odoc. In practice, this has such effects as we can't very freely edit the URL structure of references. odoc is currently coded with fairly strong assumptions about where everything will be, and that the output is, indeed, HTML.
  • Both odig and Dune currently generate module and package index pages. odoc should do that.

The information needed for odoc to drive itself is mostly already contained in the .cm[t][i] files. So, I suggest that we add a new command, say odoc build, that will do so. It may need some arguments, like a mapping between directories and package names. .odoc files will become a build cache that odoc manages internally. I think the logic will end up something very similar to what is now in odig.

cc @dbuenzli @rgrinberg @Octachron

@aantron aantron self-assigned this Apr 25, 2018

@dbuenzli

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2018

I'm a little bit unsure about all this; also you are conflating many different things in the same issue which deserve an orthogonal treatement (e.g. indexes).

You basically want to turn odoc into a build system which doesn't seem to be a good idea as it will be driven or need to be integrated in larger build systems. So you'll likely duplicate work or make it less build system friendly.

What should be done is to make it more build system friendly and one important point for this would be to solve #81 as -I is a notorious problem for build systems (they entail that you need to track negative information about the state of the file system). One big odoc build command that does a lot of magic will likely be a pain for build system integration.

@trefis

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2018

I completely agree with @dbuenzli .

@lpw25

This comment has been minimized.

Copy link
Contributor

commented Apr 25, 2018

I agree mostly agree with @dbuenzli, although I do think there is room for having a simpler command for letting users run odoc themselves in simple situations. As long as the original compile and link commands still exist for build systems to use. Any serious build system wants tools that execute individual steps explicitly. Dealing with commands that do their own build system things just makes it very difficult to accurately specify the dependencies and gets in the way of the parallelism, incremental building, and caching that is the main job of the build system. The current structure of odoc was designed specifically to be easy to write build rules for -- that's why it has commands that output the dependencies and the targets of individual commands -- although I think these commands may not be up-to-date or fully exploited by the current consumers of odoc.

So, by all means make it easy for users to directly run odoc without needing to learn how it works with a command that handles things for them, but it will make life harder for the build systems if you take out the existing commands.

@ostera

This comment has been minimized.

Copy link
Collaborator

commented Sep 20, 2018

+1 to a higher level workflow without removing low level commands.

Why can't an odoc build command make use of the lower-level building blocks (compile, html, etc) to provide an opinionated but straightforward way of generating documentation without compromising interoperability with build systems?

As a casual user, I'd be able to just run odoc build and get my docs out there — very little friction.

As an expert user, I'd be able to track each compilation step of the documentation as I see fit — all the control.

@rgrinberg

This comment has been minimized.

Copy link
Member

commented Sep 20, 2018

Because a higher level workflow would require odoc to understand different types of project setups. Since the utility of such a mode is highly dubious already, it doesn't seem like this is worth pursuing.

@rizo

This comment has been minimized.

Copy link
Collaborator

commented Sep 20, 2018

First of all, to make sure everyone is on the same page let me just state the basics (bare with me please if this is too obvious):

  • Currently odoc is not a full-blown documentation generator. It's a compiler that operates on individual compilation units (i.e. .cmi/.cmti files), producing intermediate .odoc files and generating HTML trees from those.
  • Being a low-level tool that requires a build system to be executed for large projects, it's similar to gcc or ocamlc. Most reactions here seem equivalent to the reactions if one proposed teaching gcc how to compile "projects", which doesn't seem to make sense.
  • A typical OCaml/BuckleScript package consists of potentially multiple interdependent compilation units. With dune it can even be multiple sub-packages.
  • odoc does not understand the "package" concept. There's no such thing in OCaml really and different tools have their own ad-hoc interpretation.
  • Cross-referencing between compilation units (which might happen both in the same package and across packages) requires a dependency-aware compilation order and a correct package lookup configuration (the -I option in odoc compile).

Please correct me if any of these statements are wrong in any way. This is just my understanding constructed from fragmented discussions and source code.

@rizo

This comment has been minimized.

Copy link
Collaborator

commented Sep 20, 2018

I personally agree that the duplication of efforts between odig, dune and soon bsb, as well as lack of high-level view of the package, have negative consequences to the overall development process and design of user-facing features like package module index, search, alternative backends, etc.

To me it's less of a "who drives odoc" problem and more a "how to ensure a great user experience" problem.

Ideally, from the user perspective (and I understand this is out of odoc's scope, but where else should this be discussed?), we should try to produce an informative and organized package page (with all package versions) similar to the one produced by Hackage or Rust Docs. I've given these examples before, but please consider again how hard it would be to implement something like that with the current state of tooling. Sure, odig does produce nice package pages with metadata and a module list, but dune doesn't. Also how does OPAM's package pages relate to all of this? Do we want to further aggravate the user experience with bsb building it's own package index page?

I think that regardless of whether odoc will or will not "drive itself", many of these issues could be addressed with a "package" definition that would include module list (like the one generated here) and metadata. This would essentially allow odoc to generate the HTML for the index page in a consistent and independent way for all build environments.

@lpw25

This comment has been minimized.

Copy link
Contributor

commented Sep 20, 2018

I personally agree that the duplication of efforts between odig, dune and soon bsb, as well as lack of high-level view of the package, have negative consequences to the overall development process and design of user-facing features like package module index, search, alternative backends, etc.

I'm not sure how much duplication of effort there actually is. The main parts of writing build rules for odoc are build-system specific. They are about the relationship between OCaml's view of the world and the file system -- which is managed by the build system and associated tools (e.g. ocamlfind) -- and they are different in each of the systems that you mentioned.

Things like the module index aren't really about whether odoc "drives itself". I think those tools make there own index pages because they want to/ odoc doesn't provide a builtin command that does it. Although to a certain extent the .mld file stuff does provide the needed support. If we want all those tools to produce the same indices then we should provide a command that produces indices and encourage its use. This is completely orthogonal to whether odoc provides some high-level odoc build command or not.

@ostera ostera referenced this issue Oct 8, 2018

Open

[meta] Roadmap #210

2 of 13 tasks complete

@aantron aantron added this to Proposed in Interop via automation Oct 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.