Deploy docs to gh-pages branch from Travis builds

[khaledhosny]

Build docs in Travis and push them to the gh-pages branch, which makes
them available at http://behdad.github.io/harfbuzz/

Behdad, you probably want to change the encrypted token with one og your own after merging this, it can be generated with:

travis encrypt -r behdad/harfbuzz GH_TOKEN=…

[khaledhosny]

I’m doing this because a few days ago I was looking for the generated HarfBuzz documentation and couldn’t find it anywhere online.

[behdad]

This is amazing, thanks! Gives us a new hope for actually getting documentation written down.

[anthrotype]

nice!

[khaledhosny]

May be we can move the relevant bits from http://www.freedesktop.org/wiki/Software/HarfBuzz/ to the gtk-doc documentation, and make harfbuzz.org point to the gh-pages website to give it more visibility and hope for the best.

[behdad]

May be we can move the relevant bits from http://www.freedesktop.org/wiki/Software/HarfBuzz/ to the gtk-doc documentation, and make harfbuzz.org point to the gh-pages website to give it more visibility and hope for the best.

I'd love to see that.

[khaledhosny]

OK, if no one beats me to it, I’ll try to figure out how to edit the docs and move the relevant information there.

[khaledhosny]

I moved some parts from FreeDesktop site to the docs, let me know what do you think.

[behdad]

Khaled, I'm having problems running the travis cli tool. Can you tell me how to fix the token?

[behdad]

Ok, I managed to set this up.

Maybe we should set this to only upload on releases?

[khaledhosny]

It should be uploading only on tagged commits now (for tags that start with a digit), that is the closeset I can find to uploading on releases only.

[behdad]

The contents of docs can be copied into HarfBuzz wiki.

We should shut down the wiki and include everything in the doc.

[twardoch]

Intro

Am I correct to assume that currently, the HarfBuzz docs are generated using GTK-Doc, a rather un-maintained tool which is not very extensible?

Doxygen

Doxygen seems to be the more-less de facto standard for documenting C++ projects. However, its outputs are not very “pretty”. But Doxygen does support a large number of inline commenting styles and can parse C++ code with those comments, and create XML files, and from them HTML intended for final consumption.

Sphinx

Sphinx is a massive project originally developed for Python documentation. It uses reStructuredText (aka reST), which is a bit more complex than Markdown but also more powerful. Sphinx has a number of sophisticated plugins that can aid the production of the documentation, and supports a number of languages. Sphinx is the engine behind the very popular ReadTheDocs site and the popular ReadTheDocs theme can also be used for documentation hosted elsewhere.

Breathe

There is a project called Breathe (docs) which takes Doxygen’s XML output files and generates files that can be consumed by Sphinx.

PyGObject/PyGI

There are PyGObject Python bindings for GI introspection, and a similar pure-Python implementation called pgi intended mostly for PyPy, the alternative Python interpreter.

pgi-docgen

There is a pgi-docgen tool that creates Sphinx documentation for gi modules using Python introspection (either PyGObject or pgi). pgi-docgen.py introspects the gi module, pulls in the gir docs and creates a sphinx environment, and pgi-docgen-build.py builds html docs using sphinx.

There already exists a repo with a number of gi-enabled docs, including one for HarfBuzz.

pandoc

The very mighty pandoc tool is a converter than can, among others, take Markdown input and generate reST output.

So...

So, I wonder: even if it were a bit awkward, perhaps it might be worth considering:

1. To use the Harfbuzz Github wiki to prototype contents for human-written documentation, in Markdown.
2. If needed, to reformat the inline code comments from the Harfbuzz source code to be doxygen-compatible (perhaps they already are).
3. Use the doxygen-breathe toolchain to produce the "machine-generated" Sphinx-compatible reST for the Harfbuzz C++ API.
4. Use the pgi-docgen tool to produce Sphinx-compatible reST for the Harfbuzz Python GI API.
5. Use pandoc to convert the Markdown-written wiki into Sphinx-compatible reST.
6. Have a set of Sphinx config files that combine the reST produced by steps 3, 4 and 5 and output nice-looking documentation that would be deployed on gh-pages, and perhaps ultimately, on ReadTheDocs.

We could utilize a similar mechanism (without doxygen-breathe) for fontTools and other projects written in Python.

[twardoch]

The reason I'm proposing this: each tool is good in its own domain. I don't think authoring XML/SGML a la this is very exciting :) so I doubt we'll see a lot of contributions if hand-editing this XML is required. Doxygen is probably best at outputting API docs for C++, Markdown-in-Github wiki is best for authoring of hand-written narratives, and Sphinx generates the best-looking output. Plus, the Sphinx workflow will be useful for fontTools as well.

[khaledhosny]

You can use (subset of) markdown with gtk-doc for API documentation, https://developer.gnome.org/gtk-doc-manual/stable/documenting_syntax.html.en, so that is probably not the reason no one is writing any.

[davelab6]

Imho Github wiki are not as good as a regular github repo with the default branch a gh-pages branch containing markdown files.

[twardoch]

Dave,

I think the Github Wiki is pretty good (though far from perfect of course). It is a regular git repo just like any other Github repo. The web UI is a bit different, but it does include a commit history, both aggregate and per page. By design, the Github Wiki system has less restrictions and is therefore "faster" to edit. It also lacks certain features such as branches. In essence, it's a simplified UI for a git repo with Markdown files (and images, which you can upload via commandline git). The files within a Github repo can be placed in subfolders, though it's best to keep them all in the root folder of the wiki repo.

The only "wiki" feature that makes it different from the regular Markdown files is the "dynamic wiki" hyperlinking which, in case of the Github wiki, can be done using the [[Page title]] syntax. If the page has the title Editing OpenType features, then it will be stored as the file Editing-OpenType-features.md in the root of the repo, and can be linked to with [[Editing OpenType features]]. If you place a link [[Editing OpenType features]] in any document and then follow it, the Github wiki will automagically create an appropriately-named .md document. If such a document is later renamed, the link won't work until modified. Of course linking via the standard Markdown syntax [Editing OpenType features](Editing-OpenType-features.md) also works, but then the automagic document creation does not happen.

Could you briefly elaborate on why you think the Github Wiki is not as good as a regular wiki with markdown files?

[behdad]

Dude, keep it simple!

Whenever someone implements an alternative solution and integrates it with our autotools build system and TravisCI and sends a pull-request, I'll consider it. Until then, I think we have a system in place to deploy docs, and that's already 100 times more motivating than what we had before.

If you have had spent writing a page of doc about a page of comment about meta-tools used for generating docs, we would have been one page closer to being doc-complete ;).

[twardoch]

Well, I cannot contribute much to the docs proper since I know no C++. I know that my proposal ain't simple. The gist is that there should be two portions: a himan-edited "intro/manual", possibly coverong a few higher-level concepts, plus a machine-generated API docs. These two types of docs are different in nature and often one workflow is better for one type and another workflow for another. :)

[behdad]

The gist is that there should be two portions: a himan-edited "intro/manual", possibly coverong a few higher-level concepts, plus a machine-generated API docs.

We support both now, in an integrated setting, so let's stick to it and propose people to write docs.

You don't need to know C++ to contribute BTW.

[twardoch]

Yeah, I know :) I'm working on something ;)

[davelab6]

Adam,

I understand that github wikis are backed by git, but there are quite
unlike regular github repos - unless I missed a trick :)

For those with direct commit access, the Github website itself has an 'edit
this file' button (the pencil icon
http://michaeljaylissner.com/posts/2014/10/06/editing-on-github-a-non-technical-explainer/)
and prose.io provides a third party interface, so for those privileged
users like yourself, the wiki system has no more or less restrictions and
is therefore the same speed to edit.

Those without direct commit access, markdown files in a real github repo
offer them the typical fork/edit/pull-request workflow, which means that
quality control is made in an on-going basis and the upstream master is
always good - and, if part of the codebase, then versioned along with the
code at releases.

Whereas with a wiki, anyone has direct commit access and there is no
quality control, and there isn't a straightforward versioning to match wiki
contents to code releases.

[twardoch]

Indeed, that is a fair comparison. A Github wiki has fewer restrictions, has a "spam risk", allows for less control but requires less management and is friendlier towards those who aren't fluent in a git-based workflow, a Github repo with Markdown files requires more management and may be deterrent for those who don't understand e.g. what "rebase" means, but allows more control.

I personally think that both can be put into good use, and I think that the right choice can be made after weighing these arguments.

[twardoch]

Thanks for the prose.io pointer, I'll check it out!

[davelab6]

Fluency with github workflow is like neoliberalism. You will be assimilated.