Web Version

[3dik]

I used your EPUB code for implementing a usable web version of the manual. My main motivation was the rigid layout of the PDF doc. This web version always tries to adapt its layout to the screen and font size. In most cases, the content should be well readable on any device. The makefile also separates all sections from each other. Together with the navigation elements, which were inspired by the "page per node" GNU manuals, these changes hopefully make navigating through the docs much comfortable.

Besides that, a very basic version of the specification chapter is added. This is may be the biggest, remaining task. Instead of formatting all those API data, the GitHub header file of each referenced element is just hyperlinked. However, since the readers would visit the web manual with their browsers anyway, this might actually be an acceptable solution.

But the commits may not be ready for a merge:

• I used many, possibly error-prone hacks. For example, if one of your section names contained a single quote, it would break the whole build process.
• Some parts make heavy use of gosh internals. Redesigning parts of gosh could simplify the build code.
• The EPUB generation might be partially broken now. The EPUB file still works when opened with mupdf. But its images are not displayed on my PocketBook.
• There are still a few minor HTML bugs:
  • footnotes do not work, e.g., in subsection "Region maps"
  • no subscripts, see "Capability delegation through capability invocation"
  • underscores instead of italic style, see "Linker scripts"
  • \clearpage, e.g. after various subsections in "Inter-component communication"

If you are not willing to merge yet, these commits might serve as a basis for a cleaner implementation. Unfortunately, I can't put more time into this for a while. However, feel free to ask me if you have any questions.

Live demo
Code, see the readme for building instructions.

[nfeske]

Wow! That's fantastic! I had an HTML version of the book in the back of my head for a long time but could not find the time to actually do it. Thank you very much for putting so much time into that. The result - including the navigation - is excellent.

I very much appreciate your effort with explaining the mechanism (README, inline comments, and this issue) and straight-away pointing out the current limitations. In my opinion, the limitations are completely acceptable. Even if not perfect, your current solution thousand times better than having no HTML version at all. ;-) It makes the material so much easier accessible.

I immediately gave the make splitted a try. Unfortunately, the makefile fails with the definition of the all_sections variable, which tries to read files that do not exist initially. In general, the parsing phase of make (where the variables are assigned) cannot depend on files generated by the rules (a later phase of make's execution). So we need to find a way to untangle this dependency. Let me think about it. Apart from this problem there are a few things that I would like to change, but nothing fundamental. Regarding the merge, how would you prefer to proceed? Given we manage to fix the dependency issue above, I could merge your branch as is and make adjustments via subsequent commits. Alternatively. I could propose fixup commits to your existing commits so that the final history would contain the final form. Which approach would you like more?

I used many, possibly error-prone hacks. For example, if one of your section names contained a single quote, it would break the whole build process.

Since the hacks are single-purpose, this is completely fine. After all, it is up to us to side-step such limitations, i.e.., by not using single quotes in section names. It's good to know the limitations though.

Redesigning parts of gosh could simplify the build code.

Please do not hesitate to suggest improvements to gosh.

From looking at the patches, I get the feeling that the section/chapter splitting could benefit most from slight improvements of gosh. E.g., with a style file that allows the filtering of sections and chapters, we could potentially overcome the need for the sed/sh-based mechanism. This may be useful for other uses of gosh too.

The EPUB generation might be partially broken now. The EPUB file still works when opened with mupdf. But its images are not displayed on my PocketBook.

I'm just guessing. Maybe this stems from the introduction of the basepath variable? With your patch, the <img> src is generated as $basepath/$tikz_name whereas the original version just used $tikz_name. If basepath is "" (default), this results in a path with a leading slash. Maybe it would be sensible to move the / into the basepath variable?

\clearpage, e.g. after various subsections

The remaining occurrences of such LaTeX markup should definitely be replaced with something similar to [raw spec/clearpage] (including a file that contains only the command, so we can simply supply a different file for the HTML version). So that is not a limitation of the mechanism but rather of the text.

[3dik]

I'm glad you like it!

I immediately gave the make splitted a try. Unfortunately, the makefile fails with the definition of the all_sections variable, which tries to read files that do not exist initially.

This surprises me. According to the GNU make manual, the file function should not exit with an error when reading from a non-existing file, but return an empty string. Does it abort at the definition of the all_sections variable?

I was aware of the dependency issue you described, which is why I made use of the makefile reloading mechanism by including the empty makefile sectionlists_update. If the empty makefile does not exist, make should create it with the rule I've specified. The section lists are prerequisites of that rule. After the empty sectionlists_update file has been created, make should reload the complete "standard" makefile. Thus, even if the directory is cleaned, a single make splitted command should build everything successfully. I admit, my makefile is a mess. I'm just wondering why it doesn't work.

However, the alternative approach which you have commited is much cleaner. Note, that I also used that inclusion mechanism for the conversion of the API class index. As an alternative, we could outsource that conversion into a separate makefile together with a include guard much simpler than mine, see MAKECMDGOALS.

[...] I could merge your branch as is and make adjustments via subsequent commits. Alternatively. I could propose fixup commits to your existing commits so that the final history would contain the final form. Which approach would you like more?

Currently, I couldn't review the commits in a careful manner. So it might be easier for you to merge to master. But do however you like. :)

Redesigning parts of gosh could simplify the build code.

Please do not hesitate to suggest improvements to gosh.

From looking at the patches, I get the feeling that the section/chapter splitting could benefit most from slight improvements of gosh. E.g., with a style file that allows the filtering of sections and chapters, we could potentially overcome the need for the sed/sh-based mechanism. This may be useful for other uses of gosh too.

I think so too. This might also make escaping special characters in section names without breaking the xref code easier.

Besides that, I see at least three sections of code which may benefit from something like generic, maybe tree-like accessor functions for stuff like xrefs:

• the navigation code of this web build
• the toc generation for the epub build
• the toc generation of the original html.gosh

The EPUB generation might be partially broken now. The EPUB file still works when opened with mupdf. But its images are not displayed on my PocketBook.

I'm just guessing. Maybe this stems from the introduction of the basepath variable? With your patch, the <img> src is generated as $basepath/$tikz_name whereas the original version just used $tikz_name. If basepath is "" (default), this results in a path with a leading slash.

Since the HTML files are in the root of the EPUB file, I thought a leading slash would be no problem. But I'll try to find the bug in the next days.

Maybe it would be sensible to move the / into the basepath variable?

I usually append such slashes within the code since it makes concatenating multiple path variables a bit easier. Is this bad practice? If it isn't and if the leading slash really is the problem, setting the default basepath to "." might be a solution.

The remaining occurrences of such LaTeX markup should definitely be replaced with something similar to [raw spec/clearpage] (including a file that contains only the command, so we can simply supply a different file for the HTML version). So that is not a limitation of the mechanism but rather of the text.

I only removed these file inclusions because my spec workaround didn't need them. But I agree. Especially if you want to print the API in HTML some day, those inclusions would make sense.

[nfeske]

In commit efea69a, I have untangled the make magic. The commit separates the www version from the epub version by using a different directory (manual/www). The Makefile in this directory creates the content by invoking your tools. I simplified the makefile by using different directories for the extracted texts (the chapter index texts go to chapters/ and the section texts go to sections/). The makefile also avoids the intermediate section-list file. This information is just kept in a make variable. Your tools are still accessed from the epub directory. In the final version, the web-related files (like the splitter tool) should better be moved to manual/www. @3dik Do you agree with this direction?

The new makefile is not fully complete yet. In particular the handling of the functional specification is missing. I hope that I haven't missed anything other important. I can happily browse the manual, images are displayed, and the navigation works nicely. :-)

Does it abort at the definition of the all_sections variable?

Yes. The error looks as follows (my make version is 4.1) :

Makefile:36: *** Invalid file operation: <splitted_txt/introduction.txt.  Stop.

But since the reworked makefile avoids the circular dependency, we don't need to further investigate.

But do however you like.

In my opinion, the best way would be squashing the final version as one commit with you as the author. This way, the commit history contains no intermediate noise that would confuse a reviewer.

I usually append such slashes within the code since it makes concatenating multiple path variables a bit easier. Is this bad practice?

Not at all. In general I also prefer it this way. In this specific case, it just had an unintended effect. Your suggestion of using '.' as basepath might work. But the resulting html code would look a bit strange. To attain html code that is tidy and neat, it's probably the best to regard the basepath as a link prefix - in the sense of a plain string that is concatenated. After all, this is what it is used for.

Besides that, I see at least three sections of code which may benefit from something like generic, maybe tree-like accessor functions for stuff like xrefs:

Thank you for the valuable pointers. As another point, it would be nice to eliminate the need for the fake chapter. But let's save that for another day. ;-)

[3dik]

In the final version, the web-related files (like the splitter tool) should better be moved to manual/www. @3dik Do you agree with this direction?

Absolutely.

I hope that I haven't missed anything other important. I can happily browse the manual, images are displayed, and the navigation works nicely. :-)

It looks almost perfect. I see only two possible, minor problems:

• The "make img/foobar.img" calls take more time now even if the images have already been generated. This is probably due to the generation of the section list at runtime, but I have a slow machine. So it may not be relevant to most people.
• Personally, I think removing the anchors from links to the first headline of a page would be nice. At least during the development of the navigation stuff, I found all that scrolling to the top quite annoying.

Does it abort at the definition of the all_sections variable?

Yes. The error looks as follows (my make version is 4.1) :
[...]

The version was probably the problem.

I usually append such slashes within the code since it makes concatenating multiple path variables a bit easier. Is this bad practice?

Not at all. In general I also prefer it this way. In this specific case, it just had an unintended effect. Your suggestion of using '.' as basepath might work. But the resulting html code would look a bit strange. To attain html code that is tidy and neat, it's probably the best to regard the basepath as a link prefix - in the sense of a plain string that is concatenated. After all, this is what it is used for.

You were right, the slash prefix was indeed the reason behind the missing images on my PocketBook. I fixed this and removed all dynamically added slashes from the code. I also pushed two other, minor commits to my branch.

[nfeske]

Thank you very much for the super quick response and for the patches!

The "make img/foobar.img" calls take more time now even if the images have already been generated. This is probably due to the generation of the section list at runtime, but I have a slow machine. So it may not be relevant to most people.

It's also slow on my machine. I fixed this now in aff3826. My solution is admittedly a bit hacky. But so is the nested invocation of make by epub_html.gosh. ;-)

Personally, I think removing the anchors from links to the first headline of a page would be nice. At least during the development of the navigation stuff, I found all that scrolling to the top quite annoying.

I agree. In 8a1070e, I added your nice sed command to accomplish this.

With the new version, I also moved the www supplements (splitter and style.css) from the epub/ to the www/ directory.

For now, the only feature I see missing is the recovery of the functional specification. Once this is done, we may revert the epub/Makefile to its original state. (for now, I kept the content for reference while working on the www/Makefile)

[nfeske]

With the commits above, I added the functional specification to the new makefile, updated the README, and reverted the changes of the epub/Makefile that are no longer needed. I went for adding the generation of class_index.mk as a manual step. This way, the www/Makefile has no side effects outside the www/ directory.

@3dik if you approve, I will squash the entire commit series into one commit, and add it to master with you as author. (I think the intermediate states are not important to preserve but make the history more confusing as it needs to be) Would that be fine for you?

[3dik]

I haven't checked out your changes, but they make sense, I guess. So feel free to squash.

[nfeske]

The web version is online now: https://genode.org/documentation/genode-foundations/19.05/index.html

Thanks @3dik for your initiative and your great work with the HTML version!

Thanks @chelmuth for the CSS styling!