Ubuntu 22.04 and 24.04 make build in p4-16/spec directory fails

[jafingerhut]

For current status of the possibility of using AsciiDoc as a replacement for Madoko in maintaing the source of the P4_16 language specification, see #1298

At least it failed as detailed below with Ubuntu 22.04 as it was on 2022-Jun-15 when I tested this. The steps below work fine on Ubuntu 18.04 and Ubuntu 20.04 systems, so the workaround is to use one of those OS versions until this problem is fixed.

This is probably an issue with one or more Ubuntu 22.04 packages, not with the files in this p4-spec repo. I am only creating this issue as a reminder to try this again in a year or so to see if it works, and to point other people at in case they try this and it fails for a similar reason.

Steps I tried:

Create a fresh minimal install Ubuntu 22.04 system as a VM for AMD64 architecture

git clone https://github.com/p4lang/p4-spec
cd p4-spec/p4-16/spec
# Edit the setup-for-ubuntu-linux.sh script to comment out the `exit 1` lines to enable the script to do its work on an Ubuntu 22.04 system, as a temporary hack
./setup-for-ubuntu-linux.sh
make

Here is the output I saw from the make command:

$ make
madoko --pdf -vv --png --odir=build P4-16-spec.mdk
process: P4-16-spec.mdk -> build/P4-16-spec.html

process: P4-16-spec.mdk -> build/P4-16-spec.tex
  P4-16-spec: rendering math.
  P4-16-spec: running latex on math...   (plain -> dvi -> png)
  P4-16-spec: rerun to resolve references...
  P4-16-spec: create image directory: build/math
  P4-16-spec: generating math images...  (plain) (1-2) (concurrency 4)
  > "dvipng" -T tight -z9 -bg Transparent -D300 -pp 1-2 -o "math/P4-16-spec-mat
  h-plain-%d.png" "P4-16-spec-math-plain"
  error: failure while typesetting math:
  This is dvipng 1.15 Copyright 2002-2015 Jan-Ake Larsson
  [1]
  dvipng warning: at (4294967295,34) unimplemented \special{dvisvgm:bbox 0.01 0
  .01} dvipng warning: at (109,34) unimplemented \special{dvisvgm:bbox 0.01 0.01} Floating point exception (core dumped)

[jnfoster]

In the interest of tidying up the set of active issues on the P4 specification repository, I'm marking this as "stalled" and closing it. Of course, we can always re-open it in the future if there is interest in resurrecting it.

[jafingerhut]

Sorry if this issue is considered clutter, but I have now tried to build the spec from Madoko source on Ubuntu 22.04 and also Ubuntu 24.04, and it fails on both of those. It still works on Ubuntu 20.04, but free support for that will end in 1 year, 2025-Apr.

It appears that perhaps Madoko is no longer being maintained by its original developer(s) (note that this issue was opened in 2019): koka-lang/madoko#60

Unless we want to do one of these things:

• kick the can down the road to 2030-Apr when 10-year support for Ubuntu 20.04 is over
• find someone who can update Madoko to work with more recent versions of other tools like LaTeX

then it seems we should consider using a different documentation tool chain for producing P4 specs, e.g. AsciiDoc and AsciiDoctor suggested in the Madoko issue linked above.

Note: I have no personal experience using AsciiDoc, so cannot personally recommend for or against it -- merely noting a recommendation given in 2019 by someone else who was switching from Madoko to something else.

[jnfoster]

Indeed, it seems like Daan has stopped maintaining Madoko. We should port the specs to some other Markdown + other stuff system, of which there are many.

…

On Sat, May 4, 2024 at 12:11 PM Andy Fingerhut ***@***.***> wrote: Sorry if this issue is considered clutter, but I have now tried to build the spec from Madoko source on Ubuntu 22.04 and also Ubuntu 24.04, and it fails on both of those. It still works on Ubuntu 20.04, but free support for that will end in 1 year, 2025-Apr. It appears that perhaps Madoko is no longer being maintained by its original developer(s) (note that this issue was opened in 2019): koka-lang/madoko#60 <koka-lang/madoko#60> Unless we want to do one of these things: - kick the can down the road to 2030-Apr when 10-year support for Ubuntu 20.04 is over - find someone who can update Madoko to work with more recent versions of other tools like LaTeX then it seems we should consider using a different documentation tool chain for producing P4 specs, e.g. AsciiDoc and AsciiDoctor suggested in the Madoko issue linked above. Note: I have no personal experience using AsciiDoc, so cannot personally recommend for or against it -- merely noting a recommendation given in 2019 by someone else who was switching from Madoko to something else. — Reply to this email directly, view it on GitHub <#1115 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOCFURJ3YVBJKZVPGTBKA3ZASX3XAVCNFSM5Y36UDPKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBZGQYTANRUGA4Q> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[jafingerhut]

I have started an experiment to convert the P4 language spec into AsciiDoc, and examining the PDF output as I convert one section at a time, to learn what changes are required. There are definitely changes required for figures, cross-references, section headings, and list items, but they are quick and easy to make once you know how. I do not have syntax highlighting set up yet for P4 code blocks, but AsciiDoctor (a tool for processing documents in AsciiDoc format) supports several syntax highlighting programs, at least one of which I think is the same one that we have been using for Madoko, so that should not take long to get working, I hope.

You can see the partial translation done here:

• https://github.com/jafingerhut/learn-asciidoc/tree/main/p4-spec-next

P4-16-spec.adoc is the partially translated source file.

This file is the notes I have been keeping on how to translate different syntactic elements: https://github.com/jafingerhut/learn-asciidoc/blob/main/p4-spec-next/README-madoko-to-asciidoc-notes.md

Amazingly enough, things like monospace font, emphasized text, superscript, subscript, are all the same, so no changes are required for those. Bold is just one asterisk surrounding the text to be bolded instead of 2 like in Madoko.

I have tested, and I can successfully install AsciiDoctor on Ubuntu 20.04, 22.04, and 24.04, both x86_64 and aarch64 processors.

[smolkaj]

That sounds promising. Thanks for looking into this!

[jafingerhut]

I have gotten further with my conversion of the P4 language spec to AsciiDoc, at the same repo linked earlier:

• https://github.com/jafingerhut/learn-asciidoc/tree/main/p4-spec-next

The look-and-feel of the generated PDF is not identical to the PDF generated by Madoko, but I am not sure how important that is. I still have not figured out how to enable the syntax highlighting for the P4 code blocks, but am confident that is possible with a day or three of reading and experiments.

I am certainly open to anyone who has a preference for some other markup system other than AsciiDoc. When changing from one to another, it is important to consider expected longevity of support (which has to be based upon one's guesstimates of how long you expect the maintenance of the underlying software tools to last).

If you are examining alternatives, I would encourage reading the section of AsciiDoc's home page with the heading "Compare AsciiDoc to other markup languages": https://asciidoc.org

[jafingerhut]

@rcgoodfellow @asl If you want to look at what I have done so far in converting the P4 language spec into AsciiDoc source, see here:

• https://github.com/jafingerhut/learn-asciidoc/tree/main/p4-spec-next

If you have any suggestions on enabling syntax highlighting for P4 code blocks to work in both the generated HTML and PDF (output by asciidoctor-pdf command -- or any other command that generates PDF, but that is the only one I have tried so far), please let me know.

[rcgoodfellow]

I've got a rouge branch that seems to be working pretty well for P4 syntax highlighting. The current work is in a single commit. I'll leave this as-is for a few days for folks from the P4 community to chime in before posting a PR upstream.

I decided to go with rouge as it seems to be very actively maintained, is used by several popular frameworks, and is implemented in Ruby so the integration with asciidoctor is pretty seamless.

I got this to work with @jafingerhut's AsciiDoc port of the P4 spec by doing three things

• Install a local build of rogue (gem build rouge.gemspec && gem install rouge-4.2.1.gem)
• Change instances of [source,P4] to [source,p4] as the convention seems to be to use lowercase tags.
• Added :source-highlighter: rouge to the metadata at the top of the spec.

[jafingerhut]

Thanks, Ryan. I tried out installing your fork of rouge on a system of mine, and your other instructions, and it worked! Yes!

Some comments on some of the differences I observed in the current P4 code snippet highlighting in Madoko (marked with (M)), and the new highlighting in AsciiDoc with your instructions above (marked with (A)):

(M) bolds built-in type names like the following, but (A) does not.

• bit
• int
• varbit
• bool
• error
• match_kind
• void
• packet_out (but (A) does colorize packet_in)
• package ((A) sometimes colorizes it, but not in the code snippet
  near the end of Section 5.1.)

I see in your single commit for rouge above, that it mentions several of these as built-ins, so it is not clear to me why those are not colorized.

key, actions, and default_action are in your list of keywords, but not size. Intentional omission, or oversight? Also it seems odd to have entries in your colorizer code categorized differently than the other table property names.

GIven that you have some built-in method names recognized like extract and verify, it might be nice to consider adding isValid, setValid, and setInvalid, too.

Thanks again!

[rcgoodfellow]

Thanks for the feedback, Andy.

• I missed packet_in and size, those have now been added to keywords.
• The lack of colorization for package at the end of section 5.1 appears to be due to a missing newline between the preceding text and the [source,p4] directive.
• I've moved entries out of decls and into keywords with the other table property names. At the time, I was thinking about these as entry declarations, but I think it makes more sense to have all the table property names together.
• Added isValid, setValid and setInvalid to builtins and also moved extract and verify from keywords to builtins.

Regarding the bolding of builtins, I belive that is a property of the color scheme in use and not an intrinsic part of the syntax highlighter. Running bundle exec rackup in the rouge repo launches a test application that highlights things with a different color scheme than asciidoctor is using. In that case, builtins are in bold. So this may just be a matter of us choosing (or creating) a suitable color scheme for our asciidoctor build to use when rendering the P4 spec.

[jafingerhut]

@rcgoodfellow Thanks for the updates! I am trying them out, and things are looking better. I think there is a subtle colorization of the builtins that I am seeing, but it is a dark blue vs. uncolorized black so it is not easy to distinguish. Agreed that we can tweak the color scheme later if it helps.

One thing I forgot to ask about before -- are P4 comments, both double-slash to end-of-line syntax, and slash-star all the way up to the next star-slash syntax, recognized by your modified version of rouge? If I understand the code you added for P4, it appears like it might be categorizing it as whitespace, rather than as comments? Or maybe this is something that also needs tweaks to color choices to make it more visually obvious?

[rcgoodfellow]

The comments (both varieties) are being tagged as Comment lexical elements in rogue. In the Asciidoc rendering, they show up as light gray as opposed to uncolored text. I've made a small change to move comments out of the :whitespace lexer state and into a state labeled :comment to make the intent clearer. This is another place where we can choose to display comments with a more contrasting color in the color scheme.

[jafingerhut]

I might have missed some keyword here or there, but I think we have at least 95% of everything covered by Madoko P4 syntax highlighting, also covered by AsciiDoc P4 syntax highlighting, so I'd say "ship it" (and we'll make minor enhancements and tweaks over time later as needed).

Any tweaking of color choices in generated HTML and/or PDF appear to be configured outside of the rouge source code, and we can make those changes as and when desired.

Thankgs again for function P4 syntax highlighting with AsciiDoc!

[jafingerhut]

I have tested an install script for asciidoctor that I have hacked up on all of Ubuntu 20.04, 22.04, and 24.04, cross-producted with {amd64, aarch64} processors, and it successfully installs asciidoctor on all of them, in such a way that it produces both PDF and HTML output for the AsciiDoc version of the P4 language spec that can be found here: https://github.com/jafingerhut/learn-asciidoc/tree/main/p4-spec-next

There are differences in the PDF and HTML output from AsciiDoc, as compared to the PDF and HTML versions generated by Madoko. You can see the PDF output generated by Madoko in the usual place here: https://p4.org/specs/ and I have published the PDF and HTML output files generated by asciidoctor here: https://github.com/jafingerhut/learn-asciidoc/tree/main/p4-spec-next/sample-output

I think the next steps would be for the P4 language design work group to decide:

• Do we want to transition to using AsciiDoc instead of Madoko?
• If yes, does the generated output from asciidoctor as used in the learn-asciidoc repo linked above look good enough, or should its configuration be modified further?
• If/when it all looks acceptable, when do we want to switch over?

[rcgoodfellow]

The upstream Rouge PR has landed.

• Add lexer for P4 rouge-ruby/rouge#2049

So we can switch to the main Rouge branch.

[jafingerhut]

So we can switch to the main Rouge branch.

Excellent. I have done so here: https://github.com/jafingerhut/learn-asciidoc/blob/main/bin/install-asciidoctor.sh#L156-L158

[chrispsommers]

Hi Andy thanks for doing this. I noticed the HTML version of the sample output is missing all the images. It looks like images need to be in a figs/ directory. Other than that, the renderings for both PDF and HTML seems quite attractive and an improvement IMO.

[jafingerhut]

I have attempted to add the figs directory so that figures can be viewed within the HTML generated from the AsciiDoc source.

[jafingerhut]

As a status update on this, Davide Scano has been spending time learning about customizing the configuration of AsciiDoctor, the free software used for creating HTML and PDF from AsciiDoc format source files, in order to produce output with a better "look & feel" than my original attempts. If you want to follow along, or assist, see this PR: jafingerhut/learn-asciidoc#2