error building asciidoctor >= 2.0.16 for RHEL/EPEL 8 (ruby-2.5.9)

[tmzullinger]

While attempting to build asciidoctor-2.0.20 for Fedora and RHEL+EPEL releases, I ran into the following error on RHEL+EPEL 8:

Installing ri documentation for asciidoctor-2.0.20
Installing darkfish documentation for asciidoctor-2.0.20
ERROR:  While executing gem ... (RDoc::Error)
    error generating Asciidoctor/Converter/ManPageConverter.html: no implicit conversion of nil into String (TypeError)

Going back, this began in 2.0.16. The RHEL+EPEL 8 has been on 2.0.15 for some time. I wish we'd have seen and reported this sooner, but I wasn't really maintaining the EPEL packages at the time. I ran into this while attempting to update the epel8 and epel9 branches.

Does this error ring any bells? Searching yields a few results which point to bugs fixed in rdoc, though that seems odd since it works in 2.0.15 with the same ruby stack which fails with newer asciidoctor releases. Perhaps it's just that the older releases didn't tickle this rdoc bug. Or perhaps that's a red herring.

[mojavelinux]

I have no idea what this is. We haven't touched the rdoc in the manpage converter in ages. This looks to me like some sort of bug in the ri generator. But I don't see it when I install the asciidoctor gem on Ruby 2.4 - 3.2.

I almost never look at or run the rdoc generator, so it doesn't get any sort of testing. Perhaps we can add a test to the suite that runs it. Though, like I said, I can't seem to make it fail. If you know how, please inform.

[mojavelinux]

Aha! I can trigger it using the following command:

gem rdoc --rdoc asciidoctor

[tmzullinger]

Cool, thanks!

FWIW, the Fedora/EPEL builds run gem install -V --local --build-root . --force --document=ri,rdoc asciidoctor-2.0.16.gem where that gem is generated earlier in the build process via gem build asciidoctor.gemspec.

A build log from a recent test run is here.

[mojavelinux]

I remember what this is now. This is an old bug in rdoc with the position of the dot in a chained method call.

See https://github.com/asciidoctor/asciidoctor/blob/main/lib/asciidoctor/converter/manpage.rb#L725-L758

I'm very reluctant to change this because it is definitely a bug in the rdoc parser. Using the dot at the start of the next line is perfectly valid Ruby.

[mojavelinux]

This is the related commit: fb34e6d

[mojavelinux]

It's also the recommended Ruby style. See https://docs.rubocop.org/rubocop/cops_layout.html#layoutdotposition. That's why we changed it. We can't pass the RuboCop rules with it the old way.

You could apply a patch that moves the dot from the start of the line to the end of the code on the previous line.

[mojavelinux]

This stopped being a problem in Ruby 2.6. So it's an issue only with Ruby 2.5 (and perhaps below).

[mojavelinux]

I still don't understand why CentOS/RHEL continue to support such outdated versions of Ruby. The Ruby project makes it very clear that these are no longer supported, and anyone using them is not going to get support from the community. It really would be nice if CentOS/RHEL at least supported the oldest Ruby that isn't already EOL for over two years. It's one thing to support core libraries in a OS (because an OS upgrade is not trivial). But these are programming languages we are talking about. The community doesn't accommodate developers who are using them just because they're running an older operation system. It just doesn't integrate with the world we live in. And it puts more burden on maintainers like myself who can't upgrade the project because a major operating system won't let users upgrade the language runtime. It's an unreasonable ask.

[mojavelinux]

I will go ahead and add a test that runs rdoc on versions of Ruby that we test again just so we know that it works on install.

[tmzullinger]

Thanks for digging up the rationale Dan!

It's definitely a pain as RHEL ages. Red Hat does support multiple versions of Ruby as "modules" but I don't believe that EPEL packages can build against anything other than the default ruby runtime module, which is still 2.5. As modules, ruby 2.6, 2.7, 3.0, and 3.1 are available.

I'll check whether the EPEL package can build against a newer ruby. If so, that'd be great. If not, then I'll just be annoyed at the lack of usefulness in the RHEL modules. :)

[mojavelinux]

I managed to figure out how to integrate building, installing, and removing the gem into the CI workflow. That's a good thing to test anyway so we catch problems (or at least be aware of them) before the user sees them.

I also updated to the code to make it easier to patch. See cc9a8d3 I want to run this on all versions of Ruby in the matrix. But since the 2.0.x branch is still running CI on Ruby 2.3, I really have no choice but to add a workaround. That workaround will not be added to the main branch since the minimum version of Ruby there is 2.7.

[mojavelinux]

It's definitely a pain as RHEL ages.

It's not just a pain, it's simply not acceptable. Not upgrading language runtimes causes a huge strain on the community and we cannot be expected to take on that work unless Red Hat is willing to put us on retainer. Even if we decide not to make a change upstream, just having to respond to the inquiry and investigate the root of the problem already costs time and money. It's one thing if it's a problem that affects the supported runtimes. But when it turns out to be due to a problem in an EOL runtime, that another thing entirely. All we're doing then is replaying bugs that are known to exist, and that will never be fixed.

Is there someone you can put me in touch with at Red Hat that I can talk to in order to communicate how much of a disruption this is for open source and the lives of maintainers? We just can't continue on the path like this.

[tmzullinger]

Unfortunately, I don't know anyone particular on the Red Hat side of things. I'm merely a volunteer contributor to Fedora and EPEL.

Maybe something that @mattdm could help with? Matthew is the Fedora Project Leader which is a Red Hat position. I hope it's not too much trouble to tag you Matthew. (I know from posts on the Fedora lists that there is a Red Hat Summit event this week, followed by the long holiday weekend in the US, so it may be a bit before we hear anything.)

I completely understand how these sort of backports are annoying and painful. It's more than reasonable to say it's out of scope. I certainly don't want to be a drag on the time you could be spending making things better for current distribution releases. (I did see 2.3 listed as the minimum CRuby version, so I thought this was worth reporting.)

I filed a bug and submitted a patch to the ruby 2.5 package in CentOS Stream 8 (RHEL 8's upstream) to backport the rdoc change. I don't know what the odds are of having that accepted for the next release of RHEL 8, which should be 8.9, in the late fall.

Cheers & thanks!

[mojavelinux]

Thanks for following up, Todd. I do appreciate your attention to the matter.

I did see 2.3 listed as the minimum CRuby version, so I thought this was worth reporting.

To be clear, the reason that version is so old is specifically for RHEL. I held it back because each time I don't, we end up in this conversation again. I think it's imperative that mainstream Linux operating systems respect the EOL dates of programming languages. (After all, they do for browsers like Firefox). No one should be developing with a programming language version that is no longer receiving updates, especially one that has been EOL for 2 years. It's not just a drain on my time, but the whole open source community around that language. It's a distraction and hurts progress.

There was a time when these versions could be held back for longer, before modern release cycles and the threat of major security incidents. But that time has passed. Runtimes are iterated a lot faster now (just like the Linux kernel itself). Policies have to be revisited to keep up with the times. Red Hat can't dictate the way software development is done in open source, it needs to adapt to it.

[tmzullinger]

I definitely understand. I see it on both sides while trying to package things for Fedora and EPEL. I'm certainly not looking to speak on behalf of Red Hat nor defend their policies. If I can help explain how things interact between RHEL and the volunteer driven Fedora EPEL project, I'd be happy to try.

On the RHEL distribution end, they are trying to solve this issue with what they call "modularity / modules." This is where there are multiple installation streams for a language stack like ruby. In RHEL 8, they have 2.5 (the default), 2.6, 2.7, 3.0, and 3.1. Unfortunately, in EPEL we can only build for the default module, which negates the benefit of the module streams if I want to ship something as an rpm. I can only hope that is something Red Hat improves. They benefit from the volunteer work done in EPEL which in turn benefits from the good work of upstream projects like Asciidoctor. No one benefits from souring these relationships.

And, if it helps at all, RHEL is backporting patches for bugs and security issues in their ruby-2.5 package. It's old but they do take on reponsibility for maintaining the stack to a reasonable extent. I'm hoping that they'll accept the patch to backport the rdoc fix I submitted.

In the git project, the oldest supported release is often either the older of RHEL or Debian oldstable (currently RHEL 7 and Debian Buster, respectively). If or when either of those become too painful to support, they're dropped. With respect to ruby, Debian Buster is also shipping ruby-2.5. I imagine they've got backports of more recent ruby versions too, though I don't follow them enough to be sure.

I also noticed I got the title wrong, it's RHEL/EPEL 8, not 9. I fixed that, just for posterity.