Generating documentation no longer works

[mcmire]

We use Pygments along with YARD to highlight source code in documentation. Unfortunately this seems to have stopped working and upon running yard doc (which happens automatically when creating new releases) we are getting:

MentosError: No header received back

This seems to have been a problem for a very long time and there is no known solution.

Perhaps instead of Pygments we should switch to Rouge instead?

[KapilSachdev]

Is this a system specific issue or an intermittent issue and happens sometimes?

BTW, Rouge looks great and would be a good replacement.

[mcmire]

@KapilSachdev I don't know if it's a system specific issue, but it's definitely repeatable on my machine (Mac). Previously there was an issue with using Python 3 vs. Python 2 — I believe I cleared that up, so there was a long stretch where it did work for a while. But now within the past 6 months I'd say, I've started receiving this error.

[KapilSachdev]

Then the alternative would be a better way to go.

However, we can also create a docker image and try to use the image for release, but that would be an overkill and also we can't be sure if it wouldn't appear in the image.

[KapilSachdev]

Looks good too.

With pgyments

With rouge

[mcmire]

Oh nice — do you have it working where it's swapped out in the YARD config, or were you just testing it out on something else?

[KapilSachdev]

Changed the yard setup doc_config/yard/setup.rb to use rouge instead of pygments.rb.

diff --git a/doc_config/yard/setup.rb b/doc_config/yard/setup.rb
index eb654f5b..381b4269 100644
--- a/doc_config/yard/setup.rb
+++ b/doc_config/yard/setup.rb
@@ -1,6 +1,6 @@
 YARD::Templates::Engine.register_template_path(File.dirname(__FILE__) + '/templates')
 
-require 'pygments.rb'
+require 'rouge'
 
 module YARD
   module Templates
@@ -13,8 +13,7 @@ module YARD
         private
 
         def highlight_with_pygments(language, source)
-          html = Pygments.highlight(source, lexer: language.to_s)
-          html.sub(%r{\A<div class="highlight">\s*<pre>}, '').sub(%r{</pre>\s*</div>\Z}, '')
+          Rouge::Formatters::HTML.new.format(Rouge::Lexers::Ruby.new.lex(source))
         end
       end
     end

As the only language being used was ruby, so I used Rouge::Lexers::Ruby as opposed to defining the constant from language for testing.

Can you clarify what the below substitution is doing?

html.sub(%r{\A<div class="highlight">\s*<pre>}, '').sub(%r{</pre>\s*</div>\Z}, '')

[mcmire]

@KapilSachdev Very nice, that makes sense. IIRC, Pygments was adding some additional HTML that was making it more difficult to apply CSS to the code blocks, so I got rid of that. I'm not sure if Rouge does the same thing or not.

[KapilSachdev]

It is visible when using pygments.rb and removing that substitution.

Adding or removing the block does not change anything using rouge.

[mcmire]

Ah yup, there you go, that's what it was. I knew it was something CSS-related.