New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Rouge for source highlighting #1040

Closed
mojavelinux opened this Issue Jul 28, 2014 · 51 comments

Comments

Projects
None yet
@mojavelinux
Copy link
Member

mojavelinux commented Jul 28, 2014

Integrate the Rouge library as an alternate or replacement syntax highlighting library to Pygments.rb. Rouge is a port of Pygments to Ruby. The current Pygments integration calls Pygments by way of an interprocess communication bridge. This bridge doesn't work in AsciidoctorJ.

If we integrate Rouge and verify that it provides the same experience as Pygments, we can make it a drop-in replacement. If there are big enough differences, we can simply recognize it as an additional syntax-highlighter option named "rouge".

Rouge: https://github.com/jneen/rouge

@mojavelinux mojavelinux added this to the v1.6.0 milestone Jul 28, 2014

@AndorChen

This comment has been minimized.

Copy link

AndorChen commented Aug 25, 2014

👍

@mojavelinux mojavelinux modified the milestones: v1.6.0, v1.7.0 Dec 21, 2015

@AlexanderZobkov

This comment has been minimized.

Copy link
Contributor

AlexanderZobkov commented Jan 30, 2016

What is status of integration of Rouge?
Looking on the following line: https://github.com/asciidoctor/asciidoctor-maven-examples/blob/master/asciidoctor-pdf-example/pom.xml#L61-L63
it has been already integrated and this particular issue needs to be closed.

@AlexanderZobkov

This comment has been minimized.

Copy link
Contributor

AlexanderZobkov commented Jan 30, 2016

Probably asciidoc documentation needs to be updated as well. Currently the documentation mention support for pygments and coderay only.

@abelsromero

This comment has been minimized.

Copy link
Member

abelsromero commented Jan 30, 2016

Documentation is partially accurate and probably needs a review.
The thing is that not all source highlighters work on all backends and integration layers.
For instance rouge only works for PDF with asciidoctor-pdf, and pygments does not work in Java.

I always check the table here for the details, but maybe we should add a new column with the supported backends.

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Jan 30, 2016

That's exactly correct @abelsromero. Rouge was integrated into Asciidoctor PDF first as a trial. The alpha status of Asciidoctor PDF presented us with an opportunity to experiment with this library before committing to it. The results were good. Rouge works great! That's the point when we decided to integrate it into core. That part is not yet done.

Pygments is going to continue to be a problem because it just doesn't play well with JRuby (and other Ruby implementations). CodeRay is mediocre at best. That's why I think Rouge is the future.

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Jan 30, 2016

I think when we integrate Rouge into core, we're going to drop CodeRay (and alias coderay to rouge). We have to keep Pygments because we have a handful of faithful users of it (and because it really is the most comprehensive), but we won't recommend it.

@AlexanderZobkov

This comment has been minimized.

Copy link
Contributor

AlexanderZobkov commented Jan 31, 2016

I think It would be better to update the table/section mentioned by @abelsromero with information on Rouge. It will make users (existing and new ones) aware on the integration with Rouge and allow more users to experiment with it so that you can get more feedback on this feature.

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Jan 31, 2016

I agree. Keep in mind, though, that we haven't extensively documented Asciidoctor PDF yet because it is still in alpha and we are changing a lot of things. I'm hopeful we'll be emerging from alpha soon and thus getting ready to integrate the documentation into the user manual.

@mojavelinux mojavelinux changed the title Integrate Rouge as an alternate / replacement for Pygments.rb Integrate Rouge in place of CodeRay for source highlighting Mar 13, 2016

@AlexanderZobkov

This comment has been minimized.

Copy link
Contributor

AlexanderZobkov commented Mar 14, 2016

I see a strange issue when using rouge syntax highlighter.

[ERROR] Failed to execute goal org.asciidoctor:asciidoctor-maven-plugin:1.5.3:process-asciidoc (generate-pdf-doc-default-theme) on project asciidoctor-pdf-with-theme-example:
Execution generate-pdf-doc-default-theme of goal org.asciidoctor:asciidoctor-maven-plugin:1.5.3:process-asciidoc failed:
org.jruby.exceptions.RaiseException: (RuntimeError) asciidoctor: FAILED: required gem 'rouge' is not installed. Processing aborted. -> [Help 1]

The issue is seen when converting a document into PDF on Windows under cygwin or Netbeans IDE. On Linux, I see no such issue.

Could you please suggest for which project I should submit an issue? For asciidoctorj, asciidoctor-maven-plugin or asciidoctor?

In addition, I think I need to ask author of asciidoctor-maven-examples to add some dummy code to example documents to catch problems with highlighters.

@abelsromero

This comment has been minimized.

Copy link
Member

abelsromero commented Mar 14, 2016

We'll need more information to see the problem, I run the asciidoctor-pdf-with-theme-example example in Cygwin with Win 8.1 and it worked.

Could you please suggest for which project I should submit an issue? For asciidoctorj, asciidoctor-maven-plugin or asciidoctor?

Deppends...are you trying to run the example "as-is" or doing something else?

In addition, I think I need to ask author of asciidoctor-maven-examples to add some dummy code to example documents to catch problems with highlighters.

First, we are all authors here :) And secondly, how to test examples is something that appears from time to time. One one side we rely on 'manual' testing and good faith', but on the other adding testing in the examples destroys the purpouse of keeping them as simple as possible. But still, we are open to ideas, feel free to file an issue in the examples projects if you have some idea in mind.

@abelsromero

This comment has been minimized.

Copy link
Member

abelsromero commented Mar 14, 2016

Correction, Windows 7

@AlexanderZobkov

This comment has been minimized.

Copy link
Contributor

AlexanderZobkov commented Mar 14, 2016

I'm using unmodified asciidoctor-pdf-with-theme-example with added a source code block to demonstrate and discuss the issue.

I think it will be enoght to add the following block to enable rouge and see the issue.

[source,ruby]  
.app.rb 
---- 
require 'sinatra'

get '/hi' do
  "Hello World!"
end
----
@abelsromero

This comment has been minimized.

Copy link
Member

abelsromero commented Mar 14, 2016

Confirmed 😞 In MacOS works, but not in Windows. Even after adding rouge as a dependency of the gem-maven-plugin it still says it's not installed.

Caused by: org.jruby.exceptions.RaiseException: (RuntimeError) asciidoctor: FAILED: required gem 'rouge' is not installed. Processing aborted.
        at RUBY.require_library(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoctor/helpers.rb:30)
        at RUBY.convert_listing_or_literal(uri:classloader:/gems/asciidoctor-pdf-1.5.0.alpha.11/lib/asciidoctor-pdf/converter.rb:948)
        at RUBY.convert(uri:classloader:/gems/asciidoctor-pdf-1.5.0.alpha.11/lib/asciidoctor-pdf/converter.rb:96)
        at RUBY.convert(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoctor/abstract_block.rb:71)
        at RUBY.block in content(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoctor/abstract_block.rb:80)
        at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2300)
@AlexanderZobkov

This comment has been minimized.

Copy link
Contributor

AlexanderZobkov commented Mar 15, 2016

Okay, great, it's reproducable not only on my side. Could you please suggest for which project I should submit an issue?

@robertpanzer

This comment has been minimized.

Copy link
Member

robertpanzer commented Mar 15, 2016

@abelsromero Is there maybe another nested exception?

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Mar 15, 2016

The way that we require the rouge gem is a bit different than other libraries. We require a relative file with extensions to Rouge, which in turn requires the rouge gem. It appears that this path does not resolve properly on Windows. (This looks like other path issues we've had with JRuby).

Therefore, I'd say we probably have to address this issue in Asciidoctor PDF. But first, we need to understand what to do to get JRuby to resolve the path correctly.

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Mar 15, 2016

Do we have a test in AsciidoctorJ PDF that attempts to syntax highlight using Rouge?

@robertpanzer

This comment has been minimized.

Copy link
Member

robertpanzer commented Mar 16, 2016

Yes, @abelsromero added a nice test: https://github.com/asciidoctor/asciidoctorj/blob/asciidoctorj-1.6.0/asciidoctorj-pdf/src/test/java/org/asciidoctor/WhenBackendIsPdf.java#L63
It's currently only on the 1.6.0 branch though.

Am 16.03.2016 um 00:11 schrieb Dan Allen notifications@github.com:

Do we have a test in AsciidoctorJ that attempts to syntax highlight using Rouge?


You are receiving this because you commented.
Reply to this email directly or view it on GitHub

@robertpanzer

This comment has been minimized.

Copy link
Member

robertpanzer commented Mar 16, 2016

1.6.0 is JRuby 9.0.4.0 atm

Am 16.03.2016 um 00:11 schrieb Dan Allen notifications@github.com:

Do we have a test in AsciidoctorJ that attempts to syntax highlight using Rouge?


You are receiving this because you commented.
Reply to this email directly or view it on GitHub

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Mar 16, 2016

Awesome!

@AlexanderZobkov or @abelsromero, can you try the example on Windows using JRuby 9k. I have a feeling this is another JRuby 1.7 issue (or perhaps one we've already seen).

@nfrankel

This comment has been minimized.

Copy link

nfrankel commented Dec 21, 2017

I confirm this works well in my case, thanks @miltador

@chickenkiller

This comment has been minimized.

Copy link
Contributor

chickenkiller commented Dec 21, 2017

I confirm as well. Works for me with asciidoctor-rouge extension. Thanks!

@anidotnet

This comment has been minimized.

Copy link

anidotnet commented Feb 4, 2018

How do I enable rouge in asciidoctor-gradle?

@mojavelinux mojavelinux modified the milestones: v1.5.7, v1.5.8 Apr 7, 2018

@mojavelinux mojavelinux modified the milestones: v1.5.8, v2.0.0 Oct 27, 2018

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Jan 26, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add a SyntaxHighlighter adapter implementation for Rouge
- add styles to default stylesheet to layout the line number column
- add the rouge gem as a development dependency
@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Jan 26, 2019

Now that the syntax highlighter adapter in Asciidoctor is pluggable, this issue can proceed without adding complexity to the codebase.

Instead of trying to map coderay to rouge, I've decided to simply add rouge as a proper adapter. (Besides, it happens to be more like Pygments than CodeRay).

Even though we're adding rouge to core (for convenience), it's still possible to override it with a separate implementation thanks to the pluggable syntax highlighter design.

I'm thinking about extracting coderay as a separate gem. Since coderay doesn't match the quality of rouge, there's no reason to keep both in core. And if you still need it, you can just add the gem to the require path. But that will be handled as separate task.

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Jan 26, 2019

Rouge does not support the "inline" mode for line numbers, but honestly I don't think that matters since that mode is much harder to style anyway.

One thing that's missing from Rouge out of the box is highlighted (aka emphasized) lines. I'd like to see that added upstream. However, if that's a blocker, we could add a custom formatter for Rouge that adds this functionality. I'd like to do that in a separate issue, though.

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Jan 26, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add a SyntaxHighlighter adapter implementation for Rouge
- add styles to default stylesheet to layout the line number column
- add the rouge gem as a development dependency

@mojavelinux mojavelinux changed the title Integrate Rouge in place of CodeRay for source highlighting Integrate Rouge for source highlighting Jan 28, 2019

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add a SyntaxHighlighter adapter implementation for Rouge
- add styles to default stylesheet to layout the line number column
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- remove nested <pre> wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure library is loaded when looking up base style
- add the rouge gem as a development dependency

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add SyntaxHighlighter adapter implementation for Rouge
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested <pre> wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add SyntaxHighlighter adapter implementation for Rouge
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested <pre> wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add SyntaxHighlighter adapter implementation for Rouge
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested <pre> wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add SyntaxHighlighter adapter implementation for Rouge
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested <pre> wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add SyntaxHighlighter adapter implementation for Rouge
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested <pre> wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add SyntaxHighlighter adapter implementation for Rouge
- support CGI options attached to language (e.g., "console?prompt=$ ")
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested pre wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency

mojavelinux added a commit to mojavelinux/asciidoctor that referenced this issue Feb 3, 2019

resolves asciidoctor#1040 add syntax highlighter adapter for Rouge
- add SyntaxHighlighter adapter implementation for Rouge
- support CGI options attached to language (e.g., "console?prompt=$ ")
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested pre wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency
@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Feb 3, 2019

One thing that's missing from Rouge out of the box is highlighted (aka emphasized) lines.

I've implemented this feature as part of this PR. I'll still pursue getting it added upstream, at which point we may decide to switch the code over to using that instead.

mojavelinux added a commit that referenced this issue Feb 3, 2019

resolves #1040 add syntax highlighter adapter for Rouge (PR #3033)
- add SyntaxHighlighter adapter implementation for Rouge
- support CGI options attached to language (e.g., "console?prompt=$ ")
- change CSS class of line numbering table for Pygments and Rouge to linenotable
- reuse styles from default stylesheet to layout the line number column
- remove nested pre wrapper in Pygments output when line numbering is enabled and linenums mode is inline
- ensure syntax highlighter library is loaded when looking up base style
- add the rouge gem as a development dependency
@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Feb 3, 2019

The supported options are as follows:

  • source-highlighter=rouge (activates this integration)
  • rouge-css=style or rouge-css=class (controls whether styles are added inline or via CSS classes)
  • rouge-style=<name> (the name of a rouge theme)

As with Pygments and CodeRay, if linkcss is set (and rouge-css=class), the stylesheet will be externalized.

Note that rouge-linenums-mode is not recognized.

@Mogztter

This comment has been minimized.

Copy link
Member

Mogztter commented Feb 3, 2019

Nice work 👍
Could you please add a reference on the upstream issue ?

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Feb 3, 2019

Here are the upstream issues:

@mojavelinux

This comment has been minimized.

Copy link
Member Author

mojavelinux commented Feb 3, 2019

I'd also like to see support for customizing the CSS class prefix for tokens, which is jneen/rouge#1074.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment