Allow users to specify theme for syntax highlighting

[ramnathv]

First off, excellent work on the package. Sweave is a part of my daily workflow, and this package provides most of the features that I wished Sweave would. I have a couple of suggestions, which I will try to post from time to time. My first suggestion is to add a feature that allows users to specify the theme for syntax highlighting.

I see that you are using the highlight package, which uses a css file located in the stylesheet folder to control the highlighting. It would be nice to allow users to specify the highlighting theme using a css file. In the long run, it would be possible to convert standard themes available in highlight or textmate to css files, allowing the user to choose the style in SweaveOpts.

Let me know if you are open to this suggestion. I have browsed through your code and see that currently, the style definitions are in the variable .header.hi.tex. I can work with you in creating the mechanism that would allow user specified styles, if you would like me to.

[yihui]

This is definitely a good suggestion, and in fact you have already found the solution. The only thing left for me to do is to make the function set_header() visible to users (i.e. export it), so that you can use set_header(highlight = 'your.own.styles'). Or if you are really eager to try it out right now, you can use knitr:::set_header

[ramnathv]

Thanks for a prompt reply. So the argument highlight takes a css file as input, or a list containing the syntax highlighting definitions? If you could make that clear, it would be appreciated.

[yihui]

It takes a character vector containing your style definitions. You can take a look at knitr:::.header.hi.html (or knitr:::.header.hi.tex if you use LaTeX).

I will document this function later.

[ramnathv]

Excellent. I have a suggestion in this regard. It might be useful to allow users to define a css file along the lines of highlight. In the long-term, this makes it easier for users to convert existing syntax highlighting themes to css and build a repository of themes to choose from.

Internally, this does not create too much work, since you can do the following to get .hi.header.tex.

css.out  <- highlight::css.parser(style_file)
style  <- highlight::styler_assistant_latex(css.out)
.hi.header.tex <- stringr::str_c(c(style, boxes_latex()), collapse = "\n") 

Alternately, it might be possible to contact the author of highlight and add the additional style files directly in the highlight package, in which case, you can revert back to your earlier code, with default being over-ridden by the theme chosen by the user.

.header.hi.tex = str_c(c(styler('default', 'sty', styler_assistant_latex), boxes_latex()), collapse = '\n')

I am in the process of generating css files for all the styles available in Andre Simon's `highlight package, as well as some of the themes in CodeRay.

Let me know what you think.

[yihui]

I believe this is a good idea. I'd love to see your work on porting the styles in Andre's highlight package, and will be happy to talk to Romain about it (I noticed he copied some of the style files to his package, but I do not know how we are supposed to use them).

[ramnathv]

I have ported styles from Andre's highlight package, as well as have the working code to port from Ecllipse themes. The basic idea is to (a) map tokens from the style files to Romain's highlight package and (b) use a brew template to populate the css file. I will post the code shortly.

One bottleneck I hit, while testing is that I was unable to change the style definitions used by knitr using knitr:::set_header(highlight = my_style). For some reason, the style kept reverting back to default. Any ideas on why this might be happening, or can you suggest a workaround.

Once I have this fully tested, I will fork your code and make the required code modifications to allow users to specify an external css file or theme for syntax highlighting.

[yihui]

Oh, you should call that inside a code chunk in the Rnw document, because each time you call knit(), it will use the default theme, unless you modify it later (I should probably change this behavior). The header is written after all chunks have been evaluated.

[ramnathv]

Perfect. It works now. I will put together proof-of-concept and send you the details so that you can take a look. Subsequently, we can figure out the best way to integrate it into knitR provided you find the additions convincing.

[ramnathv]

There is one issue that remains. When using dark backgrounds, the output gets completely masked as it is still in black. Do you have any suggestions on what is the best way to modify the output color, using the foreground colour specified by the theme.I was thinking either modifying the verbatim environment, or using a hook. Any suggestions?

[yihui]

Hmm... interesting question. I would not recommend modifying the verbatim environment. Certainly it can be done via a hook function like knit_hooks$set(output = function(x, options) paste('\\begin{mycolor}\\begin{verbatim}', x, '\\end{verbatim}\\end{mycolor}')), but the problem is how do we define the foreground color of the verbatim environment, or what other environments can we use for verbatim output in LaTeX in which we can define fg colors? I do not have an idea right now.

[ramnathv]

That works. I used the hook knit_hooks$set(output = function(x, options) paste('{\\color{white}\\begin{verbatim}', x, '\\end{verbatim}}')), with the white being replaced by the foreground color of the theme being used.

[yihui]

Actually I tried \color{white}{\begin{verbatim} ... \end{verbatim}} before and failed. It is good to know {\color{} ...} works.

[ramnathv]

I have successfully converted all themes into css files, that highlight can understand. Here is a pdf with all the themes ( http://dl.dropbox.com/u/1161356/all.pdf) The key was to create a mapping between the tokens defined in the style files with those in the css file. Tweaking the mapping might lead to better highlighting.

Using the same mapping approach, I was also able to convert themes on http://www.eclipsecolorthemes.org/ to css files that can be read by highlight. Here is an example of using sunburst: http://dl.dropbox.com/u/1161356/sunburst.pdf.

I am still thinking on what is the best way to proceed on this. My current inclination would be to (a) have a themes folder in knitr which is populated with the 100 odd css files that I have, (b) add a theme option to allow users to specify the theme by name, and (c) write an internal function, that reads the theme file and sets the latex header, background, and output colour appropriately.

Let me know what you think about this. Based on your preference, I can fork your code and write my implementation and send a pull request.

[yihui]

Oh, this is really really cool! I do not have any additional comments; please go ahead with your ideas. Thanks!

BTW, what is the total file size of these css files?

[ramnathv]

Great. The css files are really small at around 4KB per file. So to accommodate 100 themes, it would around 400KB.

So I will go ahead and fork your project, and do the following:

1. create a folder called themes under the inst folder with css files.
2. add a function called theme2header, which converts the theme to a latex header.
3. add knitr:::set_style(highlight = theme2header(theme)), so that the header is set up accordingly

[ramnathv]

I am almost done with the implementation. The one issue I am facing is while using themes with dark backgrounds. Using a hook based on your suggestion, I am able to adjust the color of any text output. But figures still get rendered in black, which makes them invisible. I tried setting par(col = white), but doesn't seem to work. Any thoughts on how the default plotting color for all graphics (base and grid) can be set based on the theme used?

I am also adding some really cool themes from http://www.eclipsecolorthemes.org/.

[yihui]

I will address the issue of the background color of plots in another manual for graphics. In short, you can use par(bg = 'white'); see http://yihui.name/en/2011/12/knitr-elegant-flexible-and-fast-dynamic-report-generation-with-r/#comment-11066

Your work sounds terrific and it seems people will never need to consider the LaTeX listings package again, since the R package highlight can already render so beautiful output.

[ramnathv]

I have added the implementation of themes to my fork of knitr. I have set it up such that the themes are expected to be in the themes folder of the binary. The easiest way to check it out without disrupting anything else would be to

1. copy the themes folder to knitr/themes.
2. source the file R/themes.R which contains the relevant functions
3. run knit_to_pdf on the file inst/examples/knitr-themes.Rnw

I still need to add details to the documentation.

Let me know if it works for you, or if I have inadvertently hard-coded anything for my system.

[yihui]

I will take a look tomorrow. Thanks!

[yihui]

I'm reading your theme code, and it looks great. A few issues first:

1. Romain's highlight borrows the styles from Andre's highlight: http://www.andre-simon.de/zip/download.html Now I see the style files in Romain's package are kind of outdated. Does it require a lot of work if you use the files in Andre's package?
2. Instead of generating themes in real-time, I'm thinking of gathering them as a dataset in knitr named, say, all_themes, which is a list like all_patterns, then set_theme() directly uses all_themes; you can create all_themes at the package build time (i.e. use all_themes <- the.list.of.themes in the R script). This will be hopefully faster for later uses after the package is built.
3. knitr::: is not actually necessary when you write the package; internally you can use any R objects in this package; if you want to test the package, you can use load_all() in devtools, which will make all objects in the package visible to you.
4. knitr has a similar functionality as brew, so perhaps we can knit() the css template; I will give a try later.

[ramnathv]

Thanks for the comments Yihui. Here are my responses to the issues you raised.

1. If the styles in Andre's highlight package retained the same format, then it would be easy to regenerate the css files, since it is just a matter to re-running the generation code.
2. The themes are not generated in real-time. I included the code files themes-highlight.R and themes-ecllipse.R to document the process by which the css files are generated, so it would be easy to update them when necessary. One exception to this is the function save_eclipse_theme, which allows a user to save and use a theme from www.ecllipsecolorthemes.org, by specifying the theme number.

I think it would be easier to provide the css style files as a folder which gets installed at build time. We could provide a function all_themes, that allows the user to access these themes. The advantage of this approach is that the user can easily add more themes to this folder, and the package will automatically pick it up. Let me know what you think.

1. I understand. I am new to devtools, so was unable to make load_all work. Let me see if I can figure it out.
2. Sure, if you make some progress on this, let me know and I can change the brew code to knit.

A few more things. I noticed that you are using themes to specify the type of document (e.g. tex, sweave, html etc.). It would be confusing to the user, if themes are used to denote syntax highlighting schemes too. Any suggestions for an alternate name. I was thinking, it could be named explicitly, say something like codecss or say codestyle.

[ramnathv]

I checked the themes on Andre's website. The formats have changed, so I need to rewrite the parser that will extract the elements from his themes and populate the brew template to generate the css files. Not too much work, but will take some time.

My preference would be to agree on a draft design of how code themes would operate. Here is how I envision it:

1. Syntax highlighting themes are provided as css files in a codethemes folder, which is installed during build time.
2. Users can get a list of themes, or access a specific theme using a get_code_theme function.
3. Users can call set_code_theme inside a knitr document to set the code theme they desire.

I will try to get (1) - (3) done today, use devtools to test how it works, and issue a pull request so that you can double check on how it affects the knitr codebase.

In parallel, I can focus on generating new css files based on the new themes in Andre's highlight package, without having to worry about how it would impact knitr.

Let me know if this sounds good.

In the long run, I think it might make sense to provide codetheme as an option which the user could set using \SweaveOpts{codetheme = 'sunburst'}.

[ramnathv]

I got it working with devtools. One question remains. All the css files are under the inst/codethemes folder in the source. In the built version they are going to be in codethemes. How do I refer to it while testing with devtools? Any ideas?

[ramnathv]

A quick question. Can you take a look at http://www.eclipsecolorthemes.org/? I have written a function that can take a theme number, download the XML style file and convert it to css that highlight can use.

Given that they have a library of 4939 color themes, I was thinking it makes sense to restrict attention solely to themes on that site. A user can visit the site and preview the themes, and would be able to add a theme on the fly. For example, to use solarized dark (http://www.eclipsecolorthemes.org/?view=theme&id=1115), they would add the following code:

download_eclipse_theme(1115)
set_code_theme('solarized.dark')

Let me know what you think. If you are in agreement, then I can restrict my attention to just the themes on this site, and probably provide the top 100 themes for direct use in knitr.

[yihui]

I think this site is enough (may be far more than enough, haha).

Last night I suddenly realized that the theme can be generated in real time, because knitr has the unbeatable cache! :) For example, we can create the theme in the first chunk and cache it, then the theme is only generated for the first time; in the second theme, we call a function to set the theme. It should be fairly fast. But you need to do a little bit more work -- let your theme functions return the character string (the header/preamble), which can be cached in the first chunk and used in the second chunk.

[ramnathv]

My preference would be to add this option on top of the pre-packaged css theme files, so that users can work on documents offline. I have started drafting a syntax highlighting manual http://dl.dropbox.com/u/1161356/knitr-highlighting.pdf, (already sent you the link on google+) that illustrates the use of themes, and how users can customize.

One issue to figure out is the copyrights for ecllipse color themes. The website does not say anything, but might be worth checking before using them in knitr

[yihui]

Hmm... I did not find anything on copyrights in this site either. We can first include those from Andre's highlight package (it is under GPL), and leave the ecllipse themes alone; if the users really like the latter ones, they can use the cache method I mentioned above to generate the theme in real time, and we are free of copyright problems.

BTW, the highlight manual looks great!

[ramnathv]

Makes sense. As I had indicated before, the format of style files has changed in Highlight 3.0, which means it will take some work to convert them into the css specification. I will try to get that done soon so that I can issue a pull request and get this thing working.

[ramnathv]

I am done converting themes from Andre's Highlight 3.0 to css files. The codethemes folder currently stands at a massive 150 themes. I am working on the manual to document everything, including the process for generating the css files.

One issue that remains is the foreground color of plots, especially when using dark backgrounds. Is there a consistent way to change the foreground color to the foreground color of the theme so that plots are rendered visibly?

[yihui]

as long as you have a line \definecolor{fgcolor}{...} in the preamble (I believe you have done so), everything will be fine

[ramnathv]

It does not seem to be working with plots. I am still getting plots with black outlines, when using a dark background. I will upload a sample file so that you can test it out.

[yihui]

That is another story; I expect the users to par(fg, col.axis, col.lab, ...) by themselves in the case of a dark background. It is not straightforward to do it automatically on our side, and it is better done with the fig hook so it applies to all chunks.

[ramnathv]

It is now possible to use any theme on www.eclipsecolorthemes.org by adding the following chunk to your Rnw file.

theme <- save_eclipse_theme(10)
set_theme(theme)

This enhances the number of themes accessible to a user manifold, and I think is pretty cool !

[yihui]

Yes, this is exactly what I want :)

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.