This repository has been archived by the owner. It is now read-only.

Should we switch to Jekyll for our lessons? #279

Closed
gvwilson opened this Issue Jun 25, 2015 · 15 comments

Comments

Projects
None yet
7 participants
@gvwilson
Member

gvwilson commented Jun 25, 2015

This issue continues the discussion started in http://lists.software-carpentry.org/pipermail/discuss_lists.software-carpentry.org/2015-June/003118.html - please add further comments on that thread here. In particular, please read this post from @jdblischak on us having come full circle, and the cost of repeated changes to our build system, and this one from @ocefpaf on using Travis-CI to build our HTML.

@cryptarch

This comment has been minimized.

Show comment
Hide comment
@cryptarch

cryptarch Jun 25, 2015

From the email discussion, I have a problems with @r-gaia-cs's defence of pandoc. Pandoc (and therefore R Studio) are not available in Arch Linux; even their Arch User Repos are broken. I've had serious problems contributing to R material and helping at R-based workshops because of this.

You might think, well, that's just one distro. However, it is an important and popular distro, and this problem with pandoc is furthermore symptomatic of a deeper problem: it is built on Haskell, which - while interesting and supported by many worthy nerds - is not robust in the slightest. We cannot rely on pandoc to not break in any other arbitrary distro or OS at any time.

I can only support a solution that is as robust and cross-platform as possible, with the least amount of dependencies or arbitrary "gotchas". That means dropping pandoc until such time as the Haskell community can get its act together on package management.

cryptarch commented Jun 25, 2015

From the email discussion, I have a problems with @r-gaia-cs's defence of pandoc. Pandoc (and therefore R Studio) are not available in Arch Linux; even their Arch User Repos are broken. I've had serious problems contributing to R material and helping at R-based workshops because of this.

You might think, well, that's just one distro. However, it is an important and popular distro, and this problem with pandoc is furthermore symptomatic of a deeper problem: it is built on Haskell, which - while interesting and supported by many worthy nerds - is not robust in the slightest. We cannot rely on pandoc to not break in any other arbitrary distro or OS at any time.

I can only support a solution that is as robust and cross-platform as possible, with the least amount of dependencies or arbitrary "gotchas". That means dropping pandoc until such time as the Haskell community can get its act together on package management.

@ocefpaf

This comment has been minimized.

Show comment
Hide comment
@ocefpaf

ocefpaf Jun 25, 2015

Member

Just to be clear, regarding my comment in the mailing list, if people need pandoc we can remove the burden of creating the HTML using Travis-CI. Travis-CI does comes with a cost though. (Complex configs, unexpected broken builds, convoluted token generation, etc.) On the other hand, if people are OK leaving pandoc behind I do believe Jekyll is a better option.

PS: On an unrelated note 😜 , my favorite site generator is Mkdocs, but it is way to simple for SWC lessons 😒

Member

ocefpaf commented Jun 25, 2015

Just to be clear, regarding my comment in the mailing list, if people need pandoc we can remove the burden of creating the HTML using Travis-CI. Travis-CI does comes with a cost though. (Complex configs, unexpected broken builds, convoluted token generation, etc.) On the other hand, if people are OK leaving pandoc behind I do believe Jekyll is a better option.

PS: On an unrelated note 😜 , my favorite site generator is Mkdocs, but it is way to simple for SWC lessons 😒

@rgaiacs

This comment has been minimized.

Show comment
Hide comment
@rgaiacs

rgaiacs Jun 25, 2015

Contributor

Pandoc (and therefore R Studio) are not available in Arch Linux;

R Studio isn't available in others popular distro repositories. Users need to install it using the package provide at http://www.rstudio.com/products/rstudio/download/. Pandoc has similar issue and the package is provided at https://github.com/jgm/pandoc/releases.

I can try maintaining Pandoc for Arch after the summer.

We cannot rely on pandoc to not break in any other arbitrary distro or OS at any time.

I think this is true for any program.

I understand your point and I think that we find a way to workaround it.

I can only support a solution that is as robust and cross-platform as possible

Are you saying that you will only support Java based solutions?

Using Jekyll because of GitHub Pages only solve the problem because it is web based, i.e. you still have the problem to install Jekyll (or any other software) in some platforms.

Contributor

rgaiacs commented Jun 25, 2015

Pandoc (and therefore R Studio) are not available in Arch Linux;

R Studio isn't available in others popular distro repositories. Users need to install it using the package provide at http://www.rstudio.com/products/rstudio/download/. Pandoc has similar issue and the package is provided at https://github.com/jgm/pandoc/releases.

I can try maintaining Pandoc for Arch after the summer.

We cannot rely on pandoc to not break in any other arbitrary distro or OS at any time.

I think this is true for any program.

I understand your point and I think that we find a way to workaround it.

I can only support a solution that is as robust and cross-platform as possible

Are you saying that you will only support Java based solutions?

Using Jekyll because of GitHub Pages only solve the problem because it is web based, i.e. you still have the problem to install Jekyll (or any other software) in some platforms.

@cryptarch

This comment has been minimized.

Show comment
Hide comment
@cryptarch

cryptarch Jun 25, 2015

I can try maintaining Pandoc for Arch after the summer.

Haha, that would be awesome, although I don't think it will fix my underlying beef with Haskell, since I've also had xmonad break too many times ;)

you still have the problem to install Jekyll (or any other software) in some platforms.

But this can be done per-user with gems. I've never had any problems with anything based on Ruby.

cryptarch commented Jun 25, 2015

I can try maintaining Pandoc for Arch after the summer.

Haha, that would be awesome, although I don't think it will fix my underlying beef with Haskell, since I've also had xmonad break too many times ;)

you still have the problem to install Jekyll (or any other software) in some platforms.

But this can be done per-user with gems. I've never had any problems with anything based on Ruby.

@cboettig

This comment has been minimized.

Show comment
Hide comment
@cboettig

cboettig Jun 26, 2015

Member

At @gvwilson 's suggestion, posting my comments on kramdown here:


tl;dr use redcarpet instead of kramdown.

If we tell people they can "use markdown" they probably think Github-flavored markdown, e.g. that they write fenced code blocks with language indicated:

```python
import matplotlib
```

which is identical to pandoc, and thus familiar to RStudio users and all existing lessons in pandoc. I suspect users will be surprised to learn this doesn't work in Jekyll kramdown. Most users Googling syntax highlighting in Jekyll will find things saying they should not put code in markdown at all, but use Jekyll's Liquid format [1], like so:

{% highlight python %}
import matplotlib
{% endhighlight %}

Further confusing things. Least obvious might be the solution of "trailing styles"

~~~
import matplotlib
~~~
{: .language-ruby}

Is that really what we want? I realize Jekyll is now an independent community project from GitHub with some historical legacy issues, but I have no idea why it continues to default to a syntax that is so divergent from Github's own markdown style. Anyway, I would suggest we at least use redcarpet as the parser so that the code block notation we have been using in pandoc, Github, etc continues to work. Trying to patch all knitr and these other tools to use trailing styles does not seem consistent with the goal of using familiar out-of-the-box tools.

Member

cboettig commented Jun 26, 2015

At @gvwilson 's suggestion, posting my comments on kramdown here:


tl;dr use redcarpet instead of kramdown.

If we tell people they can "use markdown" they probably think Github-flavored markdown, e.g. that they write fenced code blocks with language indicated:

```python
import matplotlib
```

which is identical to pandoc, and thus familiar to RStudio users and all existing lessons in pandoc. I suspect users will be surprised to learn this doesn't work in Jekyll kramdown. Most users Googling syntax highlighting in Jekyll will find things saying they should not put code in markdown at all, but use Jekyll's Liquid format [1], like so:

{% highlight python %}
import matplotlib
{% endhighlight %}

Further confusing things. Least obvious might be the solution of "trailing styles"

~~~
import matplotlib
~~~
{: .language-ruby}

Is that really what we want? I realize Jekyll is now an independent community project from GitHub with some historical legacy issues, but I have no idea why it continues to default to a syntax that is so divergent from Github's own markdown style. Anyway, I would suggest we at least use redcarpet as the parser so that the code block notation we have been using in pandoc, Github, etc continues to work. Trying to patch all knitr and these other tools to use trailing styles does not seem consistent with the goal of using familiar out-of-the-box tools.

@aaren

This comment has been minimized.

Show comment
Hide comment
@aaren

aaren Jun 26, 2015

Member

One benefit of the new approach is that you can fork a lesson and make changes without leaving the browser, using Github's inbuilt editor.

I forked Greg's example repo and added some content, all using the browser: see https://github.com/gvwilson/using-jekyll/pull/3

This makes collaboration very easy.

Member

aaren commented Jun 26, 2015

One benefit of the new approach is that you can fork a lesson and make changes without leaving the browser, using Github's inbuilt editor.

I forked Greg's example repo and added some content, all using the browser: see https://github.com/gvwilson/using-jekyll/pull/3

This makes collaboration very easy.

@aaren

This comment has been minimized.

Show comment
Hide comment
@aaren

aaren Jun 26, 2015

Member

@cboettig I don't like trailing styles either, but it appears that you can still specify the language of the code block as in GFM: http://kramdown.gettalong.org/syntax.html#language-of-code-blocks

Member

aaren commented Jun 26, 2015

@cboettig I don't like trailing styles either, but it appears that you can still specify the language of the code block as in GFM: http://kramdown.gettalong.org/syntax.html#language-of-code-blocks

@aaren

This comment has been minimized.

Show comment
Hide comment
@aaren

aaren Jun 26, 2015

Member

@cboettig I looked at the code blocks a bit more. tldr: we can use the Kramdown GFM parser.

The default kramdown parser doesn't support backtick fences. It has a GFM parser (disabled but available by default) that does support them, and still appears to support trailing styles for the blockquotes (callouts etc.) - it does need hard wrapping disabling though. _config.yml needs to look like this.

We would still need the trailing styles for blockquote styling, which is unfortunate.

Member

aaren commented Jun 26, 2015

@cboettig I looked at the code blocks a bit more. tldr: we can use the Kramdown GFM parser.

The default kramdown parser doesn't support backtick fences. It has a GFM parser (disabled but available by default) that does support them, and still appears to support trailing styles for the blockquotes (callouts etc.) - it does need hard wrapping disabling though. _config.yml needs to look like this.

We would still need the trailing styles for blockquote styling, which is unfortunate.

@gvwilson

This comment has been minimized.

Show comment
Hide comment
@gvwilson

gvwilson Jun 26, 2015

Member
Member

gvwilson commented Jun 26, 2015

@gvwilson

This comment has been minimized.

Show comment
Hide comment
@gvwilson

gvwilson Jun 26, 2015

Member
Member

gvwilson commented Jun 26, 2015

@cboettig

This comment has been minimized.

Show comment
Hide comment
@cboettig

cboettig Jun 26, 2015

Member

@gvwilson The notation

```python

in this setup will just make html like

<pre><code class='language-python'>

so you could just define CSS for that class to code it as input. Output and error messages would have to be explicitly indicated as such in the markdown.

@gvwilson curious about your thoughts on syntax highlighting. I think SWC has opted primarily for just coding all input text in one color and output text in another color. I can only assume that research has shown that syntax highlighting helps spot errors; (though maybe that doesn't matter in this context?).

It looks to me like syntax highlighting would have to be done by javascript in kramdown (which I think is why Jekyll documents the Liquid syntax instead; crazy mess). Pandoc and redcarpet both generate markup around the actual code, allowing you to do syntax highlighting via css (e.g. using GitHub's css)

Member

cboettig commented Jun 26, 2015

@gvwilson The notation

```python

in this setup will just make html like

<pre><code class='language-python'>

so you could just define CSS for that class to code it as input. Output and error messages would have to be explicitly indicated as such in the markdown.

@gvwilson curious about your thoughts on syntax highlighting. I think SWC has opted primarily for just coding all input text in one color and output text in another color. I can only assume that research has shown that syntax highlighting helps spot errors; (though maybe that doesn't matter in this context?).

It looks to me like syntax highlighting would have to be done by javascript in kramdown (which I think is why Jekyll documents the Liquid syntax instead; crazy mess). Pandoc and redcarpet both generate markup around the actual code, allowing you to do syntax highlighting via css (e.g. using GitHub's css)

@gvwilson

This comment has been minimized.

Show comment
Hide comment
@gvwilson

gvwilson Jun 26, 2015

Member
Member

gvwilson commented Jun 26, 2015

@cboettig

This comment has been minimized.

Show comment
Hide comment
@cboettig

cboettig Jun 26, 2015

Member

@gvwilson I'm okay with anything, I was just curious if the lack of syntax highlighting was based on theoretical objections or practical implementation ;-).

From my perspective, I imagine most people are familiar with how markdown looks and displays on Github, and so it seems ideal to emulate that as much as possible. redcarpet is essentially GFM, such that you can just drop in the github-light.css style sheet and things should just work.

Github's Automatic Page Generator (https://help.github.com/articles/creating-pages-with-the-automatic-generator/) which does not use Jekyll is like this -- completely consistent with the markdown on Github. Kramdown isn't, and introducing yet another flavor with it's esoteric markings that are different than the markdown we write anywhere else doesn't make sense to me.

Redcarpet ships built in to jekyll, runs automatically on Github no travis nonsense or whatnot. GFM is essentially redcarpet: https://github.com/blog/832-rolling-out-the-redcarpet https://github.com/github/markup

Of course, regarding the input, output and error styling, GitHub' s sytnax highlighting implementation really isn't that great; and one might want to extend the css to make more explicit classes for output and errors.

Member

cboettig commented Jun 26, 2015

@gvwilson I'm okay with anything, I was just curious if the lack of syntax highlighting was based on theoretical objections or practical implementation ;-).

From my perspective, I imagine most people are familiar with how markdown looks and displays on Github, and so it seems ideal to emulate that as much as possible. redcarpet is essentially GFM, such that you can just drop in the github-light.css style sheet and things should just work.

Github's Automatic Page Generator (https://help.github.com/articles/creating-pages-with-the-automatic-generator/) which does not use Jekyll is like this -- completely consistent with the markdown on Github. Kramdown isn't, and introducing yet another flavor with it's esoteric markings that are different than the markdown we write anywhere else doesn't make sense to me.

Redcarpet ships built in to jekyll, runs automatically on Github no travis nonsense or whatnot. GFM is essentially redcarpet: https://github.com/blog/832-rolling-out-the-redcarpet https://github.com/github/markup

Of course, regarding the input, output and error styling, GitHub' s sytnax highlighting implementation really isn't that great; and one might want to extend the css to make more explicit classes for output and errors.

@twhitehead

This comment has been minimized.

Show comment
Hide comment
@twhitehead

twhitehead Jun 29, 2015

You might think, well, that's just one distro. However, it is an important and popular distro, and this problem with pandoc is furthermore symptomatic of a deeper problem: it is built on Haskell, which - while interesting and supported by many worthy nerds - is not robust in the slightest. We cannot rely on pandoc to not break in any other arbitrary distro or OS at any time.

@cryptarch I'm a bit surprised your experience with Haskell has been so bad. Is this a general thing with Haskell itself or primarily related to the issues you have had with the xmonad package? I ask as, although I've never used xmonad myself, I wouldn't be surprised it was a bit touchy being a low level unconventional piece of software (i.e., a niche tiling window manager with compile time configuration).

My experience over the past few years using pandoc under Debian for lecture notes (generating html, slides, and handouts from the same source) has been quite good (i.e., I can't recall it every having been broken). I would have expected the core of Haskell, that supported in the Haskell Platform on Mac, Windows, and Unix, to be quite stable. Especially given the amount of industry uptake.

twhitehead commented Jun 29, 2015

You might think, well, that's just one distro. However, it is an important and popular distro, and this problem with pandoc is furthermore symptomatic of a deeper problem: it is built on Haskell, which - while interesting and supported by many worthy nerds - is not robust in the slightest. We cannot rely on pandoc to not break in any other arbitrary distro or OS at any time.

@cryptarch I'm a bit surprised your experience with Haskell has been so bad. Is this a general thing with Haskell itself or primarily related to the issues you have had with the xmonad package? I ask as, although I've never used xmonad myself, I wouldn't be surprised it was a bit touchy being a low level unconventional piece of software (i.e., a niche tiling window manager with compile time configuration).

My experience over the past few years using pandoc under Debian for lecture notes (generating html, slides, and handouts from the same source) has been quite good (i.e., I can't recall it every having been broken). I would have expected the core of Haskell, that supported in the Haskell Platform on Mac, Windows, and Unix, to be quite stable. Especially given the amount of industry uptake.

@cryptarch

This comment has been minimized.

Show comment
Hide comment
@cryptarch

cryptarch Jun 30, 2015

@twhitehead It's been a few different things.

Part of it is just that Haskell package management in Arch Linux is a bit of a mess; as you can see on the Arch wiki, there are multiple approaches which conflict each other. If you want the stability of the core repos, you suffer the price of not many packages, and even then I seem to recall the core xmonad package randomly breaking every now and then.

The non-core Arch Haskell repo would seem to be a good alternative except the last time I checked (a few months ago) it no longer offers Pandoc.

Or perhaps one could just install Pandoc as a user into your home directory and not worry about having it as a system package, but then since cabal is not a package manager this always seems to result in conflicts.

Aside from the package management situation, I find certain aspects of the language itself a bit problematic, although probably not a deal breaker by themselves. For example, compare with R. In R, if a function uses too much memory, R will kill the function and tell you you're out of memory, but other than that everything's safe. In Haskell, an elementary task is building a vector that can contain a potentially infinite number of elements. If you then ask for the length of that vector, Haskell will gobble up all your memory and crash your computer without warning.

If I have to be worrying about computer memory, then I'd rather work in a language that lets me talk to the computer directly, such as C; C doesn't pretend that hardware constraints are not a problem. Haskell on the other hand seems to think that physical computational machines are beneath its notice.

cryptarch commented Jun 30, 2015

@twhitehead It's been a few different things.

Part of it is just that Haskell package management in Arch Linux is a bit of a mess; as you can see on the Arch wiki, there are multiple approaches which conflict each other. If you want the stability of the core repos, you suffer the price of not many packages, and even then I seem to recall the core xmonad package randomly breaking every now and then.

The non-core Arch Haskell repo would seem to be a good alternative except the last time I checked (a few months ago) it no longer offers Pandoc.

Or perhaps one could just install Pandoc as a user into your home directory and not worry about having it as a system package, but then since cabal is not a package manager this always seems to result in conflicts.

Aside from the package management situation, I find certain aspects of the language itself a bit problematic, although probably not a deal breaker by themselves. For example, compare with R. In R, if a function uses too much memory, R will kill the function and tell you you're out of memory, but other than that everything's safe. In Haskell, an elementary task is building a vector that can contain a potentially infinite number of elements. If you then ask for the length of that vector, Haskell will gobble up all your memory and crash your computer without warning.

If I have to be worrying about computer memory, then I'd rather work in a language that lets me talk to the computer directly, such as C; C doesn't pretend that hardware constraints are not a problem. Haskell on the other hand seems to think that physical computational machines are beneath its notice.

@gvwilson gvwilson closed this Jun 17, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.