Upgrade to jekyll 4.x #46

0xdevalias · 2020-02-06T01:47:35Z

Currently we're on v3.8.0 (ref):

https://jekyllrb.com/news/2018/04/19/jekyll-3-8-0-released/
https://jekyllrb.com/news/2018/05/01/jekyll-3-8-1-released/
https://jekyllrb.com/news/2018/05/19/jekyll-3-8-2-released/
https://jekyllrb.com/news/2018/06/05/jekyll-3-8-3-released/
https://jekyllrb.com/news/2018/09/19/security-fixes-for-3-6-3-7-3-8/
https://jekyllrb.com/news/2018/11/04/jekyll-3-8-5-released/
https://jekyllrb.com/news/2019/07/02/jekyll-3-8-6-released/
https://jekyllrb.com/news/2019/08/20/jekyll-4-0-0-released/
- Ruby 2.4.0 or greater is now required.
- Rouge 3.0 or greater is now required for syntax highlighting.
- Jekyll builds should be much faster.
- Kramdown 2.1 is now the default markdown engine.
- Sass processing should be faster.
- We dropped support for a lot of stuff, specifically:
  - Pygments
  - RedCarpet
  - RDiscount
  - https://jekyllrb.com/docs/configuration/markdown/
    - Kramdown is the default Markdown renderer for Jekyll.
https://jekyllrb.com/news/2020/05/08/jekyll-4-0-1-released/
https://jekyllrb.com/news/2020/05/27/jekyll-4-1-0-released/

Changelog:

https://jekyllrb.com/docs/history/

Upgrading:

See also:

Explore alternatives to Jekyll (Hugo, etc) #28

0xdevalias · 2020-07-21T01:34:13Z

Noting old gem version incompatibilities here for reference:

You have requested: jekyll ~> 4.1.0 The bundle currently has jekyll locked at 3.8.0. Try running 'bundle update jekyll'
jekyll-compose was resolved to 0.8.0, which depends on jekyll (~> 3.0)
- https://github.com/jekyll/jekyll-compose/releases
- v0.12+ seems to support jekyll 4

This was done after updating a couple of things already:

⇒  bundle outdated
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...

Outdated gems included in the bundle:
  ..snip..
  * jekyll (newest 4.1.1, installed 3.8.7, requested ~> 3.8.7) in group "default"
  * jekyll-admin (newest 0.10.2, installed 0.8.0) in group "jekyll_plugins"
  * jekyll-feed (newest 0.15.0, installed 0.9.3) in group "jekyll_plugins"
  * jekyll-github-metadata (newest 2.13.0, installed 2.9.4) in group "jekyll_plugins"
  * jekyll-redirect-from (newest 0.16.0, installed 0.13.0) in group "jekyll_plugins"
  * jekyll-sass-converter (newest 2.1.0, installed 1.5.2)
  * jekyll-seo-tag (newest 2.6.1, installed 2.4.0) in group "jekyll_plugins"
  * jekyll-sitemap (newest 1.4.0, installed 1.2.0) in group "jekyll_plugins"
  * jekyll-twitter-plugin (newest 2.1.0, installed 2.0.0) in group "jekyll_plugins"
  * jekyll-webmention_io (newest 3.3.5, installed 2.9.1) in group "jekyll_plugins"
  ..snip..

Resolved?

Ruby Sass has reached end-of-life and should no longer be used.

* If you use Sass as a command-line tool, we recommend using Dart Sass, the new
  primary implementation: https://sass-lang.com/install

* If you use Sass as a plug-in for a Ruby web framework, we recommend using the
  sassc gem: https://github.com/sass/sassc-ruby#readme

* For more details, please refer to the Sass blog:
  https://sass-lang.com/blog/posts/7828841

This is a dependency of jekyll-sass-converter which is bundled with jekyll. The latest version appears to correctly update this to use sassc as recommended:

https://github.com/jekyll/jekyll-sass-converter/blob/master/jekyll-sass-converter.gemspec#L19
- spec.add_runtime_dependency "sassc", "> 2.0.1", "< 3.0"

0xdevalias · 2020-07-21T03:34:43Z

⇒  bin/serve
Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
Require just the needed backports instead, or 'backports/latest'.

..snip..

      Generating...
   GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

       Jekyll Feed: Generating feed for posts
Markdown processor: "redcarpet" is not a valid Markdown processor.
                    Available processors are: kramdown

  Conversion error: Jekyll::Converters::Markdown encountered an error while converting '_posts/2011-09-16-etc-initd-blog-start.md':
                    Invalid Markdown processor given: redcarpet
             ERROR: YOUR SITE COULD NOT BE BUILT:
                    ------------------------------------
                    Invalid Markdown processor given: redcarpet
                    ------------------------------------------------
      Jekyll 4.1.1   Please append `--trace` to the `serve` command
                     for any additional information or backtrace.
                    ------------------------------------------------
GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

Resolved?

Doing require 'backports' is deprecated and will not load any backport in the next major release. Require just the needed backports instead, or 'backports/latest'.

backports is required by sinatra-contrib and packable
- packable is required by nmatrix: packable (~> 1.3, >= 1.3.5), any version of backports
- sinatra-contrib is required by jekyll-admin: sinatra-contrib (~> 1.4), backports (>= 2.0)

My guess is this deprecated usage will be part of nmatrix since it's so old/seemingly unmaintained.

Resolved?

Markdown processor: "redcarpet" is not a valid Markdown processor. Available processors are: kramdown

Support for redcarpet and other markdown processors was dropped. Need to update our usages to kramdown.

Our Jekyll config will likely require changes too:

redcarpet:
  extensions:
    - "tables"
    - "autolink"
    - "strikethrough"
    - "space_after_headers"
    #- "with_toc_data"
    - "fenced_code_blocks"

https://kramdown.gettalong.org/parser/gfm.html
- https://github.com/kramdown/parser-gfm
- https://github.com/kramdown/parser-gfm#options
- this seems to support strikethrough using double tildes: ~~like this~~
https://kramdown.gettalong.org/converter/html.html#toc
https://kramdown.gettalong.org/syntax.html#tables
https://kramdown.gettalong.org/syntax.html#fenced-code-blocks
- by default this uses a different style than GFM, but the GFM parser supports triple backticks as expected
https://kramdown.gettalong.org/syntax.html#automatic-links
- Autolink in kramdown? barryclark/jekyll-now#459 (comment)
- this seems to require explicit syntax that differs from our previous GFM-styled usage
- Resolved?
  - eg. http://127.0.0.1:4000/devalias/2018/05/13/usb-reverse-engineering-down-the-rabbit-hole/
  - see Autolinking not working like GFM kramdown/parser-gfm#17

Kramdown seems to have a number of other issues too, eg:

not autolinking
linking syntax being rendered as tables when text includes |
embedded gist doesn't seem to load
- Resolved? yes
- this might be because my github username changed?
- it also seems to render as 404: Not Found in the generated HTML now.. so we could potentially find it that way?
- http://127.0.0.1:4000/devalias/2015/04/20/dogedraw-now-with-more-nyaan/
- http://127.0.0.1:4000/devalias/2014/05/15/java-scala-future-promise-map-headsplode/
- _posts/2013-07-13-nmap-sh-saving-precious-seconds.md
- _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md
- _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md
- _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md
- _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md
- _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md
- _posts/2014-05-15-java-scala-future-promise-map-headsplode.md
- _posts/2015-04-20-dogedraw-now-with-more-nyaan.md

Not specifically Kramdown related.. but:

Twitter urls being rendered by lazy twitter plugin even when inside codeblocks
- Resolved? yes
- http://127.0.0.1:4000/devalias/2017/07/16/bugcrowd-levelup-2017/

Resolved?

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

https://github.com/jekyll/github-metadata/blob/master/docs/authentication.md
- JEKYLL_GITHUB_TOKEN
- ~/.netrc
- OCTOKIT_ACCESS_TOKEN
- .env

With my dotfiles layout, I believe we could also add this to ~/.localrc.

I think using a gitignored .env.local file, and also including a .env.example or similar would probably be useful.. if supported.

0xdevalias · 2020-07-21T04:10:29Z

Resolved?

/Users/devalias/dev/_sites/devalias.net/_plugins/lazy_tweet_embedding.rb:11: warning: calling URI.open via Kernel#open is deprecated, call URI.open directly or use URI#open

It seems as though this may be the original source for this plugin:

https://github.com/takuti/jekyll-lazy-tweet-embedding

It hasn't been updated since 2017, and requires manually copying the plugin source into our project. We might be better off just removing it and manually using a twitter filter around links instead?

0xdevalias · 2020-07-21T23:16:52Z

Kramdown seems to have weird quirks that don't really match to GitHub Flavoured Markdown (GFM) as I would have expected it.

An alternative would be to use commonmark (which also appears to be what GitHub pages uses), but there seem to be 2 main versions that appear to use the same underlying commonmarker dependency, and both plugins seem somewhat outdated/unmaintained.

https://github.com/jekyll/jekyll-commonmark
- Updated in 2020, seems active
- How does this differ from jekyll-commonmark-ghpages? jekyll/jekyll-commonmark#39 (How does this differ from jekyll-commonmark-ghpages?)
https://github.com/github/jekyll-commonmark-ghpages
- Updated in 2019, doesn't seem active
- How does this differ from jekyll-commonmark? github/jekyll-commonmark-ghpages#17 (How does this differ from jekyll-commonmark?)
- Is this project still actively maintained, or is it abandoned? github/jekyll-commonmark-ghpages#18 (Is this project still actively maintained, or is it abandoned?)
https://github.com/gjtorikian/commonmarker
- Updated in 2020, seems active
https://github.com/github/cmark-gfm
- Updated in 2020, seems active

The github/pages-gem claims that it provides gem versions in line with what GitHub pages uses, but according to github/pages-gem#699 it seems to use Jekyll's default parser (Kramdown), rather than the (presumably still accurate, but seemingly unmaintained) CommonMarkGhPages gem.

Though as I note in github/jekyll-commonmark-ghpages#18:

Though looking at the pages-gem readme shows that we can also check the GH pages dependency versions online:

https://pages.github.com/versions/

Which shows a usage of jekyll 3.8.7 and jekyll-commonmark-ghpages 0.1.6, which appears to be the latest released version of this project.

commonmarker configuration:

I explore some relevant sounding choices in github/pages-gem#699 (comment), summarised as:

commonmark:
  options: ["FOOTNOTES", "STRIKETHROUGH_DOUBLE_TILDE", "VALIDATE_UTF8", "GITHUB_PRE_LANG", "HARDBREAKS"]
  extensions: ["autolink", "strikethrough", "table", "tasklist", "tagfilter"]

When I don't have UNSAFE enabled, some of my existing pages fail, instead outputting .

The main page that I noticed this on was _posts/2017-07-16-bugcrowd-levelup-2017.md, which appears to use syntax like the following:

{% raw %}
<pre>
Some text here.
</pre>
{% endraw %}

Setting the UNSAFE option 'fixes' this.. but probably isn't an ideal choice.

Searching for  in the generated pages shows up a number of offending pages:

_site/author/devalias/rss.xml
_site/coaching/index.html
- seems to strip the tags from <a href="mailto:glenn+coaching@devalias.net?subject=Bulletproof%20Coaching">glenn+coaching@devalias.net</a>
_site/devalias/2012/01/10/plesk-adding-programs-to-a-chrooted-ssh/index.html
- may have been caused by a typo on a triple backtick code fence?
_site/devalias/2013/06/17/gists-on-tumblr/index.html
_site/devalias/2013/07/13/nmap-sh-saving-precious-seconds/index.html
_site/devalias/2013/08/03/rails-lessons-learned-the-hard-way-1-db-migrate/index.html
_site/devalias/2013/08/10/erpscan-automator-because-manual-is-meh/index.html
_site/devalias/2013/08/29/reversing-powershell-securestring-for-fun-and-profit/index.html
_site/devalias/2013/11/21/vfeed-wrapper-helper-scripts-for-speed-and-efficiency/index.html
_site/devalias/2014/04/02/hacking-unicoins-for-fun-and-profit/index.html
_site/devalias/2014/05/15/java-scala-future-promise-map-headsplode/index.html
_site/devalias/2015/04/20/dogedraw-now-with-more-nyaan/index.html
_site/devalias/2016/08/24/starting-a-new-web-application-1-an-exploration-of-options/index.html
_site/devalias/2016/09/07/graphql-why-you-should-care/index.html
_site/devalias/2017/07/16/bugcrowd-levelup-2017/index.html
- seems to strip the <a> anchor tags for heading links
- as well as {% raw %}<pre>foo</pre>{% endraw %} usages (which would be better replaced by triple backticks anyway)
_site/devalias/2018/05/13/usb-reverse-engineering-down-the-rabbit-hole/index.html
- seems to strip the <a> anchor tags for heading links
_site/devalias/2018/09/15/forming-serverless-clouds-aws-cloudformation-sam-cdk-amplify/index.html
_sites/devalias.net/_site/now/index.html
- strips the <center> tags and {% include goodreads_reading.html %}
- strips {% include trakt_watching.html %}
_site/personal/interests/dance/index.html
as well as pretty much every one of the pages in _sites/devalias.net/_site/tag/*/rss.xml

Based on this, it seems like we may have to use UNSAFE. Though given this is a static site I control, that's probably not the worst thing in the world.

It also appears that the tagfilter extension breaks some of our usages in pages/templates/includes, for examples the scripts/tags used in the goodreads/watching embeds in now.md

{% include goodreads_reading.html %}
{% include trakt_watching.html %}

Again, given I control the source/inputs to this, there probably isn't a lot of real world risk in removing the tagfilter extension as well.

According to github/pages-gem#651, the best recommended way to support Jekyll 4 on GitHub pages currently is to use GitHub actions:

https://jekyllrb.com/docs/continuous-integration/github-actions/
- https://github.com/marketplace/actions/jekyll-actions
  - A GitHub Action to build and publish Jekyll sites to GitHub Pages
  - https://github.com/MichaelCurrin/jekyll-actions-quickstart
https://sujaykundu.com/blog/post/deploy-jekyll-using-github-pages-and-github-actions

0xdevalias · 2020-07-22T03:01:22Z

Fixed up most of the broken gist embeds, but one page still appears to be having issues, seems it may be a bug in jekyll-gist:

noscript output seems to break sometimes when using Commonmark jekyll/jekyll-gist#82

0xdevalias · 2020-07-22T04:20:23Z

We have previously used an inlined plugin called postfiles, that appears as though it was copied from:

https://github.com/indirect/jekyll-postfiles/blob/master/_plugins/postfiles.rb

This is used with syntax such as the following:

 ![who dares enter my domain]({% postfile cat.jpg %})

And appears to be used in the following posts:

_posts/2011-12-30-who-dares-enter-my-domain.md
_posts/2013-08-16-rockmelt-blog-rockmelt-browser-retirement-july.md
_posts/2014-02-13-y-u-no-update-kali.md
_site/devalias/2011/11/07/bye-bye-russian-spam/index.html
_site/devalias/2013/11/28/thecus-nas-devalias-says-no/index.html

Both of the .html files appear to contain it as the 'prev/next post' excerpt, so isn't really of concern to us here.

There appears to be a maintained (last commit 2020, last release 2019), gem based jekyll plugin that also provides this functionality, though it differs slightly in it's usage:

https://github.com/nhoizey/jekyll-postfiles

Instead of using a _postfiles folder, it appears to just use per-post folders under the _posts directory.

Instead of having to explicitly use a liquid filter (eg. ![who dares enter my domain]({% postfile cat.jpg %})), we can just use a regular markdown image tag with a relative path (eg. ![who dares enter my domain](cat.jpg))

It's probably a better choice to use going forward.

Resolved?

0xdevalias mentioned this issue Jul 22, 2020

Upgrade Jekyll to 4.x #84

Merged

0xdevalias linked a pull request Jul 22, 2020 that will close this issue

Upgrade Jekyll to 4.x #84

Merged

0xdevalias pinned this issue Jul 22, 2020

0xdevalias closed this as completed in #84 Jul 22, 2020

0xdevalias unpinned this issue Jul 22, 2020

0xdevalias mentioned this issue Jul 22, 2020

Dockerize Jekyll + automate site builds #27

Open

0xdevalias self-assigned this Jul 22, 2020

0xdevalias mentioned this issue Oct 5, 2020

Re-enable webmentions support #92

Open

aureliojargas mentioned this issue Oct 5, 2020

Update to Jekyll 4.x (kramdown) or move away? aureliojargas/aurelio.net#71

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to jekyll 4.x #46

Upgrade to jekyll 4.x #46

0xdevalias commented Feb 6, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 22, 2020

0xdevalias commented Jul 22, 2020 •

edited

Upgrade to jekyll 4.x #46

Upgrade to jekyll 4.x #46

Comments

0xdevalias commented Feb 6, 2020 • edited

0xdevalias commented Jul 21, 2020 • edited

0xdevalias commented Jul 21, 2020 • edited

0xdevalias commented Jul 21, 2020 • edited

0xdevalias commented Jul 21, 2020 • edited

0xdevalias commented Jul 22, 2020

0xdevalias commented Jul 22, 2020 • edited

0xdevalias commented Feb 6, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 21, 2020 •

edited

0xdevalias commented Jul 22, 2020 •

edited