Moving to Bundler for local rendering of lesson pages

[ErinBecker]

Context

GitHub Pages is the name of the service that GitHub uses to convert files within a repository into a website. This conversion is done by Jekyll, and it's also what we use when we test our websites locally by typing make serve (or jekyll serve for some repos). However, GitHub Pages doesn't only use Jekyll, it also uses many Jekyll plugins that provide more functionality to the users. We use a handful of these plugins such as the GitHub metadata plugin that allows us to use the GitHub web API to use repository-specific information in our websites.

Jekyll is updated regularly, as are the many plugins that GitHub Pages uses. To make it easier to manage all of these plugins and their versions, GitHub Pages manages a Ruby "gem" (think of it as a package or a library), called "github-pages".

As part of their installation instructions to preview GitHub Pages locally, GitHub recommends using bundler. Bundler recreates the same exact environment that GitHub Pages uses on their server and ensures that the website generated locally will be identical to the one that will be generated on GitHub's servers.

Current Status

Some of our lessons (e.g. all of the Data Carpentry Genomics and Geospatial lessons) are already using bundler. The Lesson Infrastructure Committee has approved switching to bundler for managing Ruby gems in our lesson template - which will affect all current lessons and set the stage for easier management of gems for future lessons. Using bundler in our template will also allow us to use features that have been recently added to GitHub pages, such as remote themes.

François opened a Pull Request on behalf of the Lesson Infrastructure Committee to integrate bundler into our lesson template (carpentries/styles#428). We are opening this issue to alert Maintainers to these proposed changes, and to solicit questions and concerns from the Maintainer team. We particularly welcome feedback from any Maintainers using a Windows system, as installing bundler is more difficult on these systems (see below).

How will these changes affect me?

1. In order to render your lesson locally, you will need to install bundler on your system. You can find installation instructions on the GitHub Pages help site. For Windows users, please also look at the installation instructions on the Jekyll website.

2. As part of the updates that will implement this proposed change, a new file will be added at the root of your repository called "Gemfile". As part of the build process, a file called "Gemfile.lock" will be created. These files will be added to the .gitignore file already in your repository to ensure they don't get committed to the repo. Please do not manually add these files to your commits.

If you have any issues getting bundler setup on your system, please leave a note in this repo. @fmichonneau and I (and anyone else who would like to help!) will troubleshoot with you, and if there are remaining unsolved issues, we'll set up a special office hours session to walk people through the process.

Timeline

If you have any questions or concerns about this change, please post in this thread by Friday, 20 September midnight UTC. @fmichonneau, myself, and members of the @carpentries/lesson-infrastructure-committee will respond here to answer questions and address concerns. If you would like to comment on the technical implementation, you may also [leave feedback on the PR in the styles repo](link to be added) which implements this change.

EDITS

• 2019-09-13 06:47 UTC: adding link to PR that includes bundler to the styles repo

[fpsom]

I am generally in favor of this change. I have a Win10 laptop and been using bundle for quite some time now to have a local instance of a lesson (or other jekyll based sites). So having it baked-in the lesson templates would be very convenient (personally speaking).

[hoytpr]

I would discourage Windows users from using the Windows Subsystem for Linux as described on the Jekyll website because (among other things... like multiple errors) access to your drives or folders requires you to use a different path (e.g. /mnt/c/Users/<username>/Desktop). Installing Jekyll, Ruby, and bundler seems to have worked for me using GitBash. The folder designation in the lessons is then /c/Users/<username>/Desktop as expected. I know this seems very small, but it avoids a problem.

[ErinBecker]

Thank you @fpsom and @hoytpr for your feedback as Windows users! I'm making a note here to make sure we update the "getting started" instructions on the styles and lesson-template repos (not sure which one of those has the instructions at the moment!) to include information for Windows machines.

@fpsom - I know you said you've been using bundler for a while, so may not remember how you installed it, but any chance you remember whether you used GitBash or if you followed the instructions on the Jekyll website for install (or some third option)?

[fpsom]

Hi @ErinBecker . I do not remember if I used the official instructions or not, but I am currently using it through the Win10 WSL (Windows Subsystem for Linux). Not using GitBash as the alternative Shell for windows, was always a conscious choice for me, but I completely agree with @hoytpr assessment; using GitBash as the means for installing the bundler would have a minimal impact (if at all) to how the files/directories are going to be presented to the learners.

[hoytpr]

@fpsom, Originally my reply was going to say you could not access the "Ubuntu" files from the Windows explorer.exe GUI. However this changed in the May 2019 (Windows 10 1903) release. But it "continues to be unsupported and STRONGLY recommended against" and this post and this post on Github suggests it's a bad idea to edit Unix files from the Windows GUI.

My question is: Are accessing the /mnt/c/Users/<username>/etc from Windows explorer, or are you editing files within the WSL using Nano/Vi/or something else?

Thanks!

PS: The WSL was able to load all the software for the Genomics lessons needed to run the shell-based parts ("Option B"). Even RStudio ran when called (although it popped up in Windows explorer IIRC). Filezilla ran in the WSL using an X-windows software (Xming).

[fmichonneau]

Hi Peter, from a quick look at the posts you included they seem to describe the situation of using WSL to access Linux files on the same computer. Many Linux users also have a Windows installation and they were looking into options to access/edit these Linux files when they booted their windows installation. This is not something that is indeed recommended.

I don't think there are any of these types of issues if you are editing files from one of our repos from Windows directly or through WSL. There could be issues with line endings compatibilities but most modern text editors deal with this.

From my very shallow understanding of WSL, when you use it to edit a file, you are actually using a "Linux" editor. I think you can install any editor you'd like and change the default to match your favorite one.

[hoytpr]

Thanks François, maybe my post was unclear. Assuming @fpsom uses Jekyll from within the WSL on a Windows machine I wondered if there were any problems editing the lesson files using Windows-based software (like NotePad++). Probably not, but I saw warnings and want to be sure.

But if a Linux user is running Windows, then runs the WSL, and asks: "How to access Windows folders from Bash on Ubuntu on Windows" ... I'm probably lost. ☺

[fpsom]

Hi all! Thanks @fmichonneau and @hoytpr for the thoughts and comments - I'll try to provide my experience to the best of my abilities. For the short answer (and to avoid my meandering thoughts) scroll directly to the end.

I think that the main concern is "how to access the files" using WSL. Actually I think that there are two options currently supported;

1. Access the files that are being created within the main directory structure of the WSL. This is the standard Unix-based set of directories (copying from my Ubuntu-flavor WSL, it's acct/ boot/ data/ etc/ init* lib64/ media/ opt/ root/ sbin/ sys/ usr/ bin/ cache/ dev/ home/ lib/ lost+found/ mnt/ proc/ run/ srv/ tmp/ var/).
2. Access the files, through WSL, that exist within the Win10 main file structure. As @hoytpr said, this is basically a mounted directory (so in my case that's /mnt/c/).

If you are going with option #1, there is a way that you can access them through the Win10 File Explorer (as a Network-mounted drive \\wsl$\). This has been introduced fairly recently, and as such, I'm sure there are several issues that still need to be resolved.

If you are going with option #2, then you don't really need to do anything else. :) Most of the editors work as are (such as Notepad++), and the WSL takes care of the line endings (when such an issue exists).

My usual "workflow" goes as follows:

• Navigate in the WSL to the project directory I have in Win 10. For example, that might be /mnt/c/Users/fpsom/Documents/RCode/ChipSeq/.
• Run the tool / pipeline in this folder in WSL. The tools have been installed directly in the WSL and include tools installed through apt, conda or any other potential sources.
• Make any adjustments to code / config files / input files directly through Win10. Tools I've been using daily include Notepad++, RStudio, PyCharm and Eclipse.
• Re-run, etc etc

I've also successfully run Unix-tools that have a GUI (such as Gimp). However, this required installing an X11 server in windows, which required some ... creative tinkering, so I would definitely not recommend.

So, in short, and focusing exclusively on bundle, WSL and the lesson template, I have the md files in a directory within Win10, I make any edits with Atom and then run bundle in the correct folder (i.e. starting with /mnt/c/...) through the WSL. And it works fine. :)

[gcapes]

Whatever tool we use to build the site locally, our priority should be making it easy to use/install.

I think that most contributors to our lessons don't build them locally to preview their changes before submitting a PR, and it would be good if they did. One of the reasons people don't build the site locally could be that they don't know how to - it can be quite frustrating if you follow the carpentries' official guide on how to do this, but run into difficulties because the guide isn't good enough: carpentries/lesson-example#260

[ErinBecker]

I absolutely agree that it should be as simple as possible! I had trouble myself the first time I tried to do a local render, and eventually did get it working (but only after help from @rgaiacs). That was almost four years ago, and since then I've had at least three separate instances where my jekyll has stopped working and I've needed to re-install or update on or another of the dependencies (with help from @maxim-belkin, @rgaiacs, @gvwilson, and @fmichonneau). Without their help (and the basic necessity of needing to figure it out because of my staff role!), I would have been seriously discouraged from continuing to contribute to lessons.

My understanding is that using bundler will eliminate many of these issues and make it simpler for users to keep jekyll working once they get it going. I don't think this change will affect the difficultly level of initially getting jekyll set up on your machine. Is this accurate @fmichonneau?

[fmichonneau]

Yes, I don't think it'll be any more difficult than it currently is. It might even be easier for at least some people.

[gcapes]

That's good news, and while easier is better, my main concern is about the quality of the set up instructions rather than ease of installation (the latter follows the former to a large extent).

[fmichonneau]

@gcapes that's a good point, and I think this change is a good time to make sure that the installation instructions are clearer and more comprehensive.

[hoytpr]

Although I have bundler installed (apparently):
$ bundler --version
Bundler version 2.0.2

Starting with installing Ruby + Dev Kit, it was a very difficult process getting all the software installed, there were many errors and dependencies failing. However, I followed the "advice" ("you must install X before Y, or Try installing version N.NNN) at the end of each failure and installed/re-installed until bundle exec jekyll serve --baseurl '' finally worked (jekyll serve never worked). François then showed me that "serving to localHost" meant pointing my browser to 127.0.0.1:4000 to see the rendered site. I was thrilled until the Gem needed updating. Running gem update seemed to break everything. and the process seemed to start from scratch. After many trial and error efforts, I finally got the sites to render again, but with errors during the build (the _sass not being supported comes to mind). Finally I manually edited my gemfile by placing gem 'wdm', '>= 0.1.0 at the end. This was different than the gem 'wdm', '~> 0.1.1', :install_if => Gem.win_platform? on the jekyll site, and unfortunately, I forget where I saw the shorter version, but believe it's an updated version of the same parameter. Due to teaching I didn't have time to write it all down (like now).

I will volunteer to be a Guinea pig if anyone wants. I have a computer at home running Win10 1903 that does not have anything more than GitBash installed. I'd be willing to work from there to go through an install with someone who knows what should happen, or, go through the Jekyll page and report the errors encountered.

Installing through GitBash (if possible) would likely be easier for most Windows users, but if the WSL route is indicated, I would do that also, to help get some of the instructions validated for the installs.
Let me know. My naivetè makes me a good model.

[ErinBecker]

@gcapes @fmichonneau - I agree about using this opportunity to redo the instructions for local rendering.

@hoytpr - Thanks for offering yourself as a guinea pig. I'm also happy to be a tester. I just found out this morning that my jekyll isn't working again. 😦

[alee]

I use the docker-based make docker-serve to render content locally but as you describe this likely misses the github specific plugins provided by the github-pages gem.. is there interest in updating that build target?

[maxim-belkin]

+1 to moving to bundler
It is sometimes critical to use exactly the same gems as GitHub Pages. GitHub Pages lag behind in terms of updating its dependencies and we can't use new features of some gems (example: carpentries/styles#422)

[fmichonneau]

@alee thankfully, the jekyll docker image comes with bundler, and so we can update the target to use bundler. See the draft pull request: carpentries/styles#428 that implements this change. Comments on the PR are welcome!

[ErinBecker]

Thanks everyone for the feedback and discussion. To summarize the conversation above:

• No objections to moving to bundler.
• We will update the install / setup instructions along with these changes to make them clearer and more comprehensive.
• @ErinBecker and @hoytpr will test the new install / setup instructions.

Based on the feedback received, I'm closing this issue and we will proceed as described!

[fmichonneau]

A first draft of the installation instructions for bundler and a general overhaul of the setup instructions to be able to render the lessons locally is available here carpentries/lesson-example#276

Please provide feedback based on your past experience working with these tools or on your experience trying to use these installation instructions.

I'd especially be grateful for someone to tackle writing the installation instructions for macOS.

[ErinBecker]

@fmichonneau - thanks so much for putting together this draft. I'm going to be working on the macOS install instructions, but need to update my OS first. 😱