Switch back to Jekyll

[shawnbot]

The whole Node server thing we've got going on is incurring an amount of technical debt that I'm not entirely comfortable with. I'd like to explore the idea of moving back to Jekyll so that we can leverage Federalist for deployment, and make the site easier to work with, especially for those of us without much Node experience. This involves:

1. Migrating any data that's being iterated over in templates to Jekyll data.
2. Evaluating whether Jekyll collections will solve the problem of deeply nested URL structures, like the /resources/:resource/:datatype/onshore/:state/:county hierarchy.
3. Restructuring the data build process to generate whatever full paths aren't covered by collections. If collections can't be made to generate those more complex URL structures, we would need to script the generation of them using the data as input. For instance, we would have a script that would generate a page for each combination of resource, data type, state and county in our data, and write it to the desired path.

One thing we'd get out of this would be a Sass stack that just works, rather than the troublingly broken node-sass-middleware package. Another would be static output, so we could host it on S3. And I think the advantage of creating a file structure that's more familiar would benefit everyone, since Jekyll is a well-documented thing that lots of people use and understand.

[shawnbot]

Ben Balter wrote up a nice guide to using Jekyll collections that's worth reading.

[shawnbot]

Here's another guide to using Jekyll without plugins.

[shawnbot]

As a step in this direction, I threw together a Node module last night for reading a Jekyll _data directory into a single JS object. My thinking here is that we can move the "structural" data elements like commodities.json, which define the structure of certain parts of the site, into our _data directory, where they can be treated equally by both Jekyll and our Node scripts that set up the rest of the data-driven scaffolding.

[shawnbot]

So, last week I created a repo to test Jekyll with various configurations of pages and data.

The master branch of that repo employs a Node script to build up part of the "resources" directory structure with empty HTML files with YAML front matter that describes each page's "context", all from data files in the _data directory. This includes every permutation of the following URLs:

/resources/:resource/
/resources/:resource/:datatype/
/resources/:resource/:datatype/onshore/:state/
/resources/:resource/:datatype/offshore/:area/

This hierarchy (which is not quite right, because some data types don't apply to some geographies) produces 1261 HTML files. However, this does not include all of the county-specific URLs, which would increase the total number of pages significantly. Already, though, this set of pages takes Jekyll 2.5.3 around 30-35 seconds to build, which means that you need to wait that long when running the server. Also, most changes to the _data directory would trigger a complete rebuild, which takes just a little bit less time once the server is already running (probably because some stuff is cached in memory, and doesn't have to be read from disk). This is way too long to test your work iteratively.

In the collections branch, I tried to represent the repeating directory structures (resources, data types, states, and offshore ares) as collections and create symlinks for each unique instance then describe those in the config, but that didn't work so well: build times were much faster, but I couldn't figure out how to get each "instance" of a collection to get its own metadata in the config.

Another avenue to consider is to see what Jekyll 3.0 can do with its incremental builds, but I'm not sure how to get that set up for testing. (@gemfarmer, maybe you could help me out with this?)

Anyway, right now I'm leaning toward turning the resources URL hierarchy into a client-side thing driven by data that's structured in the same way (more in #597). This would mean that we would just have one page for Jekyll, resources/index.html, that would do client-side hash routing with the same URLs as above but with a # after the /resources/ bit:

/resources/#:resource/
/resources/#:resource/:datatype/
/resources/#:resource/:datatype/onshore/:state/
/resources/#:resource/:datatype/onshore/:state/:county/
/resources/#:resource/:datatype/offshore/:area/

and the data would be broken down similarly:

/data/:resource/
/data/:resource/:datatype/
/data/:resource/:datatype/onshore/:state/
/data/:resource/:datatype/onshore/:state/:county/
/data/:resource/:datatype/offshore/:area/

The year (for the slider) would be passed along as a query string parameter in the hash, e.g.:

/resources/#oilgas/revenue/onshore/CA/?year=2010

We might also want to encode the map/chart tab selection in the URL, which could look like:

/resources/#oilgas/revenue/onshore/CA/?tab=map&year=2010

If anyone has any further thoughts at this point, I'm all ears!

[gemfarmer]

@shawnbot I just installed the jekyll 3 beta and time tested it in your jekyll-stress-test with good results – 1.559 seconds. Along with the incremental regeneration, this might be the way to go.

[shawnbot]

Whoa, nice @gemfarmer! Should we set do this with bundled then and set up a Gemfile?

[gemfarmer]

@shawnbot were you imagining that we would add jekyll to the existing project, or start a new jekyll project and add the existing site in piecemeal?

[shawnbot]

@gemfarmer I think we should migrate everything over in place. Here's my plan:

• Add a _config.yml and port over some of the global variables.
• Add most of the existing directories (views, etc.) to the config's exclude list so it doesn't try to build them.
• Get the styles ported over to the _sass directory so we have some CSS.
• Recreate the home page first by porting the existing templates over to _layouts and _includes.
• PR Wire static pages #610 for Wire up static pages for jekyll port #608: Next, work on the other static pages, for instance converting the resource to revenue pages to markdown if possible.

[shawnbot]

Okay, so I think we've done most of this. What do you think, @meiqimichelle? Should we wait to close this until we have the site running on Federalist in #604?

[meiqimichelle]

@shawnbot I don't think we need to wait until its running on Federalist since we have a separate issue for that, but are these changes reviewed/merged? I may have missed a PR.

[shawnbot]

@meiqimichelle nope, I'll file a PR. Or should we just push (rename, effectively) this branch to dev and start it that way?

[meiqimichelle]

From conversation in standup: we decided to file a PR against master because that will be the place we want Federalist to automagically push to S3 from. We'll then create dev and staging from master and starting using a dev > staging > master workflow, with features cued for acceptance off of staging.

[meiqimichelle]

Closed with #613 .