Arbitrary collection support

[benbalter]

Breaking out the notes from #1929 so we can discuss the idea with more granularity and (hopefully) track its implementation.

The basic idea is that right now Jekyll recognizes two types of content, posts and pages. Pages can live anywhere, and posts must live in a _posts folder, and must be named a with a specific file format (0000-00-00-title.md). Both posts and pages can have tags, but any tag or category YAML front matter from posts, makes its way into site.tags and site.categories. Essentially, we've got two ways to describe very similar things.

Instead, lets make the basic unit of Jekyll a CollectionObject (or something more creatively named). Think of this as similar to a Node in Drupal or a Post in WordPress. Out of the box, Jekyll will ship with Post and Page Collections initialized, but a user could introduce an infinite number.

CollectionObjects would contain the basic logic (yaml front matter, markdown, permalinks, etc), and could be extended by core collections. Posts for example, would have specific file names, Pages could live anywhere.

To implement a new collection (let's say "Puppies") a user would add the following to their config.yml file:

collections:
  - puppies

This would tell jekyll to look in the _puppies folder (_#{collection_name}) for markdown (or other) files. In that puppies folder, we could have rover.md and spot.md, and they'd be available as site.puppies (as would site.posts and site.pages continue to work). I could then iterate, filter, do whatever. Any file in the _puppies folder will be read in, with or without YAML front matter.

You could also have kittens as _kittens and cars as _cars just as easily:

collections:
  - puppies
  - kittens
  - cars

If you want each puppy rendered (e.g., site.com/puppies/rover.html (or puppies/rover/index.html) you'd add it to a render array:

collections:
  - puppies

render: 
  - puppies

Again, by default, core collections (pages, posts) are rendered just like before. The idea here is that if a user is using Posts for something other than blog posts, or hacking Pages to be categorized, Jekyll has already failed them. Instead of being categories of pages (or posts), they should be a Collection.

Here are the full, non-prose notes from the meatspace brainstorming session:

Collections

• If people are using blog posts for a non-blog post thing, Jekyll has already failed
• By default, Jekyll registers [posts, pages]
• Can describe additional collections in _config.yml
• Can describe render in _config.yml, which by default contains pages, posts
• Collections rendered, render to /collection/#{object} - maybe give it a path?
• Jekyll looks to “/_#{collection}”
• FIles read in as site.#{collection}
• Collection object basic functionality, extended by posts and pages methods
• Anything in _collections folder is processed as a thing, much like _posts
• Posts, pages extends Collection (Ruby Object) (or something)
• Posts read tags, categories in to site.tags and site.categories, have dates in file names
• Pages don’t live in _collections folder

h/t goes to @parkr for seeding the idea.

[konklone]

Makes sense to me. Why are "posts treated separately anyway? (I haven't used them, only pages.) Could you genericize this by merging "posts" back into "pages"?

[lukewil]

I think this is an absolutely fantastic idea. I use Jekyll to publish books on the web, where Jekyll page == book chapter, and with collections implemented this way every author would have a collection all his own.

Just from my days dealing with custom post types in WP, CollectionObject is probably a scary name from a new user's point of view, but it's a heck of a lot better than calling them post types and then having a built in post type called posts!

[benbalter]

Why are "posts treated separately anyway?

In practical terms, posts have dates and are sorted reverse chronologically. They also live in a _posts folder so that they can be identified as posts.

Could you genericize this by merging "posts" back into "pages"?

That's the idea. We'd have a basic CollectionObject (or whatever), which would be extended by Posts and Pages to provide use-case-specific functionality such as date handling.

Pages would be extended to allow them to live anywhere (e.g., not in a new _pages folder or something)... the existing behavior.

Posts would be extended to support date logic and to continue to pipe tag/category metadata into the site.tags and site.categories name spaces.

Think of it similar to WordPress's custom post type support, but for a static site.

[benbalter]

CollectionObject is probably a scary name from a new user's point of view

Agreed. I like Collections, which is intuitive to non-technical users, but the individual thing collected needs to be called something in the abstract... I just don't know what.

Glad to put up a personal bounty of a (cup | glass | mug | shot | pint) of your favorite beverage to anyone who thinks of the better name. 😄

[drinks]

I love this idea. Is Document too ambiguous a name for a collection's instance object?

[stevector]

Sculpin, a Jekyll-like tool built in PHP, just added a "Content Type" concept that sounds quiet like what is getting called a "collection" here. I suggested "content type" for Sculpin because I work primarily in Drupal and that is Drupal's name for the concept (well also "node type"). Here is the pull request @simensen just merged yesterday sculpin/sculpin#96

[scribu]

If people are using blog posts for a non-blog post thing

Example?

[stevector]

Hi @scribu, here's an example: the healthcare.gov "front end" which was built in Jekyll (http://developmentseed.org/blog/new-healthcare-gov-is-open-and-cms-free/) seemed to have a content-type-like concept within its _posts directory. The canonical repo was pulled down but I've still got a fork: https://github.com/stevector/healthcare.gov/tree/master/_posts

[simensen]

It is fun to see this thread here. I'd be happy to talk about Sculpin's implementation or answer questions about how we ended up with what we did.

In Sculpin, everything was always considered a Source (all of the files) and Posts were always just a special case of a Source. That is, Posts were actually Sources themselves. The "collection" of Posts was really just a special collection of sources.

Initially there was a lot of "post" specific classes created (PostsCollection, PostItem (that wrapped a Source)) but with the Content Types stuff I've finally boiled it down to just "Content Types" and you can now define them via a kernel configuration. It uses a lot of magic to determine defaults for things ( projects are found in _projects and automatically selects the layout "project" ) but most of that can be overridden.

I opted to handle taxonomies on a type-by-type basis rather than site-wide. This means that if you have a configuration like:

sculpin_content_types:
    posts:
        taxonomies: [tags, categories]
    projects:
        taxonomies: [tags]

... you end up with five data providers, posts, posts_tags, posts_categories, projects and projects_tags. Meaning taxonomies are grouped with their type.

Adding site-wide taxonomies after the fact would be easier than trying to make them global to begin with and later deciding, "we want to support a separate set of tags for posts and projects" down the road.

[troyswanson]

How would the "Collections" feature sort files within a directory? For instance, in the example @benbalter gave, how would you tell the system that you wanted "spot" to appear first and "rover" to appear second? This is the same question I posed in #670 and didn't really get a good answer.

[benbalter]

How would the "Collections" feature sort files within a directory?

Great question. I'd imagine alphabetically (or however the file system sorts it) within the site.#{collection} array, and ideally, we'd empower the edge to sort. E.g., a sort_by filter, to sort by the value of a given YAML property. Dumb core, smart edge. Allow for maximum flexibility across use cases.

[afeld]

How would the "Collections" feature sort files within a directory?

I'd imagine alphabetically (or however the file system sorts it) within the site.#{collection} array, and ideally, we'd empower the edge to sort

Yep, sounds consistent to me: #1848 and #1849

[simensen]

I wanted to ensure that collections could be sorted arbitrarily. One set (posts) might be pathname based, and another (say, projects) might be sorted based on title.

I was able to achieve this in a pretty nice way with the DiC I'm using (pasting the link to the commit below) but before I went that route I was considering making a bunch of sorters that you could configure:

collections:
  puppies:
    sort: meta
    meta_sort_key: breed
  kittens:
    sort: meta
    meta_sort_key: breed
    direction: ascending
  cars:
    sort: meta
    meta_sort_key: [size, name]
  posts:
    sort: filename
    direction: ascending

By having the collection type as a keyed array, you can have more flexibility down the road. For example, you could set the default layout or permalink strategy based for the type.

collections:
  puppies:
    sort: meta
    meta_sort_key: breed
    layout: puppy
    permalink: animals/puppies/:breed/:name

sculpin/sculpin@5043086

[troyswanson]

I'm afraid I have to be blunt - that solution is a pain in the ass.

The way I've seen a lot of file-based CMS's handle sorting is to use an integer value at the beginning of the filename. So, for instance, the file structure might look like this:

/_puppies
    01-spot.md
    02-rover.md

I suppose you could leave the sort_by filter in there in case there is a real reason for sorting by some random value (that isn't something obvious like a key/value pair like order: 2). Imo, this method eliminates so much configuration and annoyance that will very likely crop up as a result of this.

EDIT:
@simensen, I wasn't referring to your solution as a pain. That said, I'm firmly of the mind that the complexity of your solution is entirely too high for what most use cases of Jekyll are (anecdotally, obviously - I have no data around that claim). I'm probably being idealistic when it comes to simplicity, but this all seems very difficult to set up and train folks on. I'm not usually a fan of ✨ magic, but when it's obvious what is happening, it's simple, and it's consistent, I'm for it.

[budparr]

I use posts for anything date-based and segregate various post types by category. I use category folders ('/category-name/_posts/yyyy-mm-dd-post-name.md') for organization and content creator convenience. Sounds very much like I would be able to replace that set up with collections, but does collections care if the collectionObject is date-based, or not, or if a collection has mixed content?

I'm working on a project where several different posts types will fit into a common category (or would-be collection): So, under a high-level category I have multiple date-based discussion posts surrounding a common (to a group) date-based question (and possibly even pages with background information). I'm separating them out using posts and pages along with categories. I think collections would help here because it gives me a higher level hierarchy to start with because I could have categories within collections (that is, be able to filter posts down from collection to categories within the collection.

The current (in progress and subject to change) hierarchy is here: https://github.com/sonnetmedia/prototype-iliadx-aba/tree/master/international-human-rights-law

Hope that's helpful to the discussion. Your explanation is certainly helpful after I puzzled over the original vision post for a while!