Collections for directory indexes

[ollicle]

While attempting to translate my old Moveable Type generated blog to 11ty I’ve hit a snag. The posts appear in directories based on post dates e.g.
/2014/jan/04/eg-post-1.html

Each of these sub directories include indexes which list the posts whose date shares the same year, month, day. Example of the desired file output:

/2014/
	index.html  <= (lists: eg-post-1.html, eg-post-2.html, eg-post-3.html, eg-post-4.html)
	jan/
		index.html  <= (lists: eg-post-1.html, eg-post-2.html, eg-post-3.html)
		04/
			index.html  <= (lists: eg-post-1.html)
			eg-post-1.html
		05/
			index.html  <= (lists: eg-post-2.html, eg-post-3.html)
			eg-post-2.html
			eg-post-3.html
	feb/
		index.html  <= (lists: eg-post-4.html)
		25/
			index.html  <= (lists: eg-post-4.html)
			eg-post-4.html

So far I’ve attempted to do this by iterating over a large set of years/months/days, creating a glob based collection for each. By deriving the collection names from the date I hoped to later correlate these collections with each index template via its inputPath.

This is not working, but aside from that, generating these collections is really slow which makes it impractical anyway.

Is there a more effective approach to generate directory indexes for date derived sub directories with 11yt?

Thanks in advance for your ideas.

[kleinfreund]

To be clear here, the output you showed above is what you expect the output to be, correct? The following is the structure of the content that should appear in your output directory, yes?

• 2014/
  • index.html (lists: eg-post-1.html, eg-post-2.html, eg-post-3.html, eg-post-4.html)
  • jan/
    • index.html (lists: eg-post-1.html, eg-post-2.html, eg-post-3.html)
    • 04/
      • index.html (lists: eg-post-1.html)
      • eg-post-1.html
    • 05/
      • index.html (lists: eg-post-2.html, eg-post-3.html)
      • eg-post-2.html
      • eg-post-3.html
  • feb/
    • index.html (lists: eg-post-4.html)
    • 25/
      • index.html (lists: eg-post-4.html)
      • eg-post-4.html

Assuming the following structure of input files …

• src/
  • posts/
    • 2014-01-04-eg-post-1.md
    • 2014-01-05-eg-post-2.md
    • 2014-01-05-eg-post-3.md
    • 2014-02-25-eg-post-4.md
    • posts.json

… and the posts having this structure …

---
title: Title 1
date: 2014-01-04
---
# Title 1

… and the posts.json file looking like this …

{
  "layout": "base.liquid",
  "permalink": "{{ page.date | date: '%Y' }}/{{ page.date | date: '%m' }}/{{ page.date | date: '%d' }}/{{ page.fileSlug }}/index.html"
}

… you can achieve the desired output structure.

For more information on what you can do with permalinks, read https://www.11ty.io/docs/permalinks.

[ollicle]

Thanks @kleinfreund,
my outline of files (above) is the output produced from my old blog and what I am aiming to output.
I am transforming input:
2014/jan/04/post-slug.html

to output:
2014/jan/04/post-slug/index.html

This is straightforward. My difficulty is with the index files in the parent directories:

output:

2014/index.html
2014/jan/index.html
2014/jan/04/index.html

I was under the impression (apart from pagination) 11ty didn’t provide a means to generate multiple output files from a single template. I’ll be super excited if I’m wrong!?

The index files I’m after are not the post itself, but rather a list of links to posts contained by the each sub directory (e.g. https://ollicle.com/2007/, https://ollicle.com/2007/jun/). I’m looking for a means to generate and update these without manually editing each index.liquid template.

My input currently includes these subdirectories. Partly because I was assuming I needed template files in place to generate the sub-directory – mostly because I am transforming my old output to be my new input and I’m lazy:

src/
  2014/
    index.liquid  <= (*do I need a template file here to get a corresponding output directory index?)
    01/
        index.liquid <= (as above*)
        04/
          index.liquid <= (as above*)
          eg-post-1.html

I hope that is clearer.

[jevets]

Did you have a look at this #332, especially Zach's comment?
(It's not exactly what you're after but may help you get there.)

I was under the impression (apart from pagination) 11ty didn’t provide a means to generate multiple output files from a single template. I’ll be super excited if I’m wrong!?

I don't think you're wrong there.

But you could create a custom collection / object structure that multiple templates can consume and use includes to keep templates lean.

Maybe you could have a postsArchive custom collection, structured kinda like below, then set up a dynamic template file (using pagination and permalinks) for year, another for year-month, and another for year-month-day. Each template file can grab the appropriate data and include a post list component.

eleventyConfig.addCollection('postsArchive', (collection) => {
  // return posts mapped into the below kinda structure:
})

[
  2014: {
    jan: [{ post1 }, { post2 }],
    feb: [{ post1 }, { post2 }],
    may: [{ post1, post2, post3 }],
  },
  2015: {...},
  2016: {...},
]

---

If it were me, I'd consider compiling a single HTML archive for each year (or year-month if many posts), attaching HTML IDs (anchors) for each dated section, then 301 redirect those legacy URLs to their corresponding anchors on the single archive page, something like

# /2014/index.html

<h1>2014</h1>

<h2 id="jan">January 2014</h2>

<h3 id="jan-04">January 4th, 2014</h3>
<ul><li>[... posts]</li></ul>

<h2 id="feb">February 2014</h2>
<h3 id="feb-12">February 12th, 2014</h3>
<ul><li>[... posts]</li></ul>

redirect /2014/01/04 >>> /2014/index.html#jan-04 
redirect /2014/02 >>> /2014/index.html#feb

[ollicle]

Thank-you @jevets,
I had not seen #332. Appears the fundamental mistake I have been making is building multiple flat collections rather than one structured one. Will give it a fresh go.
Also thanks for your suggestions regarding anchoring the months. I will certainly take this opportunity to re-consider the structure.

[edwardhorsford]

@ollicle I think this is similar to #316 - I posted some code there that I use to create collections around dates.

[zachleat]

Thank you everyone for participating here! @ollicle did this get you far enough along or do you want to pivot this issue into a feature request?

[ollicle]

Sorry @zachleat life has intruded on my opportunities to try this thus far. Feel free to close for now. The code solutions generously offered seem feasible at a glace.
Index templates seem like something that would be nice to make simpler, although I feel inadequate with my grasp on how this would fit into the bigger 11ty picture to offer useful suggestions at this point.

[ollicle]

I’ve adapted the approach suggested by @edwardhorsford but I am struggling to see output collection.

Running DEBUG=Eleventy* eleventy I can see the contentByMonth collection has been created:

Eleventy:TemplateMap Collection: collections.all size: 418 +3ms
Eleventy:TemplateMap Collection: collections.post size: 192 +2ms
Eleventy:TemplateMap Collection: collections._port size: 191 +1ms
Eleventy:TemplateMap Collection: collections.contentByMonth size: 1 +49ms

Specifically how do I eye-ball the content of a collection, presuming it is not in the shape I am expecting.

Does anyone have general debugging strategies when building and using custom collections such as this?

[Ryuno-Ki]

Hm, Nunjucks has dump for that.
However, it looks like that there will no come a JSON filter to Liquid.

You could define one as filter, though.

[edwardhorsford]

@ollicle I'm assuming you modified my code since it was somewhat specific to my usage of tags.

My first step might be to put some console logs in the js generating the collection - so you can see what data structure you're giving to eleventy.

A couple things I do / have set up on the Nunjucks side:

A debug filter which logs the thing to the server console. FYI this can be rather long for eleventy objects. So: {{collections.contentByMonth | debug}}. You might also try {{collections.contentByMonth | length}} to get the size.

Or iterate through them:

{% for month, data in collections.contentByMonth %}
{{month}}: {{data | length}}
{% endfor %}

[ollicle]

Thank-you @Ryuno-Ki and @edwardhorsford you guys are most excellent!

I switched to nunjucks and got console logging and now I am getting somewhere.

{{thing | dump}} proved problematic – JSON.stringify complained with circular reference errors.

@edwardhorsford I had indeed changed your code, although insufficiently as it turned out. Not sure I have the collections correct yet - but now I can see the output things are looking up.

I took your iteration idea a step further :)

<details>
  <summary>Item data</summary>
    {% for key, value in post %}
      {% if ''+value == '[object Object]' %}
        <details>
          <summary>{{key}}</summary>
            <ul>
              {% for k, val in value %}
                <li><strong>{{k}}</strong>:{{val}}</li>
              {% endfor %}
            </ul>
          </details>
      {% else %}
          <p><strong>{{key}}</strong>:<span>{{value}}</span> </p>
      {% endif %}
  {% endfor %}
</details>

[edwardhorsford]

@ollicle try javascript-stringify to make your own stringify filter - the default JSON.stringify doesn't work for some reason - see #266.

I prefer going the console than the page though.

[ollicle]

Thanks again @edwardhorsford for sharing the paths you have travelled on this.
On that note. For the purpose of year/month lists of posts I have got the basic custom collection generation working.

The ah-ha moment for me has been understanding:

1. That the collection API methods output an array of template objects (posts).
2. To make a custom collection based on an existing collection one can take one of these arrays, do what we like with it before plugging it with a name into a eleventyConfig.addCollection call.

contentByDate.js:

const moment = require("moment");

function sortByDate(a, b) {
	return b.date - a.date;
}

function makeDateFormatter(datePattern) {
	return function (date) {
		return moment(date).format(datePattern);
	}
}

function generatePostDateSet(posts, dateFormatter) {
	const fomattedDates = posts.map(item => {
		return dateFormatter(item.data.page.date);
	});
	return [...new Set(fomattedDates)];
}

function getPostsByDate(posts, date, dateFormatter){
	return posts.filter(post => {
		return dateFormatter(post.data.page.date) === date;
	}).sort(sortByDate);
}

const contentByDateString = (posts, dateFormatter) => {
	return generatePostDateSet(posts, dateFormatter)
		.reduce(function(collected, fomattedDate){
			return Object.assign({}, collected, {
				// lowercase to match month directory page.url
				[fomattedDate.toLowerCase()]: getPostsByDate(posts, fomattedDate, dateFormatter)
			})
		},{});
}

exports.contentByMonth = collection => {
  return contentByDateString(
  	collection.getFilteredByTag('post'),
  	makeDateFormatter("/YYYY/MMM/")
  );
}

exports.contentByYear = collection => {
  return contentByDateString(
  	collection.getFilteredByTag('post'),
  	makeDateFormatter("/YYYY/")
  );
}

eleventy.js:

const monthsCollection = require("./utils/collections/contentByDate").contentByMonth;
const yearsCollection = require("./utils/collections/contentByDate").contentByYear;

module.exports = function(eleventyConfig) {
	eleventyConfig.addCollection("contentByMonth", monthsCollection);
	eleventyConfig.addCollection("contentByYear", yearsCollection);
	…

snippet of njk layout:

{% for post in collections.contentByMonth[page.url] %}
	<article>
		<h2><a href="{{post.url}}">{{ post.data.title }}</a></h2>
		<p>{{ post.data.description }}</p>
		<p><a href="{{post.url}}">Read {{ post.data.title }} in full</a></p>
	</article>
{% endfor %}

I’ve got lots of work to do before I’ll mark this off as a real world solution – but I consider my question answered, thanks all.

[gerwitz]

Thanks for sharing all of this, it helped me find a way to seriously increase performance. I had initially created a template that filtered for content, but pre-processing into a collection-of-collections is much more efficient.

[nhoizey]

@ollicle hi, I'm looking for the same feature, and looking at your layout code (thanks for sharing!), it seems you don't use pagination.

How do you generate pages for each year and month?

[UPDATE] I've managed to do it, with a template for months and another for years!

[edwardhorsford]

@nhoizey same - I have a template for each and a collection for each.