Pug templates should be cached

[mtsknn]

Is your feature request related to a problem? Please describe.

Build times of Eleventy sites using Pug templates can quickly get out of hand because:

1. Compiling Pug templates is relatively slow.
2. Pug layouts are compiled multiple times per build.

Describe the solution you'd like

Cache Pug templates to speed up builds. Two example implementations are presented below.

Other template types might benefit from caching as well (I haven't tested), so I'm not sure if a Pug-specific caching solution is ideal.

Pug has a cache option (see Pug's API reference), but it "only applies to render functions." Eleventy uses the compile function, so the cache option is not an option (pun intended).

Describe alternatives you've considered

Moving away from Pug templates. 😄 I'm going to do it soon anyway (Preact ftw 💪) so implementing Pug template caching is not critical for me, but I think Pug (or at least Pug layouts) is a poor choice for Eleventy projects until caching is implemented.

Additional context

Here's how I reduced build times from ~6 seconds to ~2.5 seconds and subsequent builds from ~4.5 seconds to less than a second.

Build times from my Eleventy site (+ content files from a separate, private repo):

$ NODE_ENV=production npx @11ty/eleventy --quiet --watch
Copied 24 files / Wrote 98 files in 6.08 seconds (62.0ms each, v0.12.1)
Watching…
File changed: content\index.md
Copied 24 files / Wrote 98 files in 4.58 seconds (46.7ms each, v0.12.1)
Watching…

Let's see where some of the time is spent by adding console loggings to node_modules/@11ty/eleventy/src/Engines/Pug.js:

 async compile(str, inputPath) {
   let options = this.getPugOptions();
   if (!inputPath || inputPath === "pug" || inputPath === "md") {
     // do nothing
   } else {
     options.filename = inputPath;
   }

-  return this.pugLib.compile(str, options);
+  console.time(`Compiling ${inputPath}`);
+  const result = this.pugLib.compile(str, options);
+  console.timeEnd(`Compiling ${inputPath}`);
+  return result;
 }

$ NODE_ENV=production npx @11ty/eleventy --quiet
Compiling ./layouts/feeds.pug: 138.862ms
Compiling ./layouts/home.pug: 90.025ms
Compiling ./layouts/_layout.pug: 70.217ms
Compiling ./layouts/sitemap.pug: 1.618ms
Compiling ./layouts/blog.pug: 60.811ms
Compiling ./layouts/blog-tag.pug: 50.042ms
Compiling ./layouts/weekly-log.pug: 56.489ms
Compiling ./layouts/blog-post.pug: 58.545ms
Compiling ./layouts/blog-post.pug: 46.379ms
Compiling ./layouts/blog-post.pug: 41.69ms
Compiling ./layouts/blog-post.pug: 49.629ms
Compiling ./layouts/blog-post.pug: 47.151ms
Compiling ./layouts/blog-post.pug: 42.664ms
Compiling ./layouts/blog-post.pug: 41.404ms
Compiling ./layouts/blog-post.pug: 44.92ms
Compiling ./layouts/blog-post.pug: 35.886ms
Compiling ./layouts/blog-post.pug: 44.233ms
Compiling ./layouts/blog-post.pug: 41.294ms
Compiling ./layouts/blog-post.pug: 35.953ms
Compiling ./layouts/blog-post.pug: 40.19ms
Compiling ./layouts/blog-post.pug: 38.175ms
Compiling ./layouts/blog-post.pug: 35.886ms
Compiling ./layouts/blog-post.pug: 35.373ms
Compiling ./layouts/blog-post.pug: 37.86ms
Compiling ./layouts/blog-post.pug: 37.293ms
Compiling ./layouts/blog-post.pug: 32.97ms
Compiling ./layouts/blog-post.pug: 35.178ms
Compiling ./layouts/blog-post.pug: 35.551ms
Compiling ./layouts/blog-post.pug: 36.691ms
Compiling ./layouts/blog-post.pug: 34.45ms
Compiling ./layouts/blog-post.pug: 33.714ms
Compiling ./layouts/blog-post.pug: 34.614ms
Compiling ./layouts/blog-post.pug: 35.878ms
Compiling ./layouts/blog-post.pug: 34.796ms
Compiling ./layouts/blog-post.pug: 36.148ms
Compiling ./layouts/blog-post.pug: 35.129ms
Compiling ./layouts/blog-post.pug: 33.364ms
Compiling ./layouts/blog-post.pug: 34.846ms
Compiling ./layouts/blog-post.pug: 31.681ms
Compiling ./layouts/blog-post.pug: 33.75ms
Compiling ./layouts/blog-post.pug: 32.745ms
Compiling ./layouts/blog-post.pug: 34.428ms
Compiling ./layouts/blog-post.pug: 32.531ms
Compiling ./layouts/blog-post.pug: 33.69ms
Compiling ./layouts/blog-post.pug: 35.748ms
Compiling ./layouts/blog-post.pug: 32.748ms
Compiling ./layouts/blog-post.pug: 35.833ms
Compiling ./layouts/blog-post.pug: 31.495ms
Compiling ./layouts/blog-post.pug: 32.383ms
Compiling ./layouts/blog-post.pug: 33.21ms
Compiling ./layouts/blog-post.pug: 33.113ms
Compiling ./layouts/blog-post.pug: 30.86ms
Compiling ./layouts/blog-post.pug: 32.407ms
Compiling ./layouts/blog-post.pug: 34.384ms
Compiling ./layouts/blog-post.pug: 34.338ms
Compiling ./layouts/blog-post.pug: 31.16ms
Compiling ./layouts/blog-post.pug: 38.132ms
Compiling ./layouts/blog-post.pug: 34.037ms
Compiling ./layouts/blog-post.pug: 32.529ms
Compiling ./layouts/blog-post.pug: 35.182ms
Compiling ./layouts/blog-post.pug: 30.767ms
Compiling ./layouts/blog-post.pug: 31.771ms
Compiling ./layouts/blog-post.pug: 34.428ms
Compiling ./layouts/blog-post.pug: 35.279ms
Compiling ./layouts/blog-post.pug: 32.739ms
Compiling ./layouts/blog-post.pug: 31.34ms
Compiling ./layouts/blog-post.pug: 34.47ms
Compiling ./layouts/blog-post.pug: 31.203ms
Compiling ./layouts/blog-post.pug: 35.819ms
Compiling ./layouts/blog-post.pug: 33.668ms
Compiling ./layouts/blog-post.pug: 34.075ms
Compiling ./layouts/blog-post.pug: 33.338ms
Compiling ./layouts/blog-post.pug: 33.744ms
Compiling ./layouts/blog-post.pug: 32.177ms
Compiling ./layouts/blog-post.pug: 35.484ms
Compiling ./layouts/blog-tags.pug: 35.683ms
Compiling ./layouts/blog-post.pug: 34.884ms
Compiling ./layouts/blog-post.pug: 36.529ms
Compiling ./layouts/blog-post.pug: 33.07ms
Compiling ./layouts/blog-post.pug: 35.898ms
Compiling ./layouts/blog-post.pug: 33.485ms
Compiling ./layouts/blog-post.pug: 35.539ms
Compiling ./layouts/weekly-log-entry.pug: 36.904ms
Compiling ./layouts/weekly-log-entry.pug: 40.974ms
Compiling ./layouts/weekly-log-entry.pug: 32.911ms
Compiling ./layouts/weekly-log-entry.pug: 35.647ms
Compiling ./layouts/weekly-log-entry.pug: 33.011ms
Compiling ./layouts/weekly-log-entry.pug: 40.249ms
Compiling ./layouts/weekly-log-entry.pug: 32.898ms
Compiling ./layouts/weekly-log-entry.pug: 33.9ms
Compiling ./layouts/weekly-log-entry.pug: 31.833ms
Compiling ./layouts/weekly-log-entry.pug: 33.963ms
Compiling ./layouts/weekly-log-entry.pug: 34.613ms
Compiling ./layouts/weekly-log-entry.pug: 33.033ms
Compiling ./layouts/weekly-log-entry.pug: 35.005ms
Compiling ./layouts/weekly-log-entry.pug: 34.522ms
Compiling ./layouts/weekly-log-entry.pug: 34.465ms
Compiling ./layouts/weekly-log-entry.pug: 33.936ms
Compiling ./layouts/weekly-log-entry.pug: 33.482ms
Compiling ./layouts/weekly-log-entry.pug: 31.985ms
Compiling ./layouts/weekly-log-entry.pug: 36.188ms
Compiling ./layouts/weekly-log-entry.pug: 36.18ms
Compiling ./layouts/weekly-log-entry.pug: 32.65ms
Compiling ./layouts/weekly-log-entry.pug: 38.105ms
Compiling ./layouts/weekly-log-entry.pug: 33.64ms
Compiling ./layouts/weekly-log-entry.pug: 34.06ms
Compiling ./layouts/feed-rss.pug: 5.272ms
Compiling ./layouts/weekly-log-entry.pug: 37.964ms
Compiling ./layouts/weekly-log-entry.pug: 34.161ms
Compiling ./layouts/weekly-log-entry.pug: 35.379ms
Compiling ./layouts/feed-rss.pug: 4.525ms
Compiling ./layouts/feed-rss.pug: 3.575ms
Copied 24 files / Wrote 98 files in 5.85 seconds (59.7ms each, v0.12.1)

What's happening?

• Compiling simple Pug templates easily takes dozens of milliseconds.
• Pug layouts are compiled multiple times!

Let's add simple caching to Pug.js:

• Add this.cache = {}; to the constructor.

•  async compile(str, inputPath) {
  +  if (inputPath && this.cache[inputPath]) {
  +    return this.cache[inputPath]
  +  }
     
     let options = this.getPugOptions();
     if (!inputPath || inputPath === "pug" || inputPath === "md") {
       // do nothing
     } else {
       options.filename = inputPath;
     }
  
     console.time(`Compiling ${inputPath}`);
     const result = this.pugLib.compile(str, options);
     console.timeEnd(`Compiling ${inputPath}`);
  +  this.cache[inputPath] = result
     return result;
   }

Result:

$ NODE_ENV=production npx @11ty/eleventy --quiet --watch
Compiling ./layouts/_layout.pug: 139.765ms
Compiling ./layouts/feeds.pug: 80.41ms
Compiling ./layouts/weekly-log.pug: 74.165ms
Compiling ./layouts/blog.pug: 61.092ms
Compiling ./layouts/sitemap.pug: 1.808ms
Compiling ./layouts/home.pug: 50.343ms
Compiling ./layouts/blog-tag.pug: 44.649ms
Compiling ./layouts/blog-post.pug: 49.737ms
Compiling ./layouts/blog-tags.pug: 43.039ms
Compiling ./layouts/weekly-log-entry.pug: 43.78ms
Compiling ./layouts/feed-rss.pug: 8.713ms
Copied 24 files / Wrote 98 files in 2.30 seconds (23.5ms each, v0.12.1)
Watching…
File changed: content\index.md
Compiling ./layouts/blog.pug: 63.541ms
Compiling ./layouts/_layout.pug: 44.574ms
Compiling ./layouts/feeds.pug: 42.633ms
Compiling ./layouts/sitemap.pug: 1.168ms
Compiling ./layouts/weekly-log.pug: 47.233ms
Compiling ./layouts/home.pug: 44.062ms
Compiling ./layouts/blog-tag.pug: 40.611ms
Compiling ./layouts/blog-post.pug: 41.386ms
Compiling ./layouts/blog-tags.pug: 44.304ms
Compiling ./layouts/weekly-log-entry.pug: 38.597ms
Compiling ./layouts/feed-rss.pug: 5.207ms
Copied 24 files / Wrote 98 files in 1.52 seconds (15.5ms each, v0.12.1)
Watching…

Now Pug templates are compiled only once per build, making builds much faster.

We can make builds even faster by implementing a cache that persists between builds:

• Add const cache = {}; to Pug.js, before the class declaration.

•  async compile(str, inputPath) {
  -  if (inputPath && this.cache[inputPath]) {
  -    return this.cache[inputPath]
  -  }
  +  if (inputPath && cache[inputPath] && cache[inputPath].str === str) {
  +    return cache[inputPath].result;
  +  }
     
     let options = this.getPugOptions();
     if (!inputPath || inputPath === "pug" || inputPath === "md") {
       // do nothing
     } else {
       options.filename = inputPath;
     }
  
     console.time(`Compiling ${inputPath}`);
     const result = this.pugLib.compile(str, options);
     console.timeEnd(`Compiling ${inputPath}`);
  -  this.cache[inputPath] = result
  +  cache[inputPath] = { str, result };
     return result;
   }

Result:

$ NODE_ENV=production npx @11ty/eleventy --quiet --watch
Compiling ./layouts/_layout.pug: 140.383ms
Compiling ./layouts/blog.pug: 99.065ms
Compiling ./layouts/feeds.pug: 59.639ms
Compiling ./layouts/home.pug: 58.495ms
Compiling ./layouts/weekly-log.pug: 57.211ms
Compiling ./layouts/sitemap.pug: 1.573ms
Compiling ./layouts/blog-tag.pug: 56.662ms
Compiling ./layouts/blog-post.pug: 55.007ms
Compiling ./layouts/blog-tags.pug: 51.63ms
Compiling ./layouts/weekly-log-entry.pug: 49.124ms
Compiling ./layouts/feed-rss.pug: 9.07ms
Copied 24 files / Wrote 98 files in 2.53 seconds (25.8ms each, v0.12.1)
Watching…
File changed: content\index.md
Copied 24 files / Wrote 98 files in 1.07 seconds (10.9ms each, v0.12.1)
Watching…
File changed: layouts\blog-post.pug
Compiling ./layouts/blog-post.pug: 56.5ms
Copied 24 files / Wrote 98 files in 0.88 seconds (9.0ms each, v0.12.1)
Watching…
File changed: content\index.md
Copied 24 files / Wrote 98 files in 0.94 seconds (9.6ms each, v0.12.1)
Watching…

Now Pug templates are cached between builds, making subsequent builds take less than a second. Nice. 😎

Not sure how robust my implementation is, but it's clear that caching would be extremely nice.

Didn't find similar issues. 11ty/eleventy#108 and 11ty/eleventy#984 might be somewhat related, but they are still different.

[stephenhutchings]

One of the issues with the suggested change above is that the caching is based on the inputPath, not the str. Any changes to a file will use the previously cached version. Using the template string as the key should get the desired results.

It's actually possible to replicate this change in your 11ty project configuration by overriding the Pug library with a wrapper to cache the templates.

// .eleventy.js

const pug = require("pug")
const pugCache = {}

module.exports = function (config) {
  config.setLibrary("pug", {
    compile: (str, options) => {
      if (pugCache[str]) return pugCache[str]

      pugCache[str] = pug.compile(str, options)

      return pugCache[str]
    },
  })
}

[stephenhutchings]

It's also worth noting that this doesn't handle dependencies in any way. Any changes to files that are included in your templates will be ignored on subsequent builds.

[j-f1]

Caching wouldn’t be an issue if the cache was cleared at the end of each build though, right?

[stephenhutchings]

Caching wouldn’t be an issue if the cache was cleared at the end of each build though, right?

No it wouldn't be an issue. Dependencies will only cause a problem if you persist the cache, which would happen in the code snippet I've added above.

Is there hook in 11ty to run a function before or after a build runs, so the cache could be reset?

[stephenhutchings]

Actually looks like 11ty's events is what I'm looking for.

[stephenhutchings]

Updated snippet that clears the cache after a build has run.

// .eleventy.js

const pug = require("pug")

module.exports = function (config) {
  let pugCache = {}
  
  // Reset the cache  
  config.on("eleventy.after", () => {
    pugCache = {}
  });

  config.setLibrary("pug", {
    compile: (str, options) => {
      if (pugCache[str]) return pugCache[str]

      pugCache[str] = pug.compile(str, options)

      return pugCache[str]
    },
  })
}

Still, it would be much nicer if this was handled by the built-in Pug compiler, and handled caching across builds while managing dependencies.

[Zearin]

(Psst! This Issue should be tagged with template-language:pug)