Persist in-memory cache

[kamituel]

In-memory cache is empty whenever Eleventy build starts from scratch. It's not an issue for sites with small number of images. But for sites with large number of large images, compressing them takes quite a lot of time.

In my case (https://kamituel.pl) I've over 300 high resolution source images that I'd ideally serve in 2-3 formats (avif, jpg, maybe webp) in multiple sizes.

I propose a new config option that would, if enabled, persist the in-memory cache to disk and reuse it the next time the build starts.

If maintainers consider this idea to be a good candidate to exist in eleventy-img itself, I'd be happy to formulate this proposal more concretely and later help / submit a PR.

[dimitrinicolas]

For thoses wanting a quick solution for caching optimized images whenever build starts from scratch, here is a solution.

Warning, it checks for already existing file outputs by looking if their output path file is existing, it may or may not be ideal depending on your filenameFormat configuration.

const fs = require('fs');
const Image = require('@11ty/eleventy-img');
const path = require('path');
const shortHash = require('shorthash2');

/** ... */

const options = {
  /** ... */
  filenameFormat: (id, src, width, format) => {
    const extension = path.extname(src);
    const name = path.basename(src, extension);

    const stats = fs.statSync(src);

    const hash = shortHash(`${src}|${stats.size}`);

    return `${name}-${hash}-${width}w.${format}`;
  },
};

const stats = Image.statsSync(src, options);

/** Creating a flat array of all the output paths from the stats object. */
const outputPaths = Object.keys(stats).reduce((acc, key) => {
  return [
    ...acc,
    ...stats[key].map((resource) => {
      return resource.outputPath;
    }),
  ];
}, []);

/** Checking if all output files exists. */
let hasImageBeenOptimized = true;
for (const outputPath of outputPaths) {
  /** Edit the output file path resolving, depending of this file */
  if (!fs.existsSync(path.resolve(__dirname, '..', '..', outputPath))) {
    hasImageBeenOptimized = false;
  }
}

if (!hasImageBeenOptimized) {
  Image(src, options);
}

/** Do whatever you want with the `stats` object */
const imageAttributes = {
  // [...]
};

const html = Image.generateHTML(stats, imageAttributes, {
  whitespaceMode: 'inline',
});

If you are deploying on Netlify you can cache your eleventy-img output directory using netlify-plugin-cache.

npm install netlify-plugin-cache

In your netlify.toml:

[[plugins]]
  package = "netlify-plugin-cache"
    [plugins.inputs]
      paths = ["<YOUR OPTIMIZED FILES OUTPUT DIR PATH>"]

[marcamos]

I would love for something like this to exist in the official plugin. I thought I was doing something wrong when I saw that builds took three minutes (converting 22 images into three formats and ten sizes each [though, maybe that's overkill?]).

[snibbo71]

Just adding my tuppence here, that I'd like this too as I'm using the Image.generateHTML code to pull images remotely. The caching side of it works, but 11ty still chucks those cached images through the resizing process which annoys my laptop and takes a lot longer than is ideal :)

I'm really not clear on how to get Dimitri's work-around to work with the generateHTML method - if indeed I can

[dimitrinicolas]

@snibbo71 My piece of code does not take into account whether the resource is remote or local. I use it with the html generation function. After my shortcode, you can generate your html code with the stats object returned by Image.statsSync , e.g.:

const imageAttributes = {
  // [...]
};

const html = Image.generateHTML(stats, imageAttributes, {
  whitespaceMode: 'inline',
});

(I've just added it to my previous comment, to be sure that every future reader will understand)

Both Image.statsSync and Image.generateHTML are not generating resized/optimized images output files. The Image.statsSync is just returning the output file paths that the async Image.stats would "normally" export.

Then in my shortcode, we check if those output files are existing (that's why we rely on a hashed src+file size string in output file names), if not, we call the Image.stats async function, but we do not have to wait for it to resolves to generate the html. The whole html code generation is async.

[snibbo71]

Heya, thanks for the quick reply. I must be doing something entirely wrong because nothing is cached in my setup - even local stuff. Remote images just blow your code up unfortunately :) (Actually, it's the fs.statSync and Image.statSync that dies with ENOENT - which isn't entirely surprising)

I shall keep playing. I have too many images to deal with to have to redo them every time I change a template :) If I come up with anything I'll post back

[snibbo71]

Having dug into this, FileSizeCache isn't persistent, which explains why it doesn't work across restarts.

At the moment I can't dig much further to see whether there's any reason not to make that another AssetCache subclass. There may be good reasons not to persist this information. And that's largely down to the fact that I'm somewhat over my head as this is my first look at anything serious with node.js anyway :)

[dimitrinicolas]

I'm sorry, I haven't been into 11ty image for a moment, and I never used remote images fetching. Good luck in your research! 😄

[zachleat]

Since it came up in the Eleventy Meetup today, here’s a quick config-based solution that won’t build images that haven’t changed (after the initial build):

const eleventyImage = require("@11ty/eleventy-img");

module.exports = function(eleventyConfig) {
  let modifiedFiles = [];

  eleventyConfig.on("beforeWatch", files => {
    modifiedFiles = files || [];
  });

  // This function will likely not match your project and will require you to customize to your build.
  eleventyConfig.addJavaScriptFunction("eleventyImageHtml", async function(src, options, htmlAttributes) {
    let stats;
    if(modifiedFiles.length === 0 || modifiedFiles.includes(src)) {
      // During watch/serve, only generate images that have been modified.
      stats = await eleventyImage(src, options);
    } else {
      // Return the metadata only without generating the Image.
      stats = eleventyImage.statsSync(src, options);
    }

    return eleventyImage.generateHTML(stats, htmlAttributes);
  });
}

Read more about Events: https://www.11ty.dev/docs/events/ This solution will work in Eleventy 0.11.1+

Sample usage in an 11ty.js template:

class Test {
  async render() {
    return this.eleventyImageHtml("./nebula.jpg", {
      widths: [300],
    }, {
      alt: "Nebula"
    });
  }
}

module.exports = Test;

[brycewray]

Thanks very much, @zachleat — but guess I’ll have to wait until it’s incorporated in the regular eleventy-img examples (https://www.11ty.dev/docs/plugins/image/#use-this-in-your-templates) since I can’t seem to translate either of these to shortcode creation as opposed to addJavaScriptFunction. That’s my fault, not yours, of course! Much obliged for your answer today in the Eleventy Meetup as well as these code samples.

[zachleat]

Oh, I’ve made a mistake in my code, sorry about that! I’ll fix it