Async nunjucks filters cause template not to render

[drivet]

Describe the bug
Async nunjucks filters cause template not to render

To Reproduce
Steps to reproduce the behavior:

I have a nunjucks filter that looks like this:

eleventyConfig.addNunjucksAsyncFilter('scrape', async (stuff, cb) => {
const data = await Promise.resolve(stuff);
cb(null, data);
});

And in a nunjucks template I have this:

{{ link | scrape }}

As you can see, the filter isn't doing anything, really - it's just returning the value passed in, but via a Promise, then awating it before calling the callback. And I know that the link variable has a value (though even if it didn't, I would still expect this to work).

The above nunjucks snippet is in a partial template which is included (via {% include %}) in a for loop inside of a larger template. The larger template renders, but the partial does not. I'm not seeing any errors emitted from the build process.

I feel like I may have a deep misunderstanding of how async filters work here. Any help is appreciated.

Expected behavior
Given that the async filter isn't doing anything special (right now at least), I'm expecting this to render without a hitch. I'm expecting to see my partial template render in the for loop.

Environment:

• Linux Mint 20.2
• Eleventy Version: 0.12.1

[zachleat]

I don't have a ton of time to respond but there is an async friendly for loop in Nunjucks. I don't think the standard for loop is async safe

[pdehaan]

The async filter works for me if I use it directly:

const axios = require("axios");

module.exports = (eleventyConfig) => {
  eleventyConfig.addNunjucksAsyncFilter('scrape', async (uri, cb) => {
    const res = await axios.get(uri);
    cb(null, `${uri} [status=${res.status}/${res.statusText}]`);
  });

  return {
    dir: {
      input: "src",
      output: "www"
    }
  };
};

---
title: 11ty is fun
link: "https://11ty.dev/"
---

result="{{ link | scrape }}"

OUTPUT

result="https://11ty.dev/ [status=200/OK]"

But yeah, things get trickier when using loops and partials.
Check out https://mozilla.github.io/nunjucks/templating.html#asynceach (which is what @zachleat was referring to above)

[pdehaan]

Here's a rough example using {% asyncEach %} ... {% endeach %} w/ an async Nunjucks filter:

---
title: 11ty is fun
links:
- https://11ty.dev/
- https://github.com/11ty/eleventy/
- https://opencollective.com/11ty
- https://11ty.dev/uh-oh-404-city/
---

{% asyncEach link in links %}
result="{{ link | scrape }}"
{% endeach %}

I had to tweak the async filter to handle errors:

  eleventyConfig.addNunjucksAsyncFilter('scrape', async (uri, cb) => {
    try {
      const res = await axios.get(uri);
      cb(null, `${uri} [status=${res.status}/${res.statusText}]`);
    } catch (err) {
      cb(null, `${uri} [error=${err.message}]`);
    }
  });

OUTPUT

result="https://11ty.dev/ [status=200/OK]"
result="https://github.com/11ty/eleventy/ [status=200/OK]"
result="https://opencollective.com/11ty [status=200/OK]"
result="https://11ty.dev/uh-oh-404-city/ [error=Request failed with status code 404]"

The error handling bit is pretty important. I think when I was returning the error directly in the async callback function (cb(err);), it was breaking the build — which you may or may not want. But in this case I just wanted to log an error message, so I passed null in the callback and an error string for the second param.

[pdehaan]

Oh, and another tip is depending on what your async filter is doing, you may want to look into some caching solutions. I quite like https://www.11ty.dev/docs/plugins/cache/.

I did notice that my build times did increase quite a bit using the basic axios fetching above (especially when adding the 404 link). So caching some data locally might speed up your builds if you don't need to constantly recheck the same data over and over on every single build and you're fine with the data being potentially stale for 5-60 minutes, or whatever.

[drivet]

Aha! Yep, the asyncEach did the trick! Thanks muchly.

And yeah, I'm using the caching plugin in conjunction with all this. As you may have guessed from the name of the filter, I'm trying to scrape a bunch of URLs for their metadata in order to display unfurling information in my templates.

[zachleat]

This is an automated message to let you know that a helpful response was posted to your issue and for the health of the repository issue tracker the issue will be closed. This is to help alleviate issues hanging open waiting for a response from the original poster.

If the response works to solve your problem—great! But if you’re still having problems, do not let the issue’s closing deter you if you have additional questions! Post another comment and we will reopen the issue. Thanks!