Reduce load of preview fetching on third-party servers

Steps to reproduce the problem

Posting an image on Mastodon causes hundreds or thousands of federated servers to fetch the preview details all at once, causing a high load for the target server and effectively creating the by now well-known "Mastodon DDoS" effect. For static websites this may not be much of an issue but it routinely brings down larger pages or those which have dynamic server-rendered content.

Expected behaviour

No DDoS

Actual behaviour

DDoS

Detailed description

The discussion at #4486 has been going on for six years and is quite a mess. I'm opening this ticket to start fresh and collect the information scattered throughout the thread into one place so that the discussion doesn't have to keep going in circles.

Rationale

It is the responsibility of software like Mastodon to be a good neighbor on the internet. DDoSing others is not being a good neighbor! It's important to figure out how to prevent this issue from occurring.

IMPORTANT: Let's make this discussion better than the last one. For Mastodon devs: don't blame the victims. It's not the website's fault that Mastodon is DDoSing them. For server operators: Mastodon is a volunteer-run free software project. Be mindful of that.

Present mitigations

Currently a random jitter is added of between 0 and 60 seconds after a federated server is made aware of a post which includes a link, before it fetches the preview details. This does not seem to be sufficient to prevent the DDoS effect from occurring.

Suggested mitigations

The discussion has focused on two main suggestions for a fix.

Federating previews

The original poster's instance can fetch the preview details and attach them to the message, federating the preview details without requiring other servers to fetch it themselves. Criticism of this solution is mainly focused around the fact that Mastodon is a zero-trust environment, so instance A cannot trust instance B's word that a preview accurately represents the URL.

Because the original post is always fetched when a post is federated, the trust space can be reduced to the origin server alone; intermediates need not be trusted. Opinions on the depth of this problem have ranged from "it's no different from posting an image" to "we absolutely cannot trust anyone ever for any reason".

Answers proposed to the objections of trust have included random sampling wherein the preview is fetched 1 in N times, and if it's found to be inconsistent with the federated preview, some action can be taken, such as setting a flag on that instance which causes future (and past?) previews to be fetched unconditionally from that server, automatically making a report to the instance admins of suspected foul play, or federating the flag so that other servers can force a sample from that instance when foul play is suggested.

Any of these changes would likely involve a slow roll-out across the fediverse. Sometimes link previews might not work for older clients as a consequence. My take: I believe this is quite acceptable, it's a small price to pay for correcting this behavior. User experience does not outweigh "don't DDoS people". Furthermore, a slow roll-out will naturally imply that the problem does not get fixed overnight, but rather that the behavior corrects gradually over time as the fix is rolled out across the fediverse -- not an issue imo.

Reducing load

No federated previews, but instances don't immediately fetch the preview. Most to least effective mitigations along this line of thought:

Add "show preview" button or so to the UI (or otherwise detect when the user is "interacting" with a post) and fetch the preview only then. Note that the preview only needs to be fetched once, if one user on an instance interacts with a post then the preview becomes available for all users without further interaction.
Fetch robots.txt (and cache it) for that remote URL. Link previews should respect robots.txt #21738
Lazily fetch the preview only when it's shown in the UI. For smaller instances, this would reduce the load if no one is actively using the server (e.g. idling Mastodon in an unfocused tab). It would also likely reduce the need to fetch previews for posts that appear on the federated timeline.
Policy-based approach, e.g. if a post comes in because a user follows the poster, fetch the preview; otherwise don't.
Increase the random jitter for fetching previews from 60 seconds to some higher number.

Some combination of these mitigations is also possible, for instance the jitter could be increased to five minutes, but done immediately if the post shows up in the UI.

Specifications

n/a, all versions affected

For my part, I think the mitigation which best strikes a balance of non-controversial, effective, and easy to implement, is robots.txt. It allows server operators to have fine-grained control so that they can indicate which routes are expensive to render. For my servers, I have already had expensive routes excluded in robots.txt for a long time to keep crawlers away; this has been a well-known mitigation for automated requests to expensive routes for some time now.

I also think that someone within Mastodon should be willing to take responsibility for implementing these mitigations. I am generally sympathetic to the incentives of FOSS projects and the nature of the volunteer labor pool, but in the case where software is causing disruptions to other services it's important for the maintainers of that software to accept responsibility for it -- expecting the victims of a DDoS to put up the labor to mitigate the DDoS is just rubbing salt into the wound. If no one is available to do the work, fetching previews should simply be disabled entirely until such a time as a mitigation can be implemented.

As a random passerby I can say, that a server, the author of a message belongs to, should be responsible to retrieve this information (subject to a timeout) and only then propagate the message along with this data. Other servers should not retrieve the data themselves.

You have to trust it, sure, buy you already have to trust it's not altering messages… And you might as well block bad actors on your server when that happens.

I wrote a document on the topic a few months ago, which I think is quite extensive on the subject: https://gist.github.com/renchap/3ae0df45b7b4534f98a8055d91d52186

I tried to highlight the challenges related to this, as well as the possible implementations and their drawbacks.

random sampling wherein the preview is fetched 1 in N times, and if it's found to be inconsistent with the federated preview, some action can be taken

Perhaps a small point, but what happens if the original web page linked to is updated in the meantime so that the preview changes (as sometimes happens with news articles for example)?

@renchap

Another path that could be taken here is to have a link preview service, which can cache the preview. That'd mean rather than every mastodon server going to the origin server to prepare a preview, the mastodon servers delegate that to a trusted preview service.

E.g., a lot of large companies use a service like https://embed.ly for generating previews. Perhaps there's room for a setup based on low-cost infrastructure (e.g., Cloudflare workers or similar on-demand compute environments) that could be deployed?

That way a server admin could choose to trust one of the preview services, instead of having their instance do the fetching (of course, you could always opt out, with the caveat that your service would amplify traffic potentially).

Logic here is that there'd be less preview services deployed than mastodon servers, and that'd prevent the thundering herd. Trust would be between mastodon server admins and the preview services, a preview service that doesn't provide accurate previews would quickly find their reputation trashed & instances would switch to a different provider.

Edit: this is Solution 5 in @renchap's gist

A few more thoughts:

the trust issue is particularly important because currently, all posts sharing the same link will reuse the same preview, and that preview is what's used in the “News” tab of the “Explore” section (trending links), so poisoning can have a pretty wide effect
not every link is worth a preview, so even without trust, I think it would be an improvement for the author to signal whether a preview card is expected or not, so that if the author decided not to include a preview, or the preview could not be fetched, other servers would not attempt fetching it

To lessen the impact of an untrustworthy server, something that might make sense is pre-fetch the link while the author is writing their posts, offer them a preview, and have them select it. Then, somehow serialize that preview when federating it. On the receiving end, if you receive a post with a preview:

if the link is already known and its preview trusted, discard the attached preview and use the already-stored one
if the link is not known, store the attached preview as non-trusted
if the link is already known but its preview is not trusted, compare it with the attached preview, if it's the same, re-use it, if it's different, schedule the preview to be fetched

Whenever a local user authors a post with a link to a known but non-trusted preview, fetch the link yourself to get a trusted preview. Whenever a non-trusted preview is eligible for trending, fetch the link yourself and update the trusted preview.

Compared to the current state, this would require (possibly breaking—in that clients not implementing this change may not get previews to generate) changes in the REST API to chose which link to preview (if any at all) and S2S protocol changes to communicate that information. It would introduce an attack vector in which a remote user could generate a misleading preview for their own posts, but without introducing a cache poisoning issue. An attacker could still purposefully send two conflicting previews to a server to cause it to fetch the link, but it should greatly reduce the impact of normal usage.

[tantek] added "Start a page on [[thundering herd]] with scenarios an…

@renchap

One option for @renchap's "Solution 6: Design and implement a protocol for websites to provide a signed preview" would be for Open Graph to define an "integrity" structured property on image/video/audio properties to allow for zero-trust redistribution of bandwidth-heavy media resources (similar to what's used used for Subresource Integrity). e.g.:

<meta property="og:image" content="https://ia.media-imdb.com/images/rock.jpg" />
<meta property="og:image:integrity" content="sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC" />

Unfortunately, the Facebook & Google groups linked for discussion from the official Open Graph page seem to be derelict and defunct respectively, so I'm not sure of the best way to get this proposed and adopted.

Edit: Another option would be to use the HTTP Digest header for this purpose.

I think it would be an improvement for the author to signal whether a preview card is expected or not, so that if the author decided not to include a preview, or the preview could not be fetched, other servers would not attempt fetching it

+1. Also, having this information and federating it would improve compatibility with federated news aggregator platforms like Kbin/Lemmy, which don't have a reliable way of extracting the "best link" from an incoming Mastodon post (LemmyNet/lemmy#2889 (comment)). If this information was added to a status as an explicitly attached Link, e.g.:

{
...
	"attachment": [{
		"href": "https://example.com",
		"type": "Link"
	}],
...
}

...this would be helpful in determining both whether a link preview should be generated or not, and which link the author intended to have shown as a preview (in the case of multiple links in a post).

(Presumably the preview property could also be added to an attached link for federating this other card preview information?)

@ryanfb

@ryanfb FWIW, in theory Open Graph uses a subset of RDFa, though which subset exactly was never clear to me (full RDFa is a bear to implement, duplicating body content in meta tags seems silly when RDFa was specifically made to reuse it, what real-world parsers support isn’t written down anywhere that I can see). And in theory, it’s an open system—as long as you use valid URIs you control and not ones under og: (aka https://ogp.me/ns#), nothing stops you from defining whatever relations you want, including “has a preview image with SRI hash of”, or perhaps you could get one registered with Schema.org or similar.

Whether there’s a good place to talk to people interested in such things, I don’t know—Facebook’s relationship with the wider Web has gotten a lot less cuddly since 2010. Perhaps those who once maintained the spec repo could be contacted?

Some questions for the group:

Do all post authors want all their posts and all links in their posts to be displayed with link previews?
Do post authors expect link previews to be visible to them when they author a post?
If link previews are visible during authoring, do posters expect the link preview seen by readers of that post to be unchanged from what the author saw during posting?
If link previews shown to readers are different than what the poster saw during authoring (either poster did not see a link preview or poster saw different preview than reader), does the poster expect this difference to be obvious to the reader of the post?
Do posters want servers to have the ability to add/modify content displayed with their post without drawing a clear distinction between authored content and generated content?
If a reader receives posts where the author hasn't included a link preview or has requested that link previews not be shown, should readers have the ability to override this and generate link previews automatically in bulk?
If a poster, a reader, or a server admin believes that link previews are wasteful (of energy) or misleading (misrepresenting authoring intent or knowledge) or essential (for accessibility/understanding where a link goes) or important (for aesthetics or engagement), what controls should be in place for these competing views to be negotiated?

There are many more questions that could be asked, particularly from the perspectives of readers and admins, but I thought I'd start with these.

The point is that the discussions so far seem a little bit focused on the current implementation which seems based on an understanding of link previews as an automatic aid to readers whereas I believe link previews should primarily be driven by posters during authoring since this is the path to full trust in the content.

As a post creator, I want to see link previews, and I don't want them to change after I have posted: they should be authored content.

As other people on this thread have indicated, some of the answers to the technical questions are clearer when this is the starting point of the feature.

Yet another site – this time a major bookmarking service – is considering blocking Mastodon because of the DDoS effect: https://twitter.com/pinboard/status/1698775481292832978

Please fix this.

Some questions for the group:

1. Do all post authors want all their posts and all links in their posts to be displayed with link previews?

Moot. It should be up to the preference of the person viewing the post, and the client they use. Also, can you stop a client creating a link preview if it wants to?

2. Do post authors expect link previews to be visible to them when they author a post?

Expect? No, it depends on if the client I'm using has the capability. It's certainly a nice to have, though.

3. If link previews are visible during authoring, do posters expect the link preview seen by readers of that post to be unchanged from what the author saw during posting?

Yes.

4. If link previews shown to readers are different than what the poster saw during authoring (either poster did not see a link preview or poster saw different preview than reader), does the poster expect this difference to be obvious to the reader of the post?

Moot, because of (3).

Do posters want servers to have the ability to add/modify content displayed with their post without drawing a clear distinction between authored content and generated content?

If by non-authored content you mean previews, then I think it's clear that link previews aren't part of the authored content. Right? Link previews are a thing across the web, they display what's on the linked page. They're not part of the post.

If a reader receives posts where the author hasn't included a link preview or has requested that link previews not be shown, should readers have the ability to override this and generate link previews automatically in bulk?

See (1).

I run a small server and use relays to fill the federated timeline. Most of those posts are probably never displayed, so only fetching previews for posts for the other timelines (home, trending, lists, local) and replies to those posts would greatly reduce the load caused by servers like mine. For the federated timeline, fetch previews for the last 200 messages or so when a user requests its display.

I'd go farther than that. Don't fetch a preview unless a post in the federated timeline that doesn't meet your criteria has been opened. I would also add fetching of preview cards for saved hashtags, although, that should be included in the home timeline fetch already by default.

I'm fetching so many posts a day and will never see most of that on my self-hosted instance unless I do a search. I have come to rely on my instance being my first stop for searching, even over Google. Can't wait for the ability for query string searches so I can set it up in my browser as a default search provider. See Help Wanted: Use /search for search

After being pushed back version after version, this was quietly removed from the roadmap last week.

This is very disappointing, to say the least. The thundering herd behaviour is obviously wrong.

@dpk

@dpk the event here is a bit misleading, all milestones were removed: https://github.com/mastodon/mastodon/milestones?state=open

I removed the milestones are they were not indicative of our real roadmap.

We did more research on this topic in the last few weeks, and, if possible with the time we have, will try to move forward on this topic for 4.4.

While waiting for Proper Solution(tm) it looks to me as if there's a way for us Fediverse instance owners to voluntarily lessen the problem. The issue here doesn't seem to be with the top #10 or top #50 instances, where most users reside, but the fact that there are so many smaller instances - even one person ones (I run a family instance myself for example).

We, the "small instance owners" could perhaps implement mitigation purely on our side right now?

If so, a helpful guide that we can spread throughout #MastoAdmin et al would be a great start. From this thread it looks like the 0-60 random timeout is a place to look into, but I'm sure there are more ways. I already change the post length limit in my docker build and adding more things to patch there seems easy.

This is on our roadmap for the next version FYI, expect some news soon.

So?

It would help to fetch brotli (pre)compressed HTML pages (Accept-Encoding: gzip, br) and alternate image media (eg WEBP, Accept: image/webp, */*) where available, to reduce bandwidth.

When I recognise a stampede request incoming by the lack of Referer I return a lower-fi JPEG or PNG, but I also have the option to share a much smaller lower-fi WEBP at little effort if the second of these (Accept) is dealt with.

Yes, reducing the overall number and density of requests would be the main win, but making each request less expensive would also help and is orthogonal.

To lessen the impact of an untrustworthy server, something that might make sense is pre-fetch the link while the author is writing their posts, offer them a preview, and have them select it. Then, somehow serialize that preview when federating it. On the receiving end, if you receive a post with a preview:
* if the link is already known and its preview trusted, discard the attached preview and use the already-stored one

If I as an author select/approve a preview, this makes it part of my message and I would strongly object to it being replaced by something I have no control over.
My whole post may be based on the fact that a newspaper article, maybe even its headline has been changed, and replacing the preview I see and refer to with an older different version may render my post inconsistent or even nonsensical.

#23662 (comment)
Shared preview cache servers could be a good solution. we can reduce thousands of requests to hundreds if we share the load. I could host a preview cache server, and preview would be generated after first request. Then every next request sees the cached version.

Preview can be discarded after DNS TTL. We now have trusted, reasonably regularly updating, preview caches. Localization is not an issue either if we use a local server.

@renchap

@renchap any update on this?

Thank you very much

Coming in with my two cents,

Link previews can also be harmful to server owners. I have been temporarily blocked from quad9 due to the sheer number of DNS requests for generating previews.
I also agree with gunchleoc that fetching so many previews when I (on my single-user instance) will inevitably only see a small fraction of the posts is a waste.

I vote for having a way to completely disable previews for a server. Although a per-toot show preview could also be nice.

I don't know the inner workings of mastodon (😅)... but can the posting account's host server fetch the page and attach an image of it to the post when the post is distributed to other instances? IIUC distributing image content in mastodon doesn't cause a DDoS on 3rd party servers (see this), so maybe this is a simpler solution to the problem?

@craftyguy

@craftyguy This was already addressed in the third comment of this thread

Navigation Menu

Repository navigation

Reduce load of preview fetching on third-party servers #23662

Steps to reproduce the problem

Expected behaviour

Actual behaviour

Detailed description

Rationale

Present mitigations

Suggested mitigations

Specifications

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Reduce load of preview fetching on third-party servers #23662

Description

Steps to reproduce the problem

Expected behaviour

Actual behaviour

Detailed description

Rationale

Present mitigations

Suggested mitigations

Specifications

Activity

ddevault commented on Feb 17, 2023

flashymittens commented on Feb 17, 2023

renchap commented on Feb 17, 2023

jimrhiz commented on Feb 18, 2023

ThisIsMissEm commented on Feb 19, 2023

ClearlyClaire commented on Feb 20, 2023

ryanfb commented on Jun 1, 2023

jonaharagon commented on Jun 2, 2023

alexshpilkin commented on Jun 2, 2023

callionica commented on Aug 25, 2023

dpk commented on Sep 5, 2023

brendanjones commented on Oct 4, 2023

11 remaining items

p37307 commented on May 6, 2024

dpk commented on Aug 26, 2024

ThisIsMissEm commented on Aug 26, 2024

renchap commented on Aug 27, 2024

troed commented on Oct 30, 2024

ddevault commented on Jan 20, 2025

DamonHD commented on Feb 9, 2025

ke352802081770314 commented on Mar 28, 2025

murtezayesil commented on Jun 23, 2025

quicoto commented on Aug 15, 2025

Jacoblightning commented on Jul 17, 2026

craftyguy commented on Sep 2, 2026

progval commented on Sep 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions