Mastodon can be used as a DDOS tool #4486
Comments
|
Tested on my account with 1.5k followers (I don't know how many instances). Result : 578 requets in 20 seconds. Examples :
|
|
duplicate of #3518 |
|
I wouldn't call it a duplicate. Related, absolutely. Not duplicate. The issue with #3518 is that a single server is making 3 different requests for the same resource. The issue in this is that hundreds of servers are requesting the same resource, as soon as they get a notification of the toot. |
|
Imagine you got 10 account on 10 instances and you post 10 links on each. |
|
I can totally see this being a problem, but I also don't see a solution to it, because people want previews. An idea could be to randomly delay the fetching of the preview, however, you would get complaints from people "why doesn't the preview show up?" when they immediately check it. Anyone have any other ideas? |
|
Perhaps "trust, but verify." Also, humans are a great source of randomness. Bear with me as my assumptions about how the federation works may not be completely accurate -- the details shouldn't matter. Have the originating server send metadata about the link's preview along with the toot. When the client requests the preview, their own server can then verify the link and update the preview if necessary. This way, there's always something to show up front. Since the Web client doesn't attempt to show the preview until the user clicks on a toot, we have a random wait -- on the order of seconds, minutes, or years -- so we aren't creating a thundering herd based only on the toot hitting the federated servers. Also, if nobody is even going to look at the preview, one never gets loaded, saving a few completely unnecessary requests. The only real concern is a malicious server sending incorrect information in the toot's metadata... That's why I say to verify. (This would also give users the ability to set a preferred preview image, as they're able to do on FB and G+, but that really belongs in a feature request rather than a discussion about a bug report) |
|
I was thinking about the same thing : couldn't the preview be generated when the toot is posted, by the server from where it's posted... then other instances would fetch this preview from the originating instance, as they do for media attachments ? Just a random idea tho. I don't know if it would fit in the mastodon federating scheme. |
|
Calling 400 HTTP requests a "dDoS" seems very exaggerated. |
|
Someone running Wordpress on a stripped down VPS without any CDNs mitigating traffic spikes would be very unhappy to see 400 HTTP requests hit in an instant. Many hosts automatically reboot VPS servers if they exceed their memory allotment, bringing the site down for minutes... and on Mastodon, the first few minutes after a link is posted accounts for 95%+ of the visits that the site is going to get from that toot. Arguably, the people who buy the cheapest VPS are the ones who need that traffic the most; by rebooting their server each time someone links to them on Mastodon, you're not just DOSing them, you're creating a bad user experience, driving away potential customers. A DDOS doesn't have to last for days, harnessing the power of millions of misconfigured routers and IP cameras to be effective... It just has to come from more than one source at once and deny normal users use of the service. |
|
@ghedipunk If I had to run a site on a "stripped down VPS", I would run a static site, for which 400 requests is nothing, even on a Raspberry Pi. But my main point is that generating 400 requests is very simple for any attacker. Mastodon is not really needed. |
|
This is not an issue just about 400 HTTP requests. The main issue here is that as Mastodon gets bigger and more instances are created, the larger and more serious the attacks can become. Another part of this is that while the attacks can be on purpose by a malicious person, they can also be on accident by an innocent person with a lot of followers who simply wants to share a meme they liked a lot. |
|
You're experienced in web hosting, as are the vast majority of people who would participate in this thread here on Github. However, looking at small business sites, it's very clear that we're in the minority. We know about Cloudflare and using nginx as a TLS terminator in front of Varnish-Cache that does intelligent edge-side includes on Apache/PHP generated content... We know what design decisions to make on a completely static HTML only site so that if we need to update the site's layout, we can do it quickly. We know that if you want to sell things online, you have to resize your 48000x28000 pixel product images before displaying them to the users. And, we know that there are users who don't know these things. This isn't about helping us, the technical elites who can run a successful site off of a Raspberry Pi. It's about being good neighbors. |
|
I really don't see where the problem is. The distributed nature of mastodon will spread the requests over a (small for humans but large for the network) interval and when you publish some link on a public media, you expect traffic coming back to the site in question. Otherwise there's little interest on publishing it. |
|
Indeed, when you post a link on Twitter, you get more requests (not from Twitter itself, because of its centralized nature, but because of all the bots that read Twitter and act). |
|
i noticed this recently too. >1k requests in <45s, >25qps. not a disaster, my site handled it fine, but still, noticeable. small thread on it here: https://mastodon.technology/@snarfed/100119606571241751 , cc @ashfurrow @neekz0r.
(qps numbers in the graph are artificially reduced due to a 60s+ aggregation interval.) |
|
Haha.. the joys of having the same handle on multiple platforms. Yeah, to me I think this should be considered a little more serious because theoretically this can be used as an amplification attack. |
|
Here is my capture of this phenomenon if anyone's interested (removed IP addresses). There are a few real users, but mostly it's mastodon servers. Click to expandIt's may not be a huge deal now, but if the network grows and this keeps happening, it might be problematic. Consider when eg gargron boosts a toot - basically the entire fediverse will query that page within seconds to minutes (to make thumbnails nobody will even look at unless they open the toot in the detailed view). |
|
To those suggesting that the sending server provide the link preview:
|
Yeah that's for sure been my concern. We've seen instances running malicious code modify payloads when federating before. |
|
This may have been proposed before and isn't ideal, but how about using
lazy load for the thumbnail, requesting it on demand? I would imagine, for
the majority of statuses, the preview won't ever be used on the servers it
federates to.
…On Tue, Jun 12, 2018, 11:09 PM Ash Furrow ***@***.***> wrote:
Can the sending server be trusted about that?
Yeah that's for sure been my concern. We've seen instances running
malicious code modify payloads when federating before.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#4486 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB8lHppOEMXsxXB4likGHLv_KZNrgE70ks5t8C4MgaJpZM4OqUaH>
.
|
|
Maybe the preview could be delayed until someone select the toot for display? This way, each server will query the URL only when one of it's own users display the toot, human does the randomised delaying. |
|
Can we attach the preview to the toot as we attach custom emoji and media ? This way, bots could even create embed view, or enriched toots. (as embed stuff on Discord) |
|
Has anyone ever tried this out on a large scale? I can't believe that the requests would be enough to cause a DoS. |
|
It's not quite a DoS but it can saturate a small web server that only has a few threads to work with. So it's a problem. @MightyPork's suggestion is probably the most viable one. |
* Spread out crawling randomly to avoid DDoSing the link Fix #4486 * Remove trailing whitespace
Sorry but you absolutely have no clue how mastodon works, abusive posts won't show up on peoples timelines unless they follow the person, follow someone who boosted it or if the post mentions you, meaning their spread is quite limited and will be fully contained if said instance gets blocked (or the user is banned) If we allow neighbouring instances to poison the thumbnail, abuse can spread so much faster and easier. Without users being able to block it, as the link itself is likely just posted from an account they trust and want to follow.
On a decentralised system like mastodon you need to use a "trust no-one" approach, the only two places you trust are:
Allow anyone else to run along this chain and we will see abuse. You simply can't expect people to be nice and fair, I wish the world was like that but it isn't. Leave a hole for trolls to wreck havoc and they will. Your ideal vision of this is absolutely detached from reality and shows me a lack of understanding |
Nothing you've said runs contrary to my understanding of Mastodon.
Have the original poster cryptographically sign the preview details? Done.
I find your worldview pretty warped as well, one in which you fail to understand the depth of your responsibility to be a good neighbor to others. We simply cannot fathom any way that we could take responsibility for the effect we have on others, and refuse to think creatively to find solutions to the problem. Easier to send complaints to /dev/null for 6 years. |
Again please go back to as much as a single comment above, I feel like I'm constantly repeating my previous points: The majority of instances will simply not include the preview! This will unfairly penalise other software, either causing us to force them to include it or leave those posts without a preview. Go and submit a proposal to W3C if you wish to make an change to the ActivityPub protocol so everyone can have feedback on it, it's not mastodon's right to force changes to the protocol by suddenly requiring a specific preview for links. |
|
Oh no, a preview might not be shown! How horrifying! |
|
Being able to change the protocol isn’t the issue, it is a trust issue. Consider this scenario where the link preview is included in the federated post:
This kind of attack has happened on Mastodon before. I was a victim of it in 2017, and had modified posts with my name and face saying hateful things spread across the fediverse. That is why remote servers fetch the original post instead of trusting the server that delivered it. I’m not saying this issue isn’t isn’t worth fixing. But I am saying that there is no easy fix. I’m quite tired of people on both sides of this. “Just configure your servers properly.” “Just federate link previews.” Both of these ideas are equally ineffective and betray how little empathy and understanding each side of this debate has for the other. Grow up. |
|
Since the malicious instance hosts the original poster, they could also do any number of things for which there is no reasonable mitigation could be employed:
Signatures would be one way to solve the problem of trusting intermediate servers (and Mastodon already fetches the original post anyway, for the reasons you mentioned), and the original server has to be trusted because it is on the other side of an airtight hatch. |
|
We can’t trust the original instance to accurately represent a link in posts, either links originating from that server or boosted to it and further boosted onwards. Signatures can’t solve this because it’s about accurately representing out-of-band websites that aren’t on the fediverse. “etc etc etc” |
|
Okay, I think I understand your reasoning. Can you explain to me what the worst case scenario you're imagining would be if servers trusted the preview details provided by the original instance? |
|
Brainstorming a few other solutions, some of which have been mentioned before: Detecting abuse A server could randomly sample (say one in every 100) the original preview and compare it against the federated preview. If a discrepancy is found, it could automatically make a report to the instance admins, or the software could mark the server as untrusted and (1) start flagging previews from that instance in the UI with a warning or (2) start unconditionally fetching previews itself for posts originating from that instance. The exact sample rate could be subject to some bikeshedding, perhaps an instance reputation value is maintained and the sample rate is higher for new instances. Or, servers which do a random sample could forward their results when federating the post, so other servers can find out about the malicious behavior without having to roll a nat 20 themselves -- ideally following up on the report by forcing a sample. Reducing the load Lazier loading.
More brainstorming is encouraged. Let's think proactively about solutions instead of throwing up our hands. |
|
More brainstorming is needed, I agree. I already outlined a worst-case scenario. Frankly, I don’t have the emotional bandwidth to engage in a discussion that is so fraught with mutual disrespect. This issue is a real problem. It should be re-opened and addressed. There is always some solution, and I hope we find and implement it soon. |
|
This discussion is so weird, I can't even tell if anyone has proposed a possible solution? I don't understand why it is turning out like this. There's two questions, right? And we have to answer (1) before we bother to consider (2).
Someone said it was a protocol issue, which presumably means it needs to be dealt with in ActivityPub before it can be considered in Mastodon. But then someone else said it's not a protocol issue. I genuinely haven't managed to glean anything else from this discussion. |
|
@Cassolotl Should we do it? Yeah. It doesn't cost much to have a |
|
Possible mitigation: Do not pre-load the preview. Wait until a user
interacts with the post. And by interact, I mean something more significant
than just having the post appear in their client.
Perhaps a "load preview" button?
There's no point in showing something that nobody has any real interest in,
is there?
Now for question 2:
While it would be great if every server was set up to gracefully handle
spike loads, site owners understood and used reverse proxies and CDNs, and
CMS devs engineered their platforms to use resources wisely (looking at
you, Drupal devs (and I'm not even looking at Wordpress, 'cause they're not
worth the time.))... That's just not reality.
It's been my personal experience working with web dev agencies that real
estate people know the least about web technologies, set up the flashiest
and least efficient websites, and increasingly depend on an online presence
to find and keep customers. (Note: anecdotes shouldn't be construed as a
statistically significant trend. I've had bad experiences with very
entitled real estate agents. That doesn't mean all are. I'm still going to
make fun of them, though.)
So yes, we should do it. Because when that real estate agent makes a new
listing and their website has a carousel that lists 118 properties, 103 of
them hidden client-side because they've already sold, and all of the images
are 30kx40k unedited jpgs that are scaled down in the HTML itself, we
shouldn't be the ones triggering their $0.99/mo hosting plan to reboot all
10,000 websites on their 1990s era sparc server.
Yes, they've made bad decisions. But blaming them for us being the straw
that broke the proverbial camel's back is still victim blaming.
…On Thu, Feb 16, 2023 at 10:57 AM Cassolotl ***@***.***> wrote:
This discussion is so weird, I can't even tell if anyone has proposed a
possible solution? I don't understand why it is turning out like this.
There's two questions, right? And we have to answer (1) before we bother
to consider (2).
1. Can we do it?
2. Should we do it?
Someone said it was a protocol issue, which presumably means it needs to
be dealt with in ActivityPub before it can be considered in Mastodon. But
then someone else said it's not a protocol issue. I genuinely haven't
managed to glean anything else from this discussion.
—
Reply to this email directly, view it on GitHub
<#4486 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAB46PX32JF26PVYYO6BMBTWXZS7DANCNFSM4DVJI2DQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
We already have this, there is a random delay between 1 and 60 seconds. However for websites that struggle this much, I highly doubt increasing this further will help. I assume a website that goes down at 20 requests per seconds will also struggle at 5. This is just from what I've noticed from the slow websites I've been on. 20req/s means you have 50ms of CPU time available per request (assuming the worst; your server only having one core) My experience is that a website either is efficient (<50ms CPU) or inefficient (>200ms), meaning a delay won't change anything to most problematic websites. Keep in mind increasing this further also reduces the usefulness of the preview, generating it way later on means it's likely already further down on someones timeline.
This has also already been suggested, the issue with it is that at that point you may as well just click the link yourself. The issue is that we simply don't know if someone is gonna be interested in the link, which is why we grab it in advance, doing that after the user is already interested has no purpose.
Although I fully understand it's often not the reality, maybe it should be? If you're hosting any website, it should be reasonably expected you know what you're doing. That unfortunately a lot of people do not have the technical expertise, I fully understand, but at that point when you put your website online you're also doing this partially at your own risk. Keep in mind you can still just block mastodon based on the user agent, you can hang up a sign saying no. We're not forcing anything, we just assume by default we're allowed to. That said, having to block user agents is annoying, which is why I do support #21738, asking about implementing support for robots.txt
I know computers, I don't know anything about selling or buying houses, which is why I would hire a real estate agent. Configuring varnish or any other cache to do the bare minimum is really easy, it's hard to configure it in a way to protect you from malicious attacks though. Thankfully to be able to handle a horde of legitimate clients the most basic, allowing configuration will suffice. (Have the cache key include cookies etc., basically make the cache barely act as a cache unless clients are submitting the exact same request, which is what mastodon is doing). If setting up varnish (<15 minutes for unexperienced admins, <5 minutes for experienced ones) takes too much time, there are various cloud services like CloudFlare who'd do it for cheap or for free with basically no technical skills required. Look, I'll be honest, this is unlikely to go away any time soon. So instead of me just saying no, allow me to describe my current "best" solution: The only solution that would not significantly reduce user experience or break zero-trust would be to have the instance which the original poster belongs to doing the lookup, and including it in the message federated instances receives. This also means that the preview thumbnail is going to have to be hosted by the original instance, other instances retrieve it from there. Treat it no different then an image attached to a post. Signing (just as how the text in posts is signed) would make modifying the link or link preview impossible, especially if the signed value includes the image hash (making it basically impossible to change the thumbnail later on). Important to this is that if another user posts the same link, it's home instance does the lookup again. It doesn't matter if the instance has seen the exact same link before, it will never try to work with historic, possibly stale or poisioned previews it has received for that link before. This does mean however, that only a single request for a preview is made per post that includes the link, instead of 500-2000 requests per post. In case the home instance does not implement this, the post won't have a preview |
|
I think the solution is simple: I do not want my instance performing arbitrary HTTP requests on demand for anyone who has the ability to host a public key on the internet, so I should be allowed to configure my instance to not perform arbitrary HTTP requests on demand. |
|
There is a big difference between 50ms and 200ms of CPU time. It's not so binary that every website either has good performance or shit performance. Some routes are just more expensive than others; I have one which is frequently brought down by Mastodon DDoSes which would probably be spared at 5 req/s. Also note that I don't really have any empathy for the argument that places "user experience" ahead of not DDoSing other people. I can't stop stabbing you, the audience is loving it! |
I'll be honest, I don't have the data to further prove my argument, but I've worked at various projects for customers regarding scalability and in my experience most sites will either crumble under more then a few requests per seconds, or survive hundreds per second. However I can still understand your point, which is why I eventually ended up posting what imo would be the most logical solution moving forward.
I'm not looking for empathy, I'm simply sharing my opinion and vision on the issue. Other then our slight argument yesterday I have no intention of making it personal, and I'd recommend you doing the same. |
|
I did not make it personal? |
This has been mentioned a number of time and seems like an obvious solution with very little drawbacks IMO. I think it would also let admin think about whether they want to waste space dealing with previews and hopefully lead to an option not to enable previews. In that case, this part of the message might even be made optional and the instance could decide whether it wants to fetch it or not (I don't know whether that is doable in AP). |
|
This discussion isn't really working. It's six years old, with people coming in and out, discussions going in circles, and not a lot of presumption of good faith from anyone. I have collected all of the information I still think is relevant into a new ticket: #23662 Let's continue this discussion there. |
Why? I'd much rather keep this issue open, especially considering there is a lot of feedback and input in here already. Opening a new issue only adds to the already existing clutter (not to mention the 3.6k still open issues!), you shouldn't open a new issue just because someone closed the old one, or if the discussion doesn't go in the direction you preferred. Very much in favour of continuing the discussion here and if key maintainers of mastodon can agree this is still an active issue, they can always reopen it.
How is opening a new issue going to change any of this? We'd just lose six years of feedback. It's not like a new issue suddenly means people won't be "coming in and out" |
This issue was closed in 2018. |
Yeah but that doesn't mean you can just keep on making new issues lol, it was closed for a reason. If maintainers care about it they can re-open it. Would be something if every time someone closes an issue, and a person doesn't agree with that that they'd just open a new one. ddevault keeps complaining about "bad faith actors" in this issue and considering they haven't complained about these "bad faith actors", until I came along, it doesn't take much imagination to figure out who they're referring to. |
|
I feel that Fediverse need own fetching service all other apps can rely to. Not a single server, federated as well, but to be used across all Fediverse apps - Mastodon, Pixelfed, Calckey, even by Hubzilla or what Fediverse apps can appear in future.
|
|
It will be nice to have a solution for this problem... |
|
@Gargron This issue lists really good possible solutions for the problem. Guess it's time to finally address this issue: After all one of Mastodon's major goals is to be a good web citizen. Regarding link previews Mastodon currently is a bad citizen. This issue undermines Mastodon's primary goals, and good by built up by major players as a strong argument, why federation does not work. Therefore this issue should be given very high priority. A first quick fix could be to provide link previews together with the initial post and to only validate them opportunistically: Either randomly, so that only a given percentages of the Mastodon instantes validates. Alternatively each Mastodon instance could validate after a random grace period. This would keep trust intact, but make Mastodon a much better web citizen. @ddevault has proposed this very easy solution in February 2023, more than a year ago. A better solution can be implemented anytime, but done is significantly better than perfect. |
and others
Hi !
Today I found this tweet : https://twitter.com/mattiasgeniar/status/892446659245993984
I tried to post a link on my instance, Mastodon.cloud, and follow the link's web server logs ->
400 instant requests.
Imagine I flood 10 link, I think that go to generate more than 4k requests..
It's not great for link's web server..
Any idea how to mitigate this on futur releases ?
Why Mastodon need to crawl the link ?
Thanks
master(If you're a user, don't worry about this).The text was updated successfully, but these errors were encountered: