Proposal to edit 4.6: Security tooling

[codewordcreative]

Success Criterion - Security tooling (Machine-testable)

Current:
Web browsing from bots has been steadily increasing in recent years. As such, it is a growing concern for security, performance, and sustainability. Use security tools that automatically block bad actors and minimize bad behavior. This results in substantially less load on the server, fewer logs, less data, less effect due to compromise, and more. The result of compromised websites is a large increase in HTTP, email, and other traffic as malicious code attempts to infiltrate other resources and exfiltrate data. Compromised websites are typically identified by anomalous patterned behavior.

Suggested:
Follow best practices to block unwanted and unnecessary third-party crawlers from accessing or downloading your content. This includes adjustments to both robots.txt and server access rules. In the process, take care to ensure your content remains accessible to search engines and any helpful, welcome crawlers. Preventing access to suspicious user agents, unwanted visitors and scrapers reduces emissions. Where these scrapers are seeking training data for large language models and similar technologies, there is an additional third-party impact to consider as your data is processed and used to expand their model(s). Blocking unwanted visitors will save energy, preserve performance, and reduce burden on your hardware, which could reduce your need to provision more resources.

Additional information:
AFAIK, Siddhesh has assembled a list of resources and studies relating to crawling and LLM impact that can be included here to accompany the guidelines.

My comments:
I'm adding this here so we can more easily comment on a more substantial edit.
Actual solutions are a little thin on the ground right now. Cloudflare does a lot. 8G/Perishable Press firewall stuff does a lot. And Dark Visitors collects a lot of data, although the free solution only covers robots.txt. Overall, many seem to not understand the need for server-level protections, and the impact of failed attempts. We may need to add a specific criterion that simply points to the security guidelines to say "do this" - and this is the "also do this, to maximize sustainability".

Ryan's comments on the security section in general:
I feel like Security is an area where we identified there should be a cross-reference to other W3C guidelines, instead of writing in depth here? And/or I would move this under a separate numbered guideline of its own.

Ryan's comments on this section:
Rose and I worked on this text together. I'm not positive it belongs under Automation, but it's specific enough now to be less about security and more about automatically blocking bad bots.

Credit: Edit drafted directly together with @ryansholin, following discussions with @systemstree (Siddhesh Wagle).

[systemstree]

Thank you @codewordcreative

Sharing few articles here to support your proposal. These articles show how damaging AI crawlers are for the entire internet in terms of both bandwidth and ethical perspectives.

AI crawlers need to be more respectful
AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%
Attack of the AI crawlers

[TzviyaSiegman]

Thanks for raising @codewordcreative, @ryansholin, and @systemstree.

For those who had trouble finding it, this is 4.6

We need to wordsmith. I'm not a fan of "Follow best practices to block unwanted and unnecessary third-party crawlers from accessing or downloading your content," because the immediate question is, what are best practices? Because this is a document about sustainability (and not fundamentally about security), it might work better to make this something like:

Block unnecessary and unwanted crawlers from accessing or downloading your content. Adjust robots.txt and server access rules as needed. Preventing access to suspicious user agents, unwanted visitors and scrapers reduces emissions. Consider that scrapers may be used to inform and enlarge large language models.

Would this be sufficient? I'm trying to at once narrow the focus of the SC and make the language more spec-y.

[codewordcreative]

Yeah, that part about best practices works. It's a legacy from the original thinking - which was that we should also follow best practices in terms of security. How are we covering the fact security is a MUST for sustainability? This is essentially plugging the gap: Sometimes secure websites could also benefit from blocking extra traffic, even when it does not pose a threat (as in, static websites can benefit from insta-blocking any visitor attempting to probe for WordPress vulnerabilities, even if they have no database to crack into - wasted data transfer).

Block unnecessary and unwanted crawlers from accessing or downloading your content. Adjust robots.txt and server access rules as appropriate. Preventing suspicious user agents, unwanted visitors, crawlers, and scrapers from gaining access reduces direct emissions. Certain scrapers may be used to obtain training data for large language models, which can have an indirect impact on emissions.

• Preventing access to is grammatically confusing (as in, who is preventing access to what?) and could cause problems in translation/reading by non-native speakers, so I've reworded that.
• As needed - I would say "as appropriate" to encourage more active engagement with the idea. Not a sort of, "Meh, no, we don't need that" attitude.
• I added crawlers - since some things people may wish to block could be SEO tools used by competitors, for example, which is useless to the organization themselves anyway, pretty harmless, but does have an impact on their servers.
• I would specify direct emissions to make it less confusing when we talk about indirect emissions (which we really should talk about).
• Too many action verbs when it's not really an action but an explanation makes it a bit harder to parse (e.g., "Consider") - reworded slightly.
• I think we do need to explain that it is training data use specifically that is more of an issue, and not all scrapers serve that purpose. Some crawlers like Web Almanac or scrapers like Wayback Machine are harmless or even positive.
• I would also then want it to be clear why this is an issue - as in, the snowballing effect of scraped data in LLMs.

The part I haven't edited back in:
"In the process, take care to ensure your content remains accessible to search engines and any helpful, welcome crawlers."
This was @ryansholin's suggestion, I think? But I did agree with it. A friend accidentally de-indexed her website recently with robots.txt rules put in the wrong order. But maybe it is hand-holding too much to tell them to be careful...?

These aspects could perhaps be moved to the benefits section, which is non-normative anyway:

Environment: Taking action to prevent your content from being scraped reduces the scope of training data available for the large language models they may be feeding. This reduces indirect emissions associated with processing and using that data.

• My thinking is that with that section being more free anyway, the explanation can go there. That's happened with some other SCs.

Economic: Blocking unwanted visitors will save energy, preserve performance, and reduce burden on your hardware, which could reduce your need to provision more resources.

• This is obviously more of a benefit to start with.

Could potentially mention the benefit of blocking SEO tools you don't use as an economic benefit, but it's a bit... granular. It's a very good idea for e-commerce, but less relevant if you just don't have so many sophisticated competitors using all the SEO tricks.

[andreadavanzo]

I agree with @TzviyaSiegman 's comment.

I would like to raise one observation. The word “block” may still feel a bit strong or contradictory to the W3C principles of openness, access, and interoperability. While I support the intent, I think we need to accept that bots, crawlers, AI agents, etc. are now a permanent part of the web infrastructure. Since this document is about web sustainability, it makes more sense to address this class of "user agents" explicitly, not necessarily from a security standpoint, but by acknowledging them and encouraging responsible access and limiting unnecessary automation.

Moreover, I don’t think we should judge crawlers as good or bad. Regardless of scope, crawling as an action has consequences and therefore cannot be entirely harmless or purely positive.

[AlexDawsonUK]

In the original SC, block was determined by tooling and thereby it was objective and not subjective to the individual as its a given that while bots and crawlers and agents are apart of the web's infrastructure, bad actors can and will abuse this position thereby it should be left to objective measurement and detection to identify where those boundaries are.

I agree that "Block unnecessary and unwanted crawlers" is much more open to interpretation and it could be problematic for those trying to implement the guidelines as the threshold for what IS necessary or wanted may not be obvious or easy to measure, especially for sustainability purposes - leaving implementors unable or unwilling to meet the criteria.

[andreadavanzo]

I understand the point about tooling but I would be cautious about calling it objective. Tools are made by humans and apply human-defined rules.

[ryansholin]

Maybe "prevent" is less judgmental than "block?"

I'm thinking of past experiences working with customers with high traffic websites that were frequently indexed, often by expected and invited spiders like Google's, and sometimes by aggressive crawlers from SEO tools they had forgotten about, scrapers that reproduced their content on their own sites, and lastly, AI companies they would prefer to block and negotiate with for more controlled access.

All of those had an impact on performance and energy consumption; some of them were unwanted; few of them were malicious.

[andreadavanzo]

I’m not sure “prevent” is better than “block”, both still carry an implication of access control that feels at odds with the open nature of the web. A better framing might be about setting expectations, not enforcing any kind of exclusion.

Think of it like this: if I have a house with a garden that people are welcome to walk through. I’ve made a shortcut as space public, but if I’ve just planted new flowers. I might put up a sign that says “Please don’t walk on the fresh flowers”. I’m not locking anyone out, I’m expressing my intent clearly.

That’s how I think we should treat bots, crawlers, and AI agents: by providing strong, explicit signals, like robots.txt, meta tags, or custom headers, not by trying to algorithmically decide who’s welcome and who’s not.

[codewordcreative]

That's sounding more like a philosophical disagreement with the principle itself. Which is fine, but the focus here needs to be on sustainability and reducing carbon impact. Cloudflare has similar rules to my own custom rules, for example. It's about that bandwidth impact and later processing impact (in the case of scraping), at the end of the day, and crawlers and malicious bots have a huge negative impact. There is a huge scaling impact as they process all the content they scrape. As we've seen, polite requests just don't work. Nor is the impact reasonable.

Malicious bots should be blocked - even WordPress vulnerability probing on static sites - because all that probing is involuntarily increasing your carbon footprint.

And when content is scraped without consent, it is arguably malicious. If you block the same bots that you requested don't crawl you in robots.txt, you wouldn't see any change in bandwidth use if everyone - even these named entities - respected the wishes of webhosts. But you do. Some neighbourhoods, it's safe enough to leave your door open. That's cosy. But the Internet isn't so cosy these days! I lock my door.

As for how to resolve it: We've already got block and that's a technically correct term. We could however say restrict access to, since rate limiting would also help a lot.

Regarding unwanted and unnecessary - I feel rewording would discourage people from engaging with it and making conscious choices. Unnecessary could be more problematic, though - since it may sound like we are deciding what is necessary. We could look at wording to encourage monitoring traffic, too - in addition to this and perhaps to underline the importance.

Maybe, somewhere, we need:
Monitor bandwidth use and attempt to identify any suspicious or particularly heavy users.

I feel like the idea of what is suspicious, unwanted, ubbexessae, or malicious is something that is derived from analysis - so we need to refer to the analysis in some way.

Wikipedia has reported a 50% increase in bandwidth being spent on multimedia requests.