Validating what a URL points to (adult content, scams, betting), not just its format

[fabioselau077]

I really like Laravel's validation rules, especially email:dns, which I use in every project. For URLs, though, url and active_url only tell us the link is well formed and resolves. They say nothing about what is on the other side.

I ran into this in a startup directory I maintain. Users submit their product's website and an AI generates a report about it. I didn't want adult content, scams or betting sites showing up on my home page, and every one of those submissions was also costing me an AI call. Checking the URL before calling the AI cut that cost a lot.

So I built rowbird/laravel-safe-url:

'website' => ['required', 'url', 'not_adult'],
'website' => ['required', 'url', 'safe_url:adult,gambling'],

It combines adult TLDs (.xxx, .porn, .adult, .sex), offline blocklists, Cloudflare's family DNS, Google Safe Browsing and optional page inspection, with caching and SSRF protection.

I know most of this belongs in a package, since it relies on external lists and services. But I'm curious: would a minimal first-party version make sense, for example a url:adult option that only checks the TLDs reserved for adult content and needs no external service?

Feedback on the package is very welcome too.

[shaedrich]

an AI generates a report about it

Not everyone would want to use AI for a variety of reasons. This functionality could be provided by other means. If Laravel was to force this particular implementation of this functionality upon its users, it should at least explicitly disclose this in the docs.

It combines adult TLDs (.xxx, .porn, .adult, .sex), offline blocklists, Cloudflare's family DNS, Google Safe Browsing and optional page inspection, with caching and SSRF protection.

I sincerely hope, you don't burn tokens for all of that, which could be checked deterministically. If you do, it is at best inefficient.

[fabioselau077]

Thanks for the feedback! I think my post wasn't clear, sorry about that.

The package doesn't use AI at all. Everything is deterministic: TLD matching, a local SQLite blocklist lookup, DNS queries and an optional HTML check. No tokens involved.

The AI is only part of my own app (the startup directory), and it has nothing to do with the package. The package runs before that AI step: if a submitted URL is adult content, a scam or a betting site, validation fails and the AI is never called. That's exactly how it cut my costs. I completely agree with you that this kind of check shouldn't burn tokens.

And to be clear, I'm not suggesting Laravel should use AI for this either. My question was only whether a minimal, deterministic check (like reserved adult TLDs) could make sense in core.

[shaedrich]

Ah, okay. Yeah, that cleared that up 👍🏻 Thanks for the explanation!

[Sophist-UK]

I am sceptical that a minimalistic 1st party validation that only checks the TLD will be adequate in any real use case. I doubt that there is any shortage of examples of adult sites that use .com.

[fabioselau077]

You're right, and that's exactly why the package doesn't rely on TLDs alone. The TLD check is just the cheapest first step. Most of the work is done by the blocklist (around 950k adult domains in the default lists, almost all of them .com) and Cloudflare's family DNS.

A useful check needs those external lists and services, with their own licenses and update cycles. That doesn't fit in core, so I agree this belongs in a package.

Thanks for the discussion! If anyone needs this today, the package is here: https://github.com/rowbird/laravel-safe-url