Filtering malicious URLs

[lemonsaurus]

We're currently filtering malicious filetypes in our filetype filter - to be specific, we're filtering all filetypes we haven't specifically whitelisted. This does not, however, apply to URLs. That means it's easy enough to get around our filter by posting the file in a DM or on a different server, and then just linking the content instead.

This also allows users to post URLs that literally point at viruses, which isn't something we should permit.

Let's write a second filter that uses the same whitelist and same logics to filter URLs as well.

Let's keep this one simple - no need to download the file and inspect it to determine exactly what its filetype is. Just filter by extension, the way we're doing now.

[MarkKoz]

Paste services sometimes include the extension in the URL which presents a problem. Either paste services would have to be whitelisted (there are so many out on the internet) or a separate whitelist would be needed for file extensions we don't want shared at all.

[lemonsaurus]

hmmm. that's a good point, @MarkKoz. Damn. I didn't think about that.

I'm not sure what the best approach here would be, but I'm open to suggestions. "more whitelists" isn't my favorite solution to this but I don't have a better suggestion myself.

[Anubhav1603]

I think regex will be good to use in this case

[Numerlor]

Not sure if this is doable without a whitelist of some kind; without keeping some kind of whitelist/blacklist for urls we could keep a list of extensions we don't want through (commonly malicious stuff like rar, exe, etc.) and if the url path ends with one of them, block the message. This shouldn't interfere with paste services since they'll end with extensions that won't be there, but some sites may use the "extension" for something else.

[Akarys42]

Since there’s no way to filter links based on extensions without having a lot of false positives, let’s close this. Feel free to add a comment if you can think about a better way of doing this.