New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anti-spam #271

Closed
tobscure opened this Issue Aug 27, 2015 · 21 comments

Comments

Projects
None yet
9 participants
@tobscure
Member

tobscure commented Aug 27, 2015

Need to work out what anti-spam strategies to implement.

@Felli

This comment has been minimized.

Show comment
Hide comment
@Felli

Felli Aug 27, 2015

As of right now (on esoTalk) a plugin called Akismet works rather well, so maybe an extension of that?

Felli commented Aug 27, 2015

As of right now (on esoTalk) a plugin called Akismet works rather well, so maybe an extension of that?

@pcrumm

This comment has been minimized.

Show comment
Hide comment
@pcrumm

pcrumm Aug 27, 2015

@tobscure is Akismet integration (or something similar) you'd like to see in core, or better off as an extension?

phpBB also supports a pluggable CAPTCHA API. Maybe a middle ground is shipping an anti-spam oriented add-on to the extensions system and one or two stock anti-spam plugins.

pcrumm commented Aug 27, 2015

@tobscure is Akismet integration (or something similar) you'd like to see in core, or better off as an extension?

phpBB also supports a pluggable CAPTCHA API. Maybe a middle ground is shipping an anti-spam oriented add-on to the extensions system and one or two stock anti-spam plugins.

@tobscure

This comment has been minimized.

Show comment
Hide comment
@tobscure

tobscure Aug 27, 2015

Member

@pcrumm Extension, but one we would definitely consider bundling with the download.

I think there are a number of other strategies worth considering (don't allow new users to post links, etc.) I'll sit down at some point this coming week and nut out some ideas.

Member

tobscure commented Aug 27, 2015

@pcrumm Extension, but one we would definitely consider bundling with the download.

I think there are a number of other strategies worth considering (don't allow new users to post links, etc.) I'll sit down at some point this coming week and nut out some ideas.

@franzliedke

This comment has been minimized.

Show comment
Hide comment
@franzliedke

franzliedke Aug 27, 2015

Member

Antispam features should all be extensions, though certainly some can be bundled.

Member

franzliedke commented Aug 27, 2015

Antispam features should all be extensions, though certainly some can be bundled.

@pcrumm

This comment has been minimized.

Show comment
Hide comment
@pcrumm

pcrumm Aug 27, 2015

@tobscure eh, I've always favored doing as much heuristically as possible. I spent a lot of time on this once upon a time, and most other strategies just end up being continually cat-and-moused.

pcrumm commented Aug 27, 2015

@tobscure eh, I've always favored doing as much heuristically as possible. I spent a lot of time on this once upon a time, and most other strategies just end up being continually cat-and-moused.

@franzliedke

This comment has been minimized.

Show comment
Hide comment
@franzliedke

franzliedke Aug 27, 2015

Member

Since we're so JavaScript-based, I think we can employ some tricks like checking for certain browser features to make sure a human is browsing our site - there's not much we can do against those anyway...

Member

franzliedke commented Aug 27, 2015

Since we're so JavaScript-based, I think we can employ some tricks like checking for certain browser features to make sure a human is browsing our site - there's not much we can do against those anyway...

@tobscure

This comment has been minimized.

Show comment
Hide comment
@tobscure

tobscure Aug 27, 2015

Member

But spam bots will just interact with the API directly, presumably?

Member

tobscure commented Aug 27, 2015

But spam bots will just interact with the API directly, presumably?

@pcrumm

This comment has been minimized.

Show comment
Hide comment
@pcrumm

pcrumm Aug 27, 2015

XRumer (super common spambot software) tries to look like a normal user, so it'd use the frontend. If flarum takes off, I wouldn't be surprised if someone gets lazy and makes a tool that targets the API, though.

IMHO, that suggests that the most effective solution will be pretty client-agnostic, though that doesn't mean we shouldn't take advantage of additional layers in places where we can (e.g. the web client).

pcrumm commented Aug 27, 2015

XRumer (super common spambot software) tries to look like a normal user, so it'd use the frontend. If flarum takes off, I wouldn't be surprised if someone gets lazy and makes a tool that targets the API, though.

IMHO, that suggests that the most effective solution will be pretty client-agnostic, though that doesn't mean we shouldn't take advantage of additional layers in places where we can (e.g. the web client).

@tobscure tobscure referenced this issue Aug 28, 2015

Closed

v0.1.0 roadmap (old) #74

19 of 53 tasks complete
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 28, 2015

Here's what we do on our forum which has close to 200K users and 1M posts.

Depending on how the API is architected (i.e. plugins get executed on API calls) this is will take care of PhantomJS type bots and API spammers.

  1. Block At Network: Should support Country, CIDR or Individual IP address. Would be nice if it took threat lists from places like StopForumSpam (but unless you run a super huge forum, I doubt you need this.

We do this at the firewall level, so that PHP process time is not consumed. We also do this at the application level in PHP as it is sometimes just easier to add the ban to just the forum and not globally.

Can also block with user agent strings. We know which ones are badly configured bots!

  1. Block on Registration: Several things are done here. Identify time on page if it is below a certain threshold, we're looking at a bot. Ask a very very simple human question (what color is next red, white...). On submission, check for banned emails (wild-carded), send the registration IP and email to stopforumspam and accept registration only if everything passes. On rejection, give the user a contact which they can email if they're legit. Hardly anyone ever does.

  2. Block / Sandbox after registration. Check for registrations from a given IP address and old user cookies (stacked away in a cookie that does not get deleted, or in localstorage). If the user's old logins have been banned, send the new user to a silent / moderated group.

  3. New users have a configurable threshold of what they can post ( 0 links for example ). All links are generally nofollowed.

Other than that .. we do get human crafted spam which is hard to detect and very "bespoke". The users / community generally reports this.

We also use cloudflare to throw up human challenges to dodgy browsers etc.

ghost commented Aug 28, 2015

Here's what we do on our forum which has close to 200K users and 1M posts.

Depending on how the API is architected (i.e. plugins get executed on API calls) this is will take care of PhantomJS type bots and API spammers.

  1. Block At Network: Should support Country, CIDR or Individual IP address. Would be nice if it took threat lists from places like StopForumSpam (but unless you run a super huge forum, I doubt you need this.

We do this at the firewall level, so that PHP process time is not consumed. We also do this at the application level in PHP as it is sometimes just easier to add the ban to just the forum and not globally.

Can also block with user agent strings. We know which ones are badly configured bots!

  1. Block on Registration: Several things are done here. Identify time on page if it is below a certain threshold, we're looking at a bot. Ask a very very simple human question (what color is next red, white...). On submission, check for banned emails (wild-carded), send the registration IP and email to stopforumspam and accept registration only if everything passes. On rejection, give the user a contact which they can email if they're legit. Hardly anyone ever does.

  2. Block / Sandbox after registration. Check for registrations from a given IP address and old user cookies (stacked away in a cookie that does not get deleted, or in localstorage). If the user's old logins have been banned, send the new user to a silent / moderated group.

  3. New users have a configurable threshold of what they can post ( 0 links for example ). All links are generally nofollowed.

Other than that .. we do get human crafted spam which is hard to detect and very "bespoke". The users / community generally reports this.

We also use cloudflare to throw up human challenges to dodgy browsers etc.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 28, 2015

"2) Block on Registration: Several things are done here. Identify time on page if it is below a certain threshold, we're looking at a bot. "

If it's something that's going to be implemented please consider making a sort of recovery mechanism, like giving them another chance to use a different CAPTCHA or something if they fill out the signup form too fast. A few weeks back I tried to join a vbulletin forum that had this in place. I've joined so many forums in my life and used autocomplete, and so it permanently banned me with no option to go "no, I'm not a spambot"...

The easiest thing would probably be to have individual plugins for 3rd-party CAPTCHAs like those from Google, SolveMedia, KeyCaptcha, etc. Those tend to be quite effective.

Traditional helper methods like asking a special question unique to the forum is another method that seems to be relatively useful. Perhaps on a Chevy/GMC truck forum one would ask "how many letters in GMC" to which the obvious answer would be "3" or "three".

The hard part is to make this all as user friendly as possible for both the member signing up and the forum admin.

As the person above mentioned, being able to report posts to forum staff is critical, not just for anti-spam but for general moderation as well.

Relying on something like CloudFlare is probably not a good idea.

ghost commented Aug 28, 2015

"2) Block on Registration: Several things are done here. Identify time on page if it is below a certain threshold, we're looking at a bot. "

If it's something that's going to be implemented please consider making a sort of recovery mechanism, like giving them another chance to use a different CAPTCHA or something if they fill out the signup form too fast. A few weeks back I tried to join a vbulletin forum that had this in place. I've joined so many forums in my life and used autocomplete, and so it permanently banned me with no option to go "no, I'm not a spambot"...

The easiest thing would probably be to have individual plugins for 3rd-party CAPTCHAs like those from Google, SolveMedia, KeyCaptcha, etc. Those tend to be quite effective.

Traditional helper methods like asking a special question unique to the forum is another method that seems to be relatively useful. Perhaps on a Chevy/GMC truck forum one would ask "how many letters in GMC" to which the obvious answer would be "3" or "three".

The hard part is to make this all as user friendly as possible for both the member signing up and the forum admin.

As the person above mentioned, being able to report posts to forum staff is critical, not just for anti-spam but for general moderation as well.

Relying on something like CloudFlare is probably not a good idea.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 28, 2015

I agree with the fall back. The time to registration is just one thing that adds to the list of measures.

Cloudflare is very good for very obvious detection. Mucks up api requests though. So that make be an issue.

Again, we need two or three measures to stop spammers. Before registration, on submission of any content (including profiles, private messages etc) and a good reporting mechanism for users.

ghost commented Aug 28, 2015

I agree with the fall back. The time to registration is just one thing that adds to the list of measures.

Cloudflare is very good for very obvious detection. Mucks up api requests though. So that make be an issue.

Again, we need two or three measures to stop spammers. Before registration, on submission of any content (including profiles, private messages etc) and a good reporting mechanism for users.

@tobscure

This comment has been minimized.

Show comment
Hide comment
@tobscure

tobscure Aug 29, 2015

Member

Gonna start with the low-hanging fruit:

  • Askimet
  • reCAPTCHA (the new version with the "I'm a human" checkbox)
  • Flood control
Member

tobscure commented Aug 29, 2015

Gonna start with the low-hanging fruit:

  • Askimet
  • reCAPTCHA (the new version with the "I'm a human" checkbox)
  • Flood control
@mtotheikle

This comment has been minimized.

Show comment
Hide comment
@mtotheikle

mtotheikle Aug 31, 2015

Contributor

As a way to get into Flarum and see how extensions work, I've taken an initial stab at the reCAPTCHA implementation. You can see it at https://github.com/mtotheikle/flarum-recaptcha and I'll have some more updates tomorrow / this week such as support for setting the keys in admin panel.

Contributor

mtotheikle commented Aug 31, 2015

As a way to get into Flarum and see how extensions work, I've taken an initial stab at the reCAPTCHA implementation. You can see it at https://github.com/mtotheikle/flarum-recaptcha and I'll have some more updates tomorrow / this week such as support for setting the keys in admin panel.

@tobscure

This comment has been minimized.

Show comment
Hide comment
@tobscure

tobscure Aug 31, 2015

Member

@mtotheikle This is fantastic! Would you be interested in working on this extension as part of the github.com/flarum organisation? reCAPTCHA is an extension we'd like to officially support and include by default.

Member

tobscure commented Aug 31, 2015

@mtotheikle This is fantastic! Would you be interested in working on this extension as part of the github.com/flarum organisation? reCAPTCHA is an extension we'd like to officially support and include by default.

@mtotheikle

This comment has been minimized.

Show comment
Hide comment
@mtotheikle

mtotheikle Aug 31, 2015

Contributor

@tobscure That sounds great! I should be able to get the admin stuff done tonight which I think is the only main piece really missing. If I complete that and don't find any issues, then it'd be great to have this packaged with next beta.

Contributor

mtotheikle commented Aug 31, 2015

@tobscure That sounds great! I should be able to get the admin stuff done tonight which I think is the only main piece really missing. If I complete that and don't find any issues, then it'd be great to have this packaged with next beta.

@tobscure tobscure added Meta and removed Feature labels Sep 1, 2015

@franzliedke franzliedke added this to the 0.1.0-beta.3 milestone Sep 3, 2015

@justjavac justjavac referenced this issue Sep 7, 2015

Open

Flarum v0.1.0 开发路线图 #3

18 of 53 tasks complete

@tobscure tobscure closed this in 415b68f Oct 22, 2015

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 31, 2015

I noticed this was closed. Have you added the reCAPTCHA? I ask because the 3rd party extension is not functional and the developer is MIA (missing in action) since September.

ghost commented Oct 31, 2015

I noticed this was closed. Have you added the reCAPTCHA? I ask because the 3rd party extension is not functional and the developer is MIA (missing in action) since September.

@tobscure

This comment has been minimized.

Show comment
Hide comment
@tobscure

tobscure Nov 1, 2015

Member

Oh sorry, I forgot to update this. Decided against first-party reCAPTCHA extension for now because:

  • We want to keep the scope of our bundled extensions down, leaving more non-essential things to third-parties
  • reCAPTCHA is ineffective against human spam, which is the type that Flarum is more likely to experience (given its JS-heavy UI)

Sorry for the confusion. Hopefully @mtotheikle will update for beta 3 when it's out (there will be updated docs!) :)

Member

tobscure commented Nov 1, 2015

Oh sorry, I forgot to update this. Decided against first-party reCAPTCHA extension for now because:

  • We want to keep the scope of our bundled extensions down, leaving more non-essential things to third-parties
  • reCAPTCHA is ineffective against human spam, which is the type that Flarum is more likely to experience (given its JS-heavy UI)

Sorry for the confusion. Hopefully @mtotheikle will update for beta 3 when it's out (there will be updated docs!) :)

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Nov 1, 2015

I'm disappointed to hear that. At the moment none of the features would be suitable for a large community.

Akismet Enterprise plan only allows a maximum 100,000 monthly checks. We have in the past used that in the first few days.

Flood control only goes so far.

The general idea would be to add a layer to keep nastiest from joining.

ghost commented Nov 1, 2015

I'm disappointed to hear that. At the moment none of the features would be suitable for a large community.

Akismet Enterprise plan only allows a maximum 100,000 monthly checks. We have in the past used that in the first few days.

Flood control only goes so far.

The general idea would be to add a layer to keep nastiest from joining.

@johnhearfield

This comment has been minimized.

Show comment
Hide comment
@johnhearfield

johnhearfield Jun 9, 2016

@Code-Name-Debian Good point. One idea is to hook into the User's Ip and check it again Stop Forum Spam and Project Honeypot, but this can cause false positives.

johnhearfield commented Jun 9, 2016

@Code-Name-Debian Good point. One idea is to hook into the User's Ip and check it again Stop Forum Spam and Project Honeypot, but this can cause false positives.

@lZzozZl

This comment has been minimized.

Show comment
Hide comment
@lZzozZl

lZzozZl Jan 17, 2017

[Suggestion] patternLock.js

It has a CAPTCHA option which is fun. It's like android phone unlock. 👍

lZzozZl commented Jan 17, 2017

[Suggestion] patternLock.js

It has a CAPTCHA option which is fun. It's like android phone unlock. 👍

@luceos

This comment has been minimized.

Show comment
Hide comment
@luceos

luceos Jan 17, 2017

Member

@lZzozZl please report feature requests on the forum, the github bug tracker has been given the sole responsibility of bug reporting and approved feature conceptualisation a while ago. Thanks for sharing your thoughts and I do hope you share them through the correct channel.

Member

luceos commented Jan 17, 2017

@lZzozZl please report feature requests on the forum, the github bug tracker has been given the sole responsibility of bug reporting and approved feature conceptualisation a while ago. Thanks for sharing your thoughts and I do hope you share them through the correct channel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment