Nothing seems to stop registration spam Plone 4.2, 4.3 #632
Comments
|
i just checked the registration form of the site http://www.spokanerpg.com/@@register it seems your captcha is always asking the same question: "Say farewell in Tolkien's Elvish" although i don't have an idea (and - shame on me - could not come up with a working translation within 1 minute) i guess other people can do so. the key with captchas is that they need to be random |
|
General discussion in IRC included suggestion to use collective.recaptcha with version 2 recaptchas, but beyond that that it would be nice to change the registration process so that emails went out FIRST before users are created. |
|
I had many different Q/A responses. I had just temporarily dropped it to one while working with the IRC folks. And I have been completely making up utterly obscure, including just random pwgen-based passwords that don't fit the question, just for testing. It seems more that captcha, whether old captcha, recaptcha, or norobots, are seeming to be simply bypassed. I am getting recaptcha v2 setup right now for testing, but I don't know that will make any difference. But I am trying anything. For testing sake for you guys, I have re-enabled self-registration, and added back a variety of Q/A examples. Thanks for trying to help. |
|
@merpdotcom you're currently not using version 2 of recaptcha. That seems to be the most promising suggestion at this point. ...It'd be nice if we could get someone who hosts websites that have self-registration chime in and let us know what works best. I've never hosted a site that allowed self-registration. |
|
I've been hosting self-registration for dozens of community sites since the late 90s. In 2003/4 I switched them over to Zope-Plone. It had some challenges with registration spam, forum spam, contact spam (but was much better than more popular platforms at the time, for example the PHP and Perl-based CMS's), but manageable. Ever since 4.x it has become completely unmanageable unfortunately, and just getting worse. Hopefully we can figure this out. I'm still trying to get the buildout to actually upgrade the recaptcha version. Hopefully I will have that working soon, then will see if any difference with v2. |
|
It doesn't really have any difference with Plone 4. The difference is the OOTB Plone is not usable with self registration right now though it seems.
|
|
That does unfortunately seem to be the case. :-( |
|
@merpdotcom Can you try installing this package: https://pypi.python.org/pypi/collective.emailconfirmationregistration It's rather simple but if it works out well, it's a concept I'd like to consider merging into Plone core. |
|
👍 we, at freitag.de, use this approach (first confirm the email, then register) and so far works perfectly. |
|
First confirming before registering seems a much better route (thought that was what the core feature was supposed to anyway, rather than the temporary accounts piling up). @vangheen I will try that too. |
|
@vangheem , what method should I use to install it? I tried as standard buildout and it didn't find, then tried adding as github source. Do I need to do something specific to add pypi-related products? My apologies for not having that down, appreciate directions for installation of the emailconfirmationregistration product. Thanks. |
|
we're using if you activated member folders, you can also use the following script to clean up spam users. it checks for users w/o a member folder and deletes them (it does not delete users with an active password request): def _removeSpamUsers(portal):
pm = portal.portal_membership
pwr = portal.portal_password_reset
mdtool = portal.portal_memberdata
acl_users = pm.acl_users
ids = pm.listMemberIds()
logger.info("Checking %d users for spam-accounts (w/o a userfolder)" % len(ids))
#userids that requested a password reset
#{'aaron': DateTime(xxx), 'seth': DateTime(...),...}
pwd_requests_by_user = dict((user, expires) for user,expires in pwr._requests.values())
#all available member folders
member_folder_ids = pm.getMembersFolder().objectIds()
delCount = 0
for memberId in ids:
if not memberId in member_folder_ids:
if memberId in pwd_requests_by_user:
# skip members that just registered and are about to reset their passwords
logger.info("Skip %s with password reset request (expires %s)" % (memberId, pwd_requests_by_user[memberId]))
continue
member = pm.getMemberById(memberId)
name, email = (member.getProperty('fullname'), member.getProperty('email'))
logger.info("Delete %s - %s (%s)" % (memberId, name, email))
mdtool.deleteMemberData(memberId)
pm.deleteLocalRoles(portal, [memberId], reindex=1, recursive=0 ) #speedup: recursive not needed, member does not have any content
acl_users.userFolderDelUsers([memberId,])
delCount += 1
logger.info("Deleted %d spam members out of %s initial members" % (delCount, len(ids)) ) |
|
@vangheem It is installed now. Is there anything to configure or should it just work now? |
|
As far as I can tell, it is still creating the account first. Maybe I'm not understanding what to check for you. Please clarify what I should be looking for so I can assist and testing it out. Thanks. Meanwhile I did finally get recaptcha v2 working correctly. |
|
@vangheem, this is the test server registration page:http://www.spokanerpg.com/@@register feel free to hit it as needed. I deleted all created accounts except my admin-level account (there were three test accounts I had created, plus two others I didn't know from api.remailed.com (maybe someone helping test?). So slate is clean, any accounts created are either you/us testing, or something else. Here is how the buildout.cfg looks now that recaptchan v2 is working, but I also tried your product. Buildout ran it. And I was able to enable it through site-setup products add-ons web page. Thanks for trying to help to figure this nightmare out, I am going to have to call it a night for now. (5:31 am here). Recaptcha v2 at least it appears to be working now, had to use the github grab (the pypi one has a zcml error they (jensen in IRC said) are fixing). I don't think the emailconfirmation product is working yet, when I run a test, it creates the account as soon as I complete the form/recaptcha. I do receive the activate email with the link. But even without responding/clicking it, the test accounts are now listed. Meanwhile will see if the spam registration bots still get past the recaptcha v2. I haven't slept in 93 hours trying to get this resolved, so will hit the hay for a while, then see if anything happened while I slept. And I will check back here for any directions to follow for testing your emailconfirmation product. Thanks for all the help. If this is finally fixed after a year of over 20+ previously vibrant community sites having user registration disabled, it will be such a relief to get the communities alive again. I will let those that helped know how it goes. It might take a few days to a week to be sure (the bots sometimes don't attack right away). thanks again. ciao. |
|
Sorry, I went to sleep myself. Looks like collective.emailconfirmationregistration and collective.registrationcaptcha both customized the register form and collective.registrationcaptcha version was being used. Please upgrade to the latest version of https://pypi.python.org/pypi/collective.emailconfirmationregistration/1.0.0a3 that I just released that should fix it. Let us know if you need help/instructions on installing a specific version of a package with buildout(look at your versions section in buildout.cfg). |
|
Ok, I've tried pinning the version. I deleted all previous src files, but it is still grabbing a1 for some reason. Here is grepped buildout -vvv output, and the buildout.cfg You see what I am doing wrong? Thanks for guidance. http://pastie.org/10246074 |
|
@merpdotcom couple things here:
You don't want to run from master in github, you want to download the released version 1.0.0a3. |
|
@vangheem thank you, that did the trick. 1.0.0a3 now in place and testing... tried for valid email address (only field offered), after a bit of a wait, "Warning: Could not verify email address you have provided ". I do not see anything in client1/event.log nor zeoserver.log. Suggestions? |
|
hhmm, I'm using https://pypi.python.org/pypi/validate_email to validate the email address with the verify=True to check to make sure the email really exists. I can just remove that check. I thought it'd be good though--worked in my tests. If there was a long wait, perhaps the server isn't allowing the outbound connection for checking from the python process? |
|
@merpdotcom fwiw, I made a 1.0.0a4 release that disables that check. Maybe give that a try. |
|
@vangheem Email: You have requested registration, please confirm your email address by clicking on this link. If that does not work, copy and paste this urls into your web browser: http://www.spokanerpg.com/@@register?confirmed_email=merpdotcom@gmail.com&confirmed_code=a67ff23c4d6f51b725b10ab25846f511e387be79 Clicked link: We’re sorry, but there seems to be an error… The error has been logged as entry number 1434596062.910.682308390256. Error from client1/event.log: 2015-06-17T19:54:22 ERROR Zope.SiteErrorLog 1434596062.910.682308390256 http://www.spokanerpg.com/@@register Tried copy and paste of the link provided for copy and paste: 2015-06-17T19:57:33 ERROR Zope.SiteErrorLog 1434596253.430.619347658767 http://www.spokanerpg.com/@@register |
|
For testing right now it is fine without recaptcha, but for production, would be important to have at least recaptcha (optionally both recaptcha with norobots option on same page as double check, would be ideal), so that spammers can't use the form for backscatter attacks, or just overloading my poor mail server. :-) But we can worry about those features later. Happy to test away! |
|
sigh, sorry about that. Like I said, I didn't test much. Just made another release to fix 1.0.0a5--give it a try please. |
|
As for the validate feature in a3, the test server is running from my home over a Comcast cable residential connection. I am testing using a valid gmail account. It is possible since it is coming from a dynamic IP from a residential connection that added delays or checks are interfering. I will be moving this server to the colo facility this Friday. It will become production for the sites (I have to do this, older server is overloaded), but I can setup a separte test instance to test on the colo from time to time. I will be wiping the old server (once everything transfered), and it will be setup identically as the new prod server, so I can use the dev server from home for staging. So most testing will still be from home, but there will be this transitional period of several days (weeks?). When the changes happen, if we are still working on this, I will specific with each test result whether it was on home-dev server or colo-prod or colo-dev instance. Okay? |
|
No worries about the errors, each iteration is making progress. I am happy to be your dev code tester. :-) (retired CTO/CIO) |
|
Testing a5. If that does not work, copy and paste this urls into your web browser: http://www.spokanerpg.com/@@register?confirmed_email=merpdotcom@gmail.com&confirmed_code=e452a6061b883ef58fae3b87f5a5535eacc03075 Clicked first link, presented with full registration form now, with Recaptcha. Welcome! You will receive an e-mail shortly containing a URL that will allow you to set your password. When you receive this e-mail, please follow the link to complete your registration process. NOTE: The password reset URL will expire on Jun 24, 2015 08:12 PM. If your password reset request expires, you may use the "Forgot your password" link to generate another password reset request. Excellent! Received user account creation email with link to "Activate" account (Is it possible to change the duration that is valid to something more like 24-48 hours?) Prompted by passwordreset page: NOTE: Did not generate error in client1/event.log Well, we're getting closer by the moment! :-) Actually, this might be user error. I think my web browser auto-completed a different username than test, and I think your code caught it, but didn't have any exception handling to point out it was a different user name. WIll test that in a second. But here is with following the desired use-case with no attempts to enter wrong email or username: I can see some users being annoyed by all these steps (two emails to set account and then set password), but hopefully in this day and age users will be more understanding about how problematic it is to fight the spammers. Tested again, just cooperating and using the valid email the entire process instead of trying to "trick" it, Okay, verifying if that password creation page error was from entering a different username than previously validated: I then went into my Site Setup > Sercurity, and enabled "Let users select their own passwords", to make life easier for users, since at this point we've gone through email validation, and then recaptcha, it might be robust enough to get away with it? EXCELLENT! |
Yes, your comments about there being too many steps are justified. Long term, this form needs to be more streamlined. I think, for now, it's important we get filtering out the bots the best we can though. Thank you for testing. Please update us with info as far as if bots are able to automate the email verification steps. I think there are a few more tricks we can try. For instance, even doing the honey pot stuff is really easy to implement and might help. Unfortunate the email dns lookup didn't work. I wish there was an open web server api or something for checking an email against know bad domains/email addresses or something. Annoying problems spammers are these days... |
|
@vangheem This is an excellent start, and well worth it! There are RBLS blacklists, graylists, and white lists versus IP addresses (I use them for my mail servers constantly). As for email addresses, there wouldn't be any point, they can autogenerate any bogus email they want, so it would seem to be too much overhead with too little potential reward. At least with the option for IP-based blocking, known attack servers or infected users, could be blocked, But you are getting into a whole morass of technologies there. I think what you did here is a much cleaner approach. And if it works to stop them fully, the Plone folks should definitely consider adding it as a checkbox or standard product to bundle to help many others. :-) Cheers! |
|
@vangheem Oh, though if possible, it would be very good to have the Recaptcha option on the first step email form entry, otherwise backscatter will be a serious problem. Is that trivial for you to add, or more complex? |
|
Sure, it'd be easy to move it there. Then we'd just remove the use of that
|
|
That would be great. Then fewer inter-package dependencies, and putting the recaptcha at the first layer (email). Seems cleaner. Thanks again! |
|
@vangheem |
|
@vangheem |
|
Alright, give version 1.1.0b2 a try. This moved the recaptcha field to the email verification page. Additionally, it adds a honey pot hidden username field. |
|
@vangheem |
|
@merpdotcom great! Thanks. @tisto @bloodbare @frapell others? Does it make sense to try and squeeze this into plone 5? |
|
I would definitely vote for it! :-) |
|
This might be what you want down the road @vangheem |
|
@vangheem interesting feature/bug? |
|
@merpdotcom thanks for your help on this. I am closing the issue now. For now, we'll advise people to install collective.emailconfirmationregistration on Plone 4. I'll see if it is reasonable to merge this into plone 5 during the anniversary sprint; otherwise, this will be PLIP'd for plone 5.1. |
|
Bad news, it appears "they" have figured out how to bypass on at least one of my sites now. :-( There are now over 140 fake registrations on the tolkienmoot website that had been working fine to stop them. This is version 1.1.0.b2. Please advise on what I should do to get you information on how they are once again bypassing? I will check the other sites to see if any others are showing registration spam, or if it is only one site. |
|
Some additional sites indeed have been bypassed, brencode, hawketalkshow, omzine, rpgresearch (over 500+!), tolkienscholars, but not merpdotcom, spokanerpg, or half a dozen other sites. The rest (~20 I still hadn't enabled self-registration pending the results of this testing). I am disabling self-registration once again for the "compromised" sites until we can figure out what to try next. Thanks in advance for any suggestions you may have. |
|
Does this issue need reopening? |
|
Version 1.20.b2 has now been created to add the option for an admin to verify/authorize a registration request as an additional layer to try to combat this problem. b2 has a bug that doesn't send an email if "let use select own password" is checked, that should hopefully be fixed soon. I am now testing b2 on the several sites that were still getting bypassed to see if that works. https://pypi.python.org/pypi/collective.emailconfirmationregistration |
|
Crud, it looks like somehow it is still being bypassed. Accounts are being created without ever going to the admin for approval. :-( 17 accounts since yesterday. |
|
I don't think this issue needs reopening. We are dealing with issues on the add-on package tracker. We'll target 5.1 to move add-on into plone. |
|
The hotfix worked beautifully for a a while. But now it looks like they have figured out how to bypass it again. :-( Now about 13 new bogus accounts in about 2 days. Nothing in the "Review Registration Requests". Have they figured out another Plone vulnerability? http://techtalk.hawkenterprising.com/news/important-plone-hotfix-20150910 |
|
@merpdotcom maybe in future if you suspect a vulnerability could you email the security team directly rather than post here (a public place)? security @ plone.org |
Registration spam has been completely out of control for over a year now.
I have had to disable user self-registration on all sites.
I kept being told to upgrade to newer Plone, and to use captcha/recaptcha.
A long process with 20+ previously active community sites.
Finally managed to setup fresh server with 4.3.5 (on linux).
I added collective.registrationcaptcha and collective.z3cforms.norobots.
Configured no robots with extremely esoteric and oddball questions.
Example url: http://www.spokanerpg.com/@@register
I enabled self-registration, but disable "let use select their own passwords", enable user folders. When I test registration page, it shows the correct norobots prompt.
It will not let me register until correctly answered.
But I am getting horrible registration spam, 20-50+ per site per day on just the test server (thousands on the main server if registration enabled).
This is no different than on the earlier 4.2 and 4.1 boxes that were trying to use captcha, recaptcha, etc.
WIthin a few days there are thousands of bogus accounts, and the mail server begins to get loaded from all the registration email processing (many of which bounce as bogus emails).
Is plone registration being injection bypassed or some such?
Please help, this has completely crippled more than 20 different community websites that used to be active and vibrant.
Thank you.
Sample Email bounce from bogus registration (get hundreds of these per day, sometimes per hour:
Here is sample bounced mail to a bogus email account from a bogus registration:
From: Mail Delivery Subsystem <mailer-daemon@…> To: spokanerpg@… Subject: Delivery Status Notification (Failure) Date: Tue, 16 Jun 2015 09:29:29 +0000
Delivery to the following recipient failed permanently:
Technical details of permanent failure: Google tried to deliver your message, but it was rejected by the server for the recipient domain mail.bg by mx-balancer.mail.bg. [2001:67c:16b8::ffff:2].
The error that the other server returned was: 550 5.2.1 <erraticdepictio35@…>: Recipient address rejected: Mailbox is temporarily on hold, not accepting messages
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
X-Received: by 10.70.127.140 with SMTP id ng12mr57235605pdb.94.1434446967093;
Return-Path: <spokanerpg@…> Received: from d2d203.dev2dev.com (c-73-221-115-191.hsd1.wa.comcast.net. [73.221.115.191])
Message-ID: <557fec76.c9f5420a.7198.176f@…> To: erraticdepictio35@… Precedence: bulk MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Subject: =?utf-8?q?User_Account_Information_for_Spokane_RPG_Role-Playing_Gaming_Co?=
From: "SpokaneRPG.com Webmaster" <spokanerpg@…> Date: Tue, 16 Jun 2015 02:29:24 -0700
-- SpokaneRPG.com Webmaster
The text was updated successfully, but these errors were encountered: