-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Drop rule for in-addr.arpa #4
Comments
Hi, Quanah. Is there no potential for that string to show up in an EHLO? |
Hm, looking at are logs, we get both valid and spam emails that have .in-addr.arpa as part of their address. :/ I.e., the rule itself seems overly broad, as it catches anything with that address space. |
Who sends valid email with a poorly configured rDNS PTR? That's almost insane as few places would easily accept it. Proof that the rule does work: Nov 3 18:38:05 svr5 postfix/smtpd[17078]: NOQUEUE: reject: RCPT from unknown[200.52.74.178]: 554 5.7.1 <178.74.52.200.in-addr.arpa>: Unverified Client host rejected: Generic - Please relay via ISP; from=lisa.king@techsolve.org to=ddzevie@netcoolusers.org proto=ESMTP helo=<techsolve.org> |
Thanks, guys. I concur this is an overly aggressive rule and have removed it. Updated version is committed. Closing this thread. :) |
I strongly object. So far zero evidence has been provided to show legitimate email being sent that is blocked by that rule. We do have evidence, and history, to show that the rule in question blocks SPAM and Viruses. I find it very disheartening that it was so easy to get a well established rule so easily discarded. |
At the risk of sounding cheeky, jimpop, this would have been much more helpful feedback to me than your earlier comment of merely "Who sends valid email with a poorly configured rDNS PTR? That's almost insane as few places would easily accept it." I didn't understand what you were trying to communicate. I hope you'll be less disheartened that I'm happy to put the rule back in, until such time as legitimate evidence can be provided that it's actually blocking legit email. I appreciate your insight and input - |
We have legitimate emails being blocked by it, as I already stated. Since we removed the rule, spamhaus has been blocking those ranges where it's actually invalid, which is preferable in our case. The specific end group that we encounter this with is a security firm (Netragard). Nov 9 08:31:17 edge02 postfix/smtpd[50403]: NOQUEUE: reject: RCPT from unknown[38.83.192.4]: 554 5.7.1 <4.0-27.192.83.38.in-addr.arpa>: Unverified Client host rejected: Generic - Please relay via ISP; from=xxx@netragard.com to=xxx@zimbra.com proto=ESMTP helo=<mail.netragard.com> |
in fact, RBLs blocked every in-addr that was invalid before even getting to this rule. So it was only rejecting valid email. Nov 9 08:10:35 edge01 postfix/postscreen[6524]: NOQUEUE: reject: RCPT from [167.63.39.65]:53116: 550 5.7.1 Service unavailable; client [167.63.39.65] blocked using zen.spamhaus.org; from=romanova18820@mail.ru, to=xxx@zimbra.com, proto=ESMTP, helo=<65.39.63.167.in-addr.arpa> |
I'd also note in the case where valid email was being blocked, the helo was correct (mail.netragard.com). markup ate it though. Here it is one again in a code block:
|
Those may be legitimate emails, but they are being sent from a misconfigured system(s). It's not proper to ask everyone else to lower their shields in order to accommodate someone else's misconfigued system(s). |
In a perfect world, maybe. However, the system itself may not be misconfigured at all, but they are certainly using network services from someone who doesn't delegate rDNS to the company to setup as they please. This isn't too unusual in my own experience, in fact I've found it to be all too common. Trying to convince your SP to update rDNS can be difficult or sometimes nearly impossible. Penalizing a company because they happen to end up w/rDNS ending in in-addr.arpa seems like an overly broad stroke to impose on everyone. I doubt there's any good data to show if a rDNS name ending like this is proven to be more or less spammy than so many other possible names so ultimately it is just a judgement call. My preference would be that this list not take such a broad stroke in the fight against spam. RBLs help take over for whatever gaps this may leave, but at least those with no rDNS control get a chance while we still get the benefit of all the other rules that remain. |
So why is the project called fqrdns.pcre, why not just call it misc.pcre? |
As far as I understand fqrdns.pcre is mostly intended to block servers with dynamic and generic rDNS. So As readme says:
Why not to create white list for known legitimate servers as part of the fqrdns.pcre project? |
@moisseev: A whitelist is beyond the scope of this project, but if you or anyone else wants to start one here on Github, I'm more than happy to link to it from the README as as complementary project. :) |
Just so I can keep track (and by saying this, I'm not saying it's a democracy... I'm just trying to see where everyone is coming from), @quanah & @plobbes believe this entry should be removed, while @jimpop-at-gmail-dot-com & @moisseev believe it should stay. After discussing and considering, have any of you been swayed the other way? I'm wondering if it might be useful to also start/move this discussion on the Postfix users list, or perhaps even better, make a post over there inviting others who wish to chime in to do so here. My general hope for this file is for it to continue in the same spirit as when Stan Hoeppner originally started it, where the rules lean conservative, while more aggressive blocking (or more liberal whitelisting) can be handled individually by the mail admin after-the-fact. |
My opinion is the same. The rule is necessary, and in-line with the project's goal. Further, it is far easier for 1 or 2 people to remove the rule, rather than requiring 4000 or 5000 other users to add the rule after they download an updated version of fqrdns.pcre. |
Received this feedback via email from Noel Jones of the Postfix Users list:
|
The "zero false positive" thought process would basically invalidate any and all of the rules. Any and everybody can come up with an excuse to show how a rule causes them some inconvenience. |
So I guess the question boils down to: is this rule doing more harm than good? And what are the chances that a system that gets "caught" by this line wouldn't have been caught by something else (RBL, normally-used Postfix restrictions, etc.)? The goal of zero false positives will likely never be reached by anything, but it's still the ideal for which to strive. It's clear we're not going to reach consensus. I'm just looking for more people to chime in. Real world data and examples carry more weight in my mind vs. "what if" scenarios. |
Just don't forget that DNSBLs are after-the-fact (aka after some spam has started spewing), whereas NXDOMAIN and fqrdns.pcre are before the spew can begin. |
I don't want perfection to be the enemy of the good. We'll never achieve perfection, but that doesn't mean we shouldn't try for good. I think the best approach, which is in-line with the original intent of the project, will always be the most conservative one, then allow individual admins to add their own more restrictive rules, based on their needs. I haven't made a decision yet, but I will say that I'm inclined to believe the argument that when Stan originally added in-addr.arpa PTR to this list, there were no known ISPs using those for legitimate hosts. I'm also inclined to believe that's no longer the case, which is a believable argument for removal from a list whose primary purpose is supposed to be avoiding false positives, even at the cost of allowing a few more things through to be dealt with by other means in subsequent steps. If the ./in-addr.arpa$/ is removed from the file, that isn't the same as whitelisting those hosts. That doesn't mean mail from those systems will automatically get through. It simply means that they won't automatically be blocked. I've asked for real-world examples, not hypothesis. So far, I've only seen log file examples of valid mail that was being blocked by this rule. I haven't seen examples of invalid mail that have been allowed by this rule -- and that wouldn't have also been blocked by standard Postfix best practices. I'd love to see some examples of that, if any of you have them. I accept that the original README says:
But "a sender you know to be legit and from whom you need to receive mail " is different than "multiple senders that could be legit and from whom you might receive mail," particularly when things have changed since the origin of the file so that in-addr.arpa hosts now might be valid. So if you have real-world examples of invalid in-addr.arpa mail that wouldn't be stopped by basic Postfix preventative measures, for which the fqrdns.pcre file would have been your only reasonable hope, now is the time to share them. :) |
This is really getting out of hand. in-addr.arpa is generic, fqrdns.pcre exists to block generic. Competent MTA admins know how to properly accept and deliver email without opening holes in their process. I've tired of explaining the above, so I'll just avoid fqrdns.pcre updates because apparently it's easier to get a rule removed than it is to explain what a generic addr is. |
@jimpop-at-gmail-dot-com I've tried to be polite to you, and listen to all inputs (including yours) before making a decision. You came out of the gates with a rude and dismissive tone, and don't seem to have backed down from it. "Getting out of hand" seems a tad excessive. This is mail administration, not the battle against ISIS. Stan had the "benefit" of being able to make changes to the file without version history, and would have simply pushed out changes as he saw fit. When I volunteered to take this over, I put it here on GitHub so the process would be more open, and therefore I'm voluntarily open to more scrutiny in my stewardship of the list. I'm also tired of you explaining the above.. but only because of your insulting tone, and not your expertise -- which I value, as I do that of any others who are generous enough to share it here. |
@jimpop-at-gmail-dot-com: Additionally, as I try to look past your complaints to the content of your comments, I note that you didn't provide what I asked for, which is real-world examples of situations where removing this line would be detrimental, rather than esoteric and hypothetical scenarios. I've given you a path to get what you want. I don't understand why you're not taking it. And if I do end up removing this line, it won't be based on any negative feelings on my part toward you as a result of your rudeness. That would be petty, and that's not my style. I'm asking for data, so that I can make a data-driven and emotionless decision. |
I would think, likely keeping the list as is, and splitting into 3 new ones, this being a combination of the 3, and the 3 other ones based on how likely false positives are to result, would be a good idea, and let people opt themselves how risky to be. As there really is no right choice to be made. Given the results on my system, I get the following: These are the results based on the last year of the company mailserver, and the last 3 months of my personal server. |
@patrickdk77 First, thank you for taking the time to dig in your log files to produce some data. Second, just to confirm, you saw only one false positive from this rule out of 1452 connections? Any rough idea of how many of those 1,173 would have been snagged by other methods? One false positive out of that many connections really isn't a lot. Very helpful info. Your idea about splitting the list into a Goldilocks-style approach (Baby Bear, Mama Bear, Papa Bear) with varying levels of risk is an interesting one. What criteria would you propose for determining what types of rules belong on which list? What type of performance penalty would occur if Postfix had to evaluate three PCRE files vs. only one? |
No, the 1 false positive was not cause of this rule, but could have It was blocked cause the dns ptr named contained a / char in it. N/A.xxx.xxx.in-addr.arpa Quoting Steve Jenkins notifications@github.com:
|
Ahhh... ok. So there's no evidence in your log files that you blocked any valid mail as a result of this rule being in the file. |
Yes, based on my results, all 1173 where NOT snagged by other rules, as I filtered those out above, 1452 total attempts would match this rule, but only 1173 made it to this rule. The 1 false positive, would have been snagged by this rule, BUT it also matched other rules before it got this far. Sorry, just reading this on the website now, the email ticket update I got seemed to truncated the question, and I didn't see all of it. As far as the issues that could happen with multible lists, I see no issues, yes there will be more effect to process 3 vs 1, but not much, just open/read commands. If people do it like I do atleast, I have an update script that collects the rules, sanity checks them, and compiles them into 1 or a few files, to get published to my servers. So the effect for me, would be 0, as I would be preprocessing them. The real slowdown is how many regex lines have to be evaluated, I imaging using IF clauses to exclude large sets of regex lines will cause much more benefit. But I have to double check, if this is valid in these maps, and not specific to header checks. |
There already are a couple of IF statements in the PCRE file, though they're pretty simple. It first checks to see if certain key words (like dynamic) appear, and if they do, it checks against that subset. If not, it checks against the manually entered hosts in the bottom half of the file. |
As an update, I am considering the idea of expanding this project into three separate PCRE files:a primary fqrdns.pcre file (that leans heavily toward avoiding false positives), a fqrdns-plus.pcre file that contains rules that would be deemed "slightly" more aggressive and that have been migrated from the primary file (such as the iana rule under discussion here) that are wanted by some but not by others, and a fqrdns-max.pcre file that contains far more general rules that seek to block the vast majority of dynamic hosts. To tinker with this idea, I've created a "develop" branch of this project and populated it with a first draft of what such files might look like. Again, this is an experiment, but one in which I welcome polite and thoughtful feedback. |
New three-file approach has been running fine for over 18 months, so I've merged it to the master branch. |
Can't see a valid reason for this rule to exist:
/.in-addr.arpa$/ REJECT Generic - Please relay via ISP
The text was updated successfully, but these errors were encountered: