Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outlook/Office365/Microsoft365 and junk mails #2851

Open
patschi opened this issue Aug 11, 2019 · 113 comments
Open

Outlook/Office365/Microsoft365 and junk mails #2851

patschi opened this issue Aug 11, 2019 · 113 comments
Labels
investigating Still under investigation

Comments

@patschi
Copy link
Member

patschi commented Aug 11, 2019

This issue is more to document and make people aware experiencing similar issues. Discussion, experiences or any tips to come to a solution might be helpful for everyone.


Office 365 / Outlook is quite special when it comes to get mails from your own mailserver delivered to said providers. In a negative aspect, unfortunately.

The problem
There are many reports from users having issues to get serious legit mails delivered to Microsofts' mailing service correctly, even with state-of-the-art non-blacklisted mailservers using latest techniques like DKIM, ARC and strict SPF and being part of their JMRP and SNDS program. In most cases delivered mails are always moved into Junk/Spam folder for absolutely no reason.

Important to notice: This is not limited to mailcow instances overall and is an ongoing issue since a quite long time.

If you have customers at Office 365 or even worse: Outlook.com you should tell them about this issue and migrate them to another service, as they will not be able to receive legit mail from clean mail servers. Business critical mail may never reach their mailbox. This is not the senders problem, this is a serious problem for the recipient and therefore the Microsoft customer.
Microsoft seems not to be able to handle their spam filters and tries to mitigate this problem by blocking whole foreign networks.

Solution
Unfortunately there is no solution available yet. Several users (including me) tried to get more information and support from Microsoft, but without any noticable improvement nor helpful reply. Apparently Microsoft has no interests at all that their users and companies - relying on Office 365 - gets legit mails of any relevance delivered.

Even analysing all headers on the Microsofts' end after delivering just gives you cryptic headers without any sort of explanation why their considered mails as spam. There are several docs around explaining a few details, but so far they are all either outdated or useless.

Following GitHub issues at Microsofts' docs repository are still pending since a longer period of time to hopefully get some more information:

  1. Missing X-Forefront-Antispam-Report fields MicrosoftDocs/OfficeDocs-o365seccomp#743
  2. Include documentation on X-Microsoft-Antispam-Mailbox-Delivery header MicrosoftDocs/OfficeDocs-o365seccomp#442
  3. WIP - What or How are the RuleID's defined? MicrosoftDocs/OfficeDocs-o365seccomp#409

What can you do?
Basically nothing. This might be a workaround.

However you are greatly welcomed to push mentioned GitHub threads above to make Microsoft more aware about this serious issues on their end. If you have more direct connections, use them.

This is going to be continuously updated...

@patschi patschi added the investigating Still under investigation label Aug 11, 2019
@MAGICCC MAGICCC pinned this issue Aug 11, 2019
@Adorfer
Copy link

Adorfer commented Aug 12, 2019

Been there, done that, since 10+ years...
If your are stuck in this limbo at M$ (office365/live.com etc) for no obvious rasons:
get new IP/IPv4 addresses for your mailserver.

My take on this is:

  • If you have a users forwarding from their account at your server to their (other) M$ accounts,
  • and they are subscribed to legitimate mailinglists
  • then they receive via your server spam through this list (and subsequently via your mailserver, forwarded to M$):
  • If those users hit the "report spam" button on their webmail/outlook more than 2-3 times: you will be in deep trouble with your mailserver.

Invite users never to flag spam in their M$-boxes which had been forwarded through your server, especially if it's coming from 3rd party mailingslists: It helps nobody and causes a lot of frustration (especially for your users at M$ services)

@patschi
Copy link
Member Author

patschi commented Aug 12, 2019

This is indeed frustrating, and so far the only provider causing that amount of issues.

In my case, this happened during an ongoing mail conversation. In my opinion that is ridiculous and shouldn't happen at all. I was communicating with my new employer - which are hosting their mail-stuff on Office 365: We wrote about 20 serious, legit mails in both ways.

At some point their mail systems suddenly dropped my mails in the junk folders for absolutely no reason. I mean, I was writing with many different mail addresses on their end, which were replying as well. I don't get it, why their systems believe that this must be spam or something harmful. During an ongoing mail conversation. Beside the fact that mails were (detectable) replies, and not entirely new mails.

You might imagine how much confusion this has caused in the end... This can't be in the sense of Microsofts' customers.

@JustAB0x
Copy link

JustAB0x commented Aug 13, 2019

My current employer uses Office365 and we have issues like this all the time with smaller companies we work with. The craziest thing is: if i try to see why the email gets flagged (mail flow report via the secruity and compliance center), i just get "Flagged as Spam: No further information available".

I myself don't have this problem. I use their SNDS which may help out a bit.

@patschi
Copy link
Member Author

patschi commented Aug 13, 2019

I use their SNDS which may help out a bit.

Did it? Did you ever received a notification from their JMRP/SNDS? I've been in their programs since basically ever, and never got any message. I'm already wondering if this works at all :D

@marrco
Copy link
Contributor

marrco commented Aug 13, 2019

Did you ever received a notification from their JMRP/SNDS?

yes to my SNDS registered abuse@mydomain, also a few weeks ago from staff (at) hotmail dot com w/ subJ "complaint about message from xx.xx.xx.xx" without text and containing as an attachment the original message that a Hotmail user manually marked as spam.

No links for followup nor to mark the issue as resolved,

@Adorfer
Copy link

Adorfer commented Aug 13, 2019

an attachment the original message that a Hotmail user manually marked as spam.

this is what i stated above:
Some M$-Endcustmer (hotmail.com, live.com) flags messages coming through your system as spam:-> YOUR SERVERs IP IS GOING TO THE GULAG, NO APPEALS POSSIBLE.

@JustAB0x
Copy link

JustAB0x commented Aug 13, 2019

I use their SNDS which may help out a bit.

Did it? Did you ever received a notification from their JMRP/SNDS? I've been in their programs since basically ever, and never got any message. I'm already wondering if this works at all :D

"may help " i'm not sure.
You do get insights into your IP status etc. So you at least now if they block you because of your IP or the messages specifically.

@SomeGeek
Copy link

SomeGeek commented Aug 17, 2019

My personal experience is that this problem is mainly on the free tier: outlook.com, hotmail.com, etc. Had this kind of problem a while ago, my mails where not allowed at all (blacklisted, for no reason, new server/ip). However, mails to the Office 365 paid subscription (someone with a own domain) received my mails just fine.

After multiple emails asking to at least say why I was blocked, they decided to 'mitigate' my ip for no reason. I never knew exactly it was blocked.

My take on this (and this might be a unpopular opinion, but regardless) is...

Punishing people who run their own server.

By making this thing a PITA, they can subtle convince people that it's not a good idea to run their own server at all (which I totally don't agree with, the more decentralized, the better), and switch to their service instead. Off course, there are other mail companies with a large customer base (and they can't afford to block them), but hey, every customer bringing their mails to them (including the data) is one right?

I don't see another good reason why they would do this.

@patschi
Copy link
Member Author

patschi commented Aug 22, 2019

There might be a possible workaround for some people: Routing all mails to *.outlook.com over a different mailserver, e.g. AWS SES, Sendgrid, Mailjet, etc. Somehow sad that this is the only way of getting mails over to Microsoft in a reliable way.

For some unimaginable reason (sarcasm!) there's even a Stackoverflow question with a good reply how to configure this sort of workaround using postfix's check_recipient_mx_access: https://serverfault.com/questions/663418/relay-host-based-on-destination-mx-record/663611#663611 (This can't be configured through mailcow UI, must be done manually)

@mkuron
Copy link
Member

mkuron commented Aug 22, 2019

@patschi, I‘ve been using transport maps to deliver to {outlook,hotmail,live}.{com,de} via sendgrid and that works nicely. Sendgrid has a free tier if you send less than 100 messages per day.

Perhaps we could implement check_recipient_mx_access in Mailcow UI similar to how we deal with transport maps and add the workaround to the documentation. It‘s disappointing that we need to resort to solutions like this, but Microsoft seems to even blacklist "good" IP addresses if they don’t deliver a certain minimum number of messages per month.

@patschi
Copy link
Member Author

patschi commented Aug 22, 2019

I‘ve been using transport maps to deliver [...]

Yes, that's possible too, however this doesn't work on custom domains or Office365-hosted domains, therefore I personally like the check_recipient_mx_access-way a bit more.

Perhaps we could implement check_recipient_mx_access in Mailcow UI similar to how we deal with transport maps

Yes, I also think that this is a good idea to offer users at least a workaround for this mess. I've already told @andryyy about the idea - it's up to him to decide if and how exactly this should be integrated like.

@andryyy
Copy link
Member

andryyy commented Aug 22, 2019

How is authentication handled? How does it interfere with transport maps?
It is not as easy as setting up a second map.

@andryyy
Copy link
Member

andryyy commented Aug 22, 2019

Something like "asd.com FILTER smtp:bla" can also lead to funny behaviors when you bcc an address. It will probably end up duplicated.

@SomeGeek
Copy link

SomeGeek commented Aug 22, 2019

It will be hard to prove that they're doing this on purpose, but if we can, that would mean they're abusing their market power, right? That could lead to some interesting legal consequences...

At least that's what I was thinking...

@mritzmann mritzmann unpinned this issue Aug 29, 2019
@mritzmann mritzmann pinned this issue Aug 29, 2019
@patschi
Copy link
Member Author

patschi commented Sep 4, 2019

So apparently Microsoft decides to ignore all the feedback, to kill the conversation and uses an comment* as an excuse to completely lock down the issue: MicrosoftDocs/OfficeDocs-o365seccomp#409 (comment). That's just so ridiculous. Seriously.

I'm wondering where's their respect (according their Code of Conduct) for all affected users which are lucky enough not to host their mailservers at Microsoft?

Conversation continued here... Wondering how long this issue stays open.

*To be fair, the comment was a little bit harsh, but neither wrong nor a reason to lock the conversation IMHO.

@Adorfer
Copy link

Adorfer commented Sep 4, 2019

To conclude: No basic change in microsofts policy to silently block mailservers since 2005:
not reacting to tickets to unblock, not caring about servers not listed on any rbl, with valid spl, dkim etc, send sending 99% of the mail volume from organic users.

If a client does not like to selfhost mail, then they should at least move over to google (they host 3rd party domains too) and provide good spam filtering.

@aixtreme84
Copy link

aixtreme84 commented Sep 9, 2019

There might be a possible workaround for some people: Routing all mails to *.outlook.com over a different mailserver, e.g. AWS SES, Sendgrid, Mailjet, etc. Somehow sad that this is the only way of getting mails over to Microsoft in a reliable way.

For some unimaginable reason (sarcasm!) there's even a Stackoverflow question with a good reply how to configure this sort of workaround using postfix's check_recipient_mx_access: https://serverfault.com/questions/663418/relay-host-based-on-destination-mx-record/663611#663611 (This can't be configured through mailcow UI, must be done manually)

I want to use that workaround. is the following way the right way ?

So i have to add check_recipient_mx_access hash:/opt/postfix/conf/finickydestination at

/opt/mailcow-dockerized/data/conf/postfix/main.cf

I have to create a file /opt/mailcow-dockerized/data/conf/postfix/finickydestination with this line
.outlook.com smtp:[some_smtp.example.com]

  • How to postmap the file ?
  • Am i right with the pathes ?
  • Which services do i have to restart ?

It would be great if someone could give me a short "how to" for that workaround

@andryyy
Copy link
Member

andryyy commented Sep 9, 2019

It will not work. You need to setup default_transport to route through a local smtpd like default_transpor t = [127.0.0.1]:2255. In this new smtpd, you can use check_mx_bla and content_filter through smtp: back into the mailcow system.

A lot of functions will break in mailcow.

@SomeGeek
Copy link

SomeGeek commented Sep 10, 2019

And all of a sudden, they blocked my server again. In the middle of a legitimate mail conversation, the mails started to bounce back:

Please contact your Internet service provider since part of their network is on our block list (S3150).

But, why?!? Here we go, again...

Edit: Off course.

Our investigation has determined that the above IP(s) do not qualify for mitigation.

@andryyy: What functions will break, exactly? Can those be worked around? This is a big issue that will render a mail server useless, if you take into account how much people are using Microsoft's email service.

@evilstiefel evilstiefel unpinned this issue Sep 18, 2019
@Ry3nlNaToR
Copy link
Contributor

Ry3nlNaToR commented Sep 20, 2019

I've been using check_recipient_mx_access on my Mailcow server for a long time to route Microsoft bound mail via a relay I haven't seen any major issue, am not 100% sure what functions might get broken using that method.

The functions I think that might get messed up or broken would be things like Sender-dependent transports, BCC/Recipient/Transport Maps,Outgoing TLS policy map overrides which I don't use any of them anyway.

@mgnisia
Copy link

mgnisia commented Sep 22, 2019

I would be also interested in a small how to in order to fix this problem! :) @Ry3nlNaToR could you maybe help us?

Edit update: Solved my issue via the a transport map in the routing menu.

@andryyy
Copy link
Member

andryyy commented Sep 23, 2019

No @SomeGeek, it does not render your server useless. If your IP reputation sucks, use Amazon SES or any other relay service for Outlook or complain at Outlook.

You can implement it and test all kind of routing in mailcow afterwards. A lot of special-case routing will break. If it does not, create a clean and tested PR. I will merge it.

This does not mean, that you are unable to relay to Outlook. It is not a bug in mailcow. It is just that Outlook chose the easy path. You can ask people on Outlook.com how they feel about having a bunch of valid mail in junk, because Outlook is unable to filter inbound and forces its senders to scan mail for them. Tell them to whitelist you and tell them why. Ask them how they feel about checking their junk each day. They probably use it like their inbox now.

@Ry3nlNaToR you are correct, these are the functions, that will break. If you don't use them, it is fine, I guess. But you probably understand, I cannot implement it like this.

Github is NOT a place to get support for mail delivery. Buy a relay service, if you cannot reach the amount of mail per day, to be trusted by Outlook. Or just keep mailing. Tell your rcpts why this is happening. They will probably miss a lot of mail from small businesses.

@gromain
Copy link

gromain commented Oct 16, 2019

Just for reference, this is happening to me too, on my own properly validated domain. It used to work at the beginning, but doesn't anymore for some reason.

It looks like Microsoft is getting busy not properly managing spam and yet still blocking legitimate emails from SBEs...

Anyway, I'll implement the workaround today with sendgrid. Thanks for the tip!

@gromain
Copy link

gromain commented Oct 16, 2019

Hello,

I've tried the proposed fix but couldn't get it to activate using check_recipient_mx_access as @Ry3nlNaToR . I've had to use the transport map and do it manually (and it's not working for some unknown reason).

I'm not sure postfix can use both the sql database and the file-based system. Or at least, I couldn't configure it properly.

For the sake of it, here are my steps, from the folder /opt/mailcow-dockerized:

  • addition of check_recipient_mx_access hash:/opt/postfix/conf/outlook as first directive in smtpd_sender_restrictions = to data/conf/postfix/main.cf (line 127).
  • addition of file data/conf/postfix/outlook with content .outlook.com FILTER smtp:smtp.sendgrid.net.
  • addition of a Sender-dependent transport for smtp.sendgrid.net with username apikey and password $APIKEY
  • postfix container restart: docker-compose restart postfix-mailcow

To test the filter, I put a BCC gromain@domain.org to data/conf/postfix/outlook in place of the FILTER part, but I saw the filter does not get activated and the smtp server never gets used.
I believe the username and password would never get used anyway (since the database query on a host being defined).

I believe (but haven't tested yet due to lack of time) that if the check_recipient_mx_access is added to master.cf it will not break existing filters: https://marc.info/?l=postfix-users&m=121926229929879&w=2 .

@andryyy
Copy link
Member

andryyy commented Oct 16, 2019

check_recipient_mx_access does break a lot of things. I tried it for a day and gave up, as there were too many configurations that needed to be changed - including routings in sql tables etc.

It works, if you don't mind breaking some other things we allow users to manage in mailcow UI. If you don't mind, you can implement it in your cow. :) Otherwise, talk to Outlook users and tell them, why they are missing important mail.

@EricThi
Copy link

EricThi commented Dec 30, 2020

@FingerlessGlov3s In first time, i have validate on my test account, when send mail directly on inbox, i have create new email and test it : Receive in Spam, search and found : miss send "many mail" all day => cron
Last test : send to email already existing and check on live : Inbox directly

After, One of domains on my mailcow is used (by a teacher) for send mail to people (parents) with many providers emails : outlook/live/gmail/free(fr ISP) => no spam

After, i have change today some params on my dns zone (clean) and reduce spf :
"v=spf1 mx a -all"
Why ? because on this website :
https://www.dmarcanalyzer.com/fr/spf-4/checker/
=> to many include, spf can be skip by isp and include google and cie is useless...

After, i validate my spf via :
https://www.kitterman.com/spf/validate.html?

send mail to check spf/dkim :
http://www.open-spf.org/Tools/

and found best practice for send mail to google and office with spf and dkim :
https://emailtrends.com/news/2020-dmarc-works-autumn-update/

I edit my previous post

@FingerlessGlov3s
Copy link

FingerlessGlov3s commented Dec 30, 2020

@EricThi, good stuff there, people recommend you use put the IPs directly in the SPF rather then using "a" or "mx". As some mail servers fail to do the mx and a lookups sometimes (less DNS lookups the better). Also MS don't use IPv6 nor TLS1.3, which I find little odd.
Example SPF: "v=spf1 ip4:51.51.51.51 ip6:2001:41d0:800::1 -all"

My emails going to spam stopped after Microsoft removed that "block" (mitigation), clicking not spam personally didn't fix anything :-(, until Microsoft did this thing.

That email trends was getting different results too me, before I contacted MS, so I wonder if my "block" was my issue. Lets hope they don't "block" me again. Otherwise I'll have to email them again.

@EricThi
Copy link

EricThi commented Dec 31, 2020

@FingerlessGlov3s Set a or mx can be used for farm server mail for me. yes, for one server mailcow (or another) it's oversized :)
IPV6 and tls1.3 it's not a standard at this time.
On my side, i remove ipv6 on all hosts and my frontal FW block ipv6.
I use tls1.3 by default with failback to tls1.2

On my side, i have request to microsoft to remove my ip too and unblock my domain, because I use a low cost ndd (.ovh personnal use, low renew) => I never thought that it's by "default" blacklisted...

At this time, if send an email to *@hotmail.com or *@outlook.com => Inbox.

After, I send "many" mail all day to Microsoft and Google for create traffic :)

@FingerlessGlov3s
Copy link

FingerlessGlov3s commented Dec 31, 2020

I use OVH services myself, but not looked in to getting their TLD yet though... Its super cheap though, compared to other TLDs. Should have better rep than a .tk tld atleast you'd think so. Which I have but unfortunately I used it free since 2011/2012 or something, so I can just moved off, but I just purchased it in 2014 and purchased it till 2028. It's one of the most popular tlds but not the most popular for spam some research suggests though. Now trying to move the registrar of the .tk domain is proving very hard, might take months to move it from freenom.

I wonder if MS block unseen IPs and domains (unless the IP is good reputation eg gsuite) by default to try help combat spam... Only thing I can think of but I still know people who get lots of spam enter their inbox on Outlook.

Have you also enabled DMARC reporting? So your mail servers send reporting emails for other domains?

@apiraino
Copy link

apiraino commented Jan 17, 2021

1. Bonus :

After mail are good (not send "spam" before contact delivery support xD)
for help microsoft/google to get data, i have create 4 little crontab on my server to send mail via my mailcow to gmail and outlook address

and

After, I send "many" mail all day to Microsoft and Google for create traffic :)

@EricThi thank you for describing all the steps you took to whitelist your domain. I have a question about the above method to "feed" good emails to keep the reputation good. Do you have proof that it works? Any way to verify that?

To be clear, I have the same problem described in this issue (new mailcow installation, SPF1, DKIM, DMARC all pass). I'm not sending mail to a recipient on MS cloud service, rather Gmail. But the effect is the same: opaque heuristics to classify as spam, impossible to debug the issue, unhelpful answers from google.

@EricThi
Copy link

EricThi commented Jan 21, 2021

I use OVH services myself, but not looked in to getting their TLD yet though... Its super cheap though, compared to other TLDs. Should have better rep than a .tk tld atleast you'd think so. Which I have but unfortunately I used it free since 2011/2012 or something, so I can just moved off, but I just purchased it in 2014 and purchased it till 2028. It's one of the most popular tlds but not the most popular for spam some research suggests though. Now trying to move the registrar of the .tk domain is proving very hard, might take months to move it from freenom.

I wonder if MS block unseen IPs and domains (unless the IP is good reputation eg gsuite) by default to try help combat spam... Only thing I can think of but I still know people who get lots of spam enter their inbox on Outlook.

Have you also enabled DMARC reporting? So your mail servers send reporting emails for other domains?

Hello @FingerlessGlov3s for tld .ovh, it's a response by 4/5 websites to switch my old email to my new mail with .ovh
=> per default .ovh is blacklisted for create new account (or change email) because it's used by "many" spam.
==> blacklisted by website or by a federate security for webshop/anothers (example for me : my mobile operator...)

My ip is neutral on reputation (it's not poor or bad, just neutral because i send low volume)
after, i don't have email reputation because no newsletter, just send direct email.

I have all report enable for dmarc to send on my main domain via :
domaintoreport.tld._report._dmarc.maindomain.tld. TXT "v=DMARC1"

After, all report are 100% good, because my email are not reject/detected to spam, just mark as spam by a post rules on gafam...

Since yesterday, my email were tagged in spam to google.. I search the reason and for me it's linked by a bad RBL /:

  • Blocked by dnsbl-3.uceprotect.net: I have 3 ip set for check "good" blacklist
    => one is my webserver and cannot send mail
    ==> one is my dns server and cannot send mail
    ===> one is my mail server
    ====> after check on this very bad rbl, i'm blocked to level3 => AS => OVH.

=> Big impact on my "reputation" for google, tagged on spam :/
==> same to outlook now... (for previous mail already send many email with scripts)

==> I have contact google and Microsoft for that.
After, for microsoft on supervision :
https://sendersupport.olc.protection.outlook.com/snds/data.aspx?wa=wsignin1.0
==> all is green, no spam send <<

I have add 2 domain on postmaster google for test it :
https://postmaster.google.com/
==> waiting to send more 100 mails per day for get statistics
===> i use my old account, because i need to remove it for leave gafam :D

I waiting response by microsoft and google support now.

1. Bonus :

After mail are good (not send "spam" before contact delivery support xD)
for help microsoft/google to get data, i have create 4 little crontab on my server to send mail via my mailcow to gmail and outlook address

and

After, I send "many" mail all day to Microsoft and Google for create traffic :)

@EricThi thank you for describing all the steps you took to whitelist your domain. I have a question about the above method to "feed" good emails to keep the reputation good. Do you have proof that it works? Any way to verify that?

To be clear, I have the same problem described in this issue (new mailcow installation, SPF1, DKIM, DMARC all pass). I'm not sending mail to a recipient on MS cloud service, rather Gmail. But the effect is the same: opaque heuristics to classify as spam, impossible to debug the issue, unhelpful answers from google.

@apiraino : yes, depend to robot/human on request support :/
https://toolbox.googleapps.com/apps/checkmx/ it's good for you .? At this time, my spf are "bad" for google <<

error
SPF must allow Google servers to send mail on behalf of your domain.
help_outline
Help center article => https://support.google.com/a/answer/33786
Decision SPF fail - not authorized
Record v=spf1 mx a ip4:my.ip.mail.server -all

and after follow many docs (very bad docs for me) :
https://support.google.com/a/answer/33786#zippy=%2Crechercher-votre-fournisseur-de-domaine%2Cfournisseurs-de-messagerie-tiers%2Cenregistrement-txt-pour-spf%2Cfacultatif-v%C3%A9rifier-votre-enregistrement-txt-pour-spf-actuel

Good configuration need to return on spf :

_spf.google.com
_netblocks.google.com suivi de plusieurs adresses IP
_netblocks2.google.com suivi de plusieurs adresses IP
_netblocks3.google.com suivi de plusieurs adresses IP

After change my spf : "v=spf1 mx a ip4:my.ip.mail.server include:_spf.google.com include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com -all"

=> checkmx see only warning because :

warning
No Google mail exchangers found. Relayhost configuration?
help_outline
Help center article => https://support.google.com/a/answer/33915
If you intentionally set up a mail server somewhere on your premises that automatically forwards all incoming mail to Google you may disregard this warning. Otherwise - this is a serious configuration error as it causes disruption of mail flow.

ok, it's for send directly via google ^^ ok, big joke by google...

After, i will not trust google on my spf... and for "good" practice, now, it's needed...
When explain on my first comment, for google, if we have "good" rules (for google) it's ok... for MS, it's very different :/

Have fun

@EricThi
Copy link

EricThi commented Jan 21, 2021

I found this another check found on google support : https://www.checktls.com/TestReceiver
(src : https://support.google.com/mail/thread/3973530?hl=en)

My result :

CheckTLS Confidence Factor for "mail@mydomain.tld": 100
all are ok at 100%
[005.141] We can use this server
[005.141] TLS is an option on this server
[005.141] ‑‑> STARTTLS
[005.230] <‑‑ 220 2.0.0 Ready to start TLS
[005.230] STARTTLS command works on this server
[005.331] Connection converted to SSL
SSLVersion in use: TLSv1_3
Cipher in use: TLS_AES_256_GCM_SHA384
Perfect Forward Secrecy: yes
Certificate #1 of 3 (sent by MX):
Cert VALIDATED: ok

I have too send request to google via answer no to all questions (and it's true) :
https://support.google.com/mail/troubleshooter/2696779?visit_id=637468153468059253-3193426294&hl=en&rd=3#AppsHC

for get this url :
https://support.google.com/mail/contact/bulk_send_new

@apiraino
Copy link

apiraino commented Jan 21, 2021

I found this another check found on google support : https://www.checktls.com/TestReceiver

Also tested that one, all is green for my mail server (as expected).

https://support.google.com/mail/contact/bulk_send_new

I have submitted a request to this form a couple of days ago. They should provide an answer in two weeks time (what a joke)

In my case, I don't use google in any form to send emails so all their tooling to check my "reputation" are mostly useless for me. I just need them to not greylist me for no apparent reason.

I'm sorry to read that because of OVH you've been blacklisted :-(

Anyway, thanks for your suggestions 👍

@Clete2
Copy link

Clete2 commented Mar 2, 2021

@gromain did you try the routing option in the admin menu with sendgrid/mailgun/AWS SES & etc. of mailcow?

Mailcow

This works for me as a bandaid. I don't like using it, but I see no other choice.

I tried the fix that @EricThi laid out. I swapped to a new IP address, but unfortunately in the middle of implementing @EricThi 's scripts, I got banned on the NEW IP address from Microsoft. My emails were handwritten between my Mailcow installation and my personal MSN account.

Like others, I had no luck with Microsoft support. Not eligible for mitigation, and they won't discuss it with me.

Edit: I sent an absolutely scathing email to Hotmail support in response to their "sorry but we aren't doing anything" response accusing them of anticompetitive behavior and that I am telling all my clients to shutdown their Microsoft accounts. They responded by reopening my case, investigating, and unblocking me. Please let this whitelist stay.

Edit2: I implemented something similar to @EricThi, but used Enron's e-mails instead. I detailed my approach here.

@mfld-pub
Copy link

mfld-pub commented May 12, 2021

1. Bonus :

After mail are good (not send "spam" before contact delivery support xD)
for help microsoft/google to get data, i have create 4 little crontab on my server to send mail via my mailcow to gmail and outlook address :

@hourly cp /usr/local/smtp/template-joke_jod /tmp/joke_jod && curl -X GET "https://api.jokes.one/jod" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_jod && cat /tmp/joke_jod | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_jod

@hourly cp /usr/local/smtp/template-joke_blonde /tmp/joke_blonde && curl -X GET "https://api.jokes.one/jod?category=blonde" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_blonde && cat /tmp/joke_blonde | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_blonde

@hourly cp /usr/local/smtp/template-joke_animal /tmp/joke_animal && curl -X GET "https://api.jokes.one/jod?category=animal" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_animal && cat /tmp/joke_animal | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_animal 

#Baconipsum
*/5 * * * * cp /usr/local/smtp/template-baconipsum /tmp/baconipsum && wget "https://baconipsum.com/api/?type=meat-and-filler&paras=5&format=text" -O /tmp/baconipsumtmp && cat /tmp/baconipsumtmp >> /tmp/baconipsum && cat /tmp/baconipsum | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/baconipsum*

This is an awesome idea to help boost sender reputation. I was curious why Google and MS postmaster tools never showed anything for my domains, even they are 2 years in prod. Turns out 250 mails a day is not enough. So about 2 weeks ago I implemented your suggestion but as of now postmaster tools still shows no data. Does this technique still work ?

I I send to 3 different gmail and 3 different microsoft accounts.

Google DMARC report for a typical day:

Ipv4
293
IPv6
307

Could it be that because I am dual stacked and each IP is treated separately the postmaster tools won't show any data for me because my ~600 a day are about 50/50 split IPv4 and IPv6 ?

Edit: I see in @Clete2 writeup from the comment above

I have 288 randomly selected e-mails being sent to Microsoft servers each day that are actively read by a user.

In my case I just have a filter rule to move them to a folder and mark as read. Are they tracking that and disregarding the messages ? I log into my receiving accounts every now and then to reply to bacon ipsum mails to show "interaction" of sorts.

@github-actions
Copy link

github-actions bot commented Jul 12, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale Please update the issue with current status, unclear if it's still open/needed. label Jul 12, 2021
@patschi patschi removed the stale Please update the issue with current status, unclear if it's still open/needed. label Jul 19, 2021
@patschi patschi reopened this Jul 19, 2021
@EricThi
Copy link

EricThi commented Aug 20, 2021

On my side,
scripts work (warning to don't raise limit mail per seconde/minute on gafam) and i have adapt it for send to same mailbox, on different virtual box (for the fun)

*/10 * * * * cp /usr/local/smtp/templates/joke_jod /usr/local/smtp/tmp/joke_jod && curl -X GET "https://api.jokes.one/jod" -H "accept: application/json" -H "content-type: application/json" -H "X-JokesOne-Api-
Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\r/ /g' | sed 's/\n/ /g' >> /usr/local/smtp/tmp/joke_jod && cat /usr/local/smtp/tmp/joke_jod | msmtp -a joke mail1+${RANDOM:0:9}@gmail.com, spa
mdebug2791+${RANDOM:0:9}@outlook.com, mail3+${RANDOM:0:9}@hotmail.com && sleep 2 && rm /usr/local/smtp/tmp/joke_jod
*/15 * * * * cp /usr/local/smtp/templates/joke_blonde /usr/local/smtp/tmp/joke_blonde && curl -X GET "https://api.jokes.one/jod?category=blonde" -H "accept: application/json" -H "content-type: application/json
" -H "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\r/ /g' | sed 's/\n/ /g' >> /usr/local/smtp/tmp/joke_blonde && cat /usr/local/smtp/tmp/joke_blonde | msmtp -a joke mail1+
${RANDOM:0:9}@gmail.com, mail2+${RANDOM:0:9}@outlook.com, mail3+${RANDOM:0:9}@hotmail.com && sleep 2 && rm /usr/local/smtp/tmp/joke_blonde
*/20 * * * * cp /usr/local/smtp/templates/joke_animal /usr/local/smtp/tmp/joke_animal && curl -X GET "https://api.jokes.one/jod?category=animal" -H "accept: application/json" -H "content-type: application/json
" -H "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\r/ /g' | sed 's/\n/ /g' >> /usr/local/smtp/tmp/joke_animal && cat /usr/local/smtp/tmp/joke_animal | msmtp -a joke mail1+
${RANDOM:0:9}@gmail.com, mail2+${RANDOM:0:9}@outlook.com, mail3+${RANDOM:0:9}@hotmail.com && sleep 2 && rm /usr/local/smtp/tmp/joke_animal

after, i have add a "clean" notification and for that, i use my nextcloud with 3 fake account (with mail 1/mail2/mail3) :
1-15/2 9,16 * * 1-5 sh /usr/local/scripts/reunion-g-r.sh
1-30/2 6,18 * * 0,6 sh /usr/local/scripts/reunion-g-o.sh

16-30/2 9,16 * * 1-5 sh /usr/local/scripts/reunion-h-r.sh
31-59/2 6,18 * * 0,6 sh /usr/local/scripts/reunion-h-o.sh

31-59/2 9,16 * * 1-5 sh /usr/local/scripts/reunion-o-r.sh
1-20/2 7,19 * * 0,6 sh /usr/local/scripts/reunion-o-o.sh

example, create a file via webdav with notification :
curl -u login:pwd -T /usr/local/scripts/reunion https://nextcloud_domain/remote.php/dav/files/login/Reunion/$(date '+%d-%b-%Y').md

and on calendar app, i have create many appointment with notification by email only

since mai, i have clean my ip and i have configure dane, mta-sts and all are good now...

After, i have test to send mail between mail2 and mail 3 (@Hotmail & @outlook.com) => mail are tagged spam (no bad mail, just a default mail )

@mfld-pub yes, if your send your mail via many ip, your reputation is divided by the number of ip
on my side, i have disable ipv6 on mailcow : https://mailcow.github.io/mailcow-dockerized-docs/firststeps-disable_ipv6/
and use only one ipv4 dedicated for mailserver

@EricThi
Copy link

EricThi commented Aug 20, 2021

I never think, use a bin mail for send many mail for increase mail per day .

Example :
https://www.mailhazard.com

after create random mail, send many mail (i test it for check if banned with script on cron every minute...)

another services with same idea (sorry, in french website, with services french and english) :
https://www.arobase.org/spam/se-proteger/adresse-jetable.htm

@milkmaker
Copy link
Contributor

milkmaker commented Oct 19, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@milkmaker milkmaker added the stale Please update the issue with current status, unclear if it's still open/needed. label Oct 19, 2021
@unixfox
Copy link
Contributor

unixfox commented Oct 19, 2021

not stale

@milkmaker milkmaker removed the stale Please update the issue with current status, unclear if it's still open/needed. label Oct 19, 2021
@JustinBack
Copy link

JustinBack commented Oct 20, 2021

I know this is an ongoing issue with problems on microsoft's side, however contacting Microsoft solved this issue for me:

Hello,
 
My name is XXXXX and I work with the Outlook.com Deliverability Support Team.
 
We will be looking into this issue along with the Escalations Team regarding IP: (XXXXX). We understand the urgency of this issue and will provide an update as soon as this is available. Rest assured that this ticket is being tracked and we will get back to you as soon as we have more information to offer.
 
Thank you for your patience.
 
Sincerely,
 
XXXXX
Outlook.com Deliverability Support

and not even an hour later the IP has been whitelisted.

Hello,
My name is XXXXX and I work with the Outlook.com Deliverability Support Team.
We have implemented mitigation for your IP (XXXXX) and this process may take 24 - 48 hours to replicate completely throughout our system.
Sincerely,
XXXXX 
Outlook.com Deliverability Support.


So for everybody considering doing a long term mail server, I suggest submitting a ticket. Super fast and E-Mails dont even land in the junk folder.

Still doesn't mean microsoft shouldnt fix it on their end 🤷🏻

@CookieCr2nk
Copy link
Contributor

CookieCr2nk commented Oct 20, 2021

@JustinBack The Hotmail Sender Support told me that they not see any issue, why i can sending mails to Outlook.com Customers, so i argumented "why i'm not a spammer" and after three mails, they escalated my issue to the Microsoft Customer Support. The Microsoft Customer Support replied quickly and after 24h my IP was unblocked from Microsoft.

I had the issue, that i received no bounce, but the mails never reached the inbox. I think the Hotmal Sender Support can't debug such issues with the Smartscreen filter, so they have to escalate it to the Microsoft Customer Support.
After they escalated it to the Microsoft Customer Support the resolution was very quickly and after 24 hours i can now sendign mails to outlook.

grafik

@JustinBack
Copy link

JustinBack commented Oct 20, 2021

Weird, never been contacted by the hotmail sender support, even though I had issues with hotmail.com. All conversations came directly through the microsoft support
grafik

2 E-Mails in total 🤷🏻

Our issue was different than yours then, We received bounces from hotmail so it could be an entirely different issue.

For reference, our issue was the S3150 blocklist:

<*******@hotmail.de>: host
eur.olc.protection.outlook.com[104.47.8.33]
    said: 550 5.7.1 Unfortunately, messages from [**********] weren't sent.
    Please contact your Internet service provider since part of their network
    is on our block list (S3150). You can also refer your provider to
    http://mail.live.com/mail/troubleshooting.aspx#errors.
    [AM5EUR03FT011.eop-EUR03.prod.protection.outlook.com] (in reply to MAIL
    FROM command)

Maybe this helps someone with a similiar error message

@lecocotier
Copy link

lecocotier commented Oct 25, 2021

Hello, I'm facing the same issue with Postfix, and can't get any support from Microsoft. So I tried at least to detect the Smartscreen abuse using delivery status notifications and sieve.

f anyof (
header :contains "Content-Type" "report-type=delivery-status",
header :contains "Content-Type" "disposition-notification"
){
if anyof(
allof (body :contains "X-MS-Exchange-Organization-SmartScreen-Diagnostics", body :contains ["FinalSCL:1","FinalSCL:2","FinalSCL:3","FinalSCL:4","FinalSCL:5","FinalSCL:6","FinalSCL:7","FinalSCL:8","FinalSCL:9"]),
allof (body :contains "X-Forefront-Antispam-Report", not body :contains "SFV:NSPM")
){
if header :matches "Subject" "*" {
set "subject" "${1}";
}
deleteheader "Subject";
addheader :last "Subject" "[WARNING: Microsoft SmartScreen false positive - USE PHONE] ${subject}";
fileinto "Inbox";
stop;
} else {
setflag "\Seen";
fileinto "Microsoft Smartscreen bug detector";
stop;
}
}

Appears to work, but probably not cover all the cases.

@Dustinlheld
Copy link

Dustinlheld commented Oct 29, 2021

DMARC protects users against forged email messages and allows you to manage communications that do not pass SPF or DKIM. DMARC protects your email accounts against spam, spoofing, and phishing.

@Adorfer
Copy link

Adorfer commented Oct 29, 2021

DMARC protects users against forged email messages

so what?

@Franselbaer
Copy link

Franselbaer commented Dec 2, 2021

In the meanthime I've had so much trouble with it that I stopped thinking about it and blocking all smtp traffic on all mailservers I get hands on for as8075. Hopefully some day critical mass is reached and they get forced by their customers to provide a proper working email system. Until then, all my mailbox slots stay closed for M$.

@FingerlessGlov3s
Copy link

FingerlessGlov3s commented Feb 23, 2022

Having to go through Microsoft Support again due to a change of Public IP due to moving from OVH to their SYS (SoYouStart) line but unfortunately can't bring the IP with me.

Thought I would share my experiences again since others may get the same issues and this may help. Before I started my journey with Microsoft support, I made sure rDNS, SPF, DKIM, DMARC is all setup correctly. I also signed up to their SNDS and JMRP. SNDS reports the IP as blocked. I also tried the delist request at sender.office.com but that didn't help either.

So now we need to use the delist form
https://support.microsoft.com/supportrequestform/8ad563e3-288e-2a61-8122-3ba03d6b8d75
So first of all the form didn't work and would error, tried different PC, different internet connection, kept erroring, so I waited 24-48 hours and it started working again, so I filled that in eventually, but very annoying. I've seen others report this issue on Microsoft's forum and they start to blame content filters or VPNs, all of which I wasn't using. I also tried 2 different internet connections, but looks like the only solution is to wait and try again.

First response from them is a "please go away", an auto reply it seems, maybe a canned response. Saying the following "...do not qualify for mitigation." and tell you to check your configuration which I've already checked and doubled checked. Gmail can deliver ok, with no warnings or failures in SPF, DKIM and DMARC when checking the headers Gmail adds.
So I reply to them telling them I've done all those things and ask for more help.

They came back to me few hours later and said I need to provide proof of purchase of the IP from my ISP. So I send them PDF copy of the invoice and PDF of the email that says the IP have been delivered. I think this pretty reasonable if they need to be able to tell the difference between legitimate requests and fake ones.

So I get another email back from Microsoft hour or so later. Telling me I now need written email from SYS to say I own that IP address. Which is annoying because the invoice and email of delivery proves I own it. So does the reverse DNS record. I logged a support ticket with SYS, which they said the only proof they will send me is the invoice with the IP address in it. If I was a service provider like them, I'd probably say the same, because it has my name, address, and IP on the invoice, no need to do it again, in a less official way through a support ticket.

I then reply back to Microsoft and attach their response as an .eml and the invoice again. I go on to say that they will not provide the confirmation via support ticket and that the invoice is the proof of ownership, I also point out that I have DNS control of the IP address as I have the PTR record set to my domain.

They get back to me about 3 hours later, saying they are going to escalate it. From past experience this means they are going to put a mitigation in.
We will be looking into this issue along with the Escalations Team regarding IP: (51.x.x.x).

Another 6 hours later, I get an email telling me the following

My name is Varsha and I work with the Outlook.com Deliverability Support Team.
We have implemented mitigation for your IP: (51.x.x.x) and this process may take 24 - 48 hours to replicate completely throughout our system.

Result! Little bit of emailing and issues with their online form I've finally got the block lifted. I can now deliver to Microsoft provided email again.

@G2G2G2G
Copy link

G2G2G2G commented Apr 2, 2022

I have 10 small businesses I run their emails. All of them go to spam the first msg to gmail, after it's moved to inbox it will never go to spam again.

Microsoft is pretty similar, occasionally I need to use their mail tools to whitelist an ip again or msg their postmaster and cry. but then they'll work again.
Been like this past 20 or so years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
investigating Still under investigation
Projects
None yet
Development

No branches or pull requests