ImportExchange script update Send Connector using Set-SendConnector

[mintylamb]

I've had issues on the last couple of scheduled renewals where outbound email flow stopped from our Hybrid Exchange 2016 server used mainly to manage our Office 365 setup, but also configured as an internal SMTP relay to allowed scoped unauthenticated sending from muli-function printers as described in the Microsoft Support article. We have to use this method due to enforced MFA authentication on Office 365.

First I have checked for Win-ACME updates as I had 2.1.8.847 installed, but didn't see anything in the release notes for newer version to indicate this issue. I have subsequently updated my win-acme to latest 2.1.14.996.

I've started troubleshooting and confirmed that the certificate is correctly generated and imported by the ImportExchange.ps1 script and bound to IIS, SMTP and IMAP as recommended in your manual. I could not see any other -services options for the command that would update the required information.
I finally came across this article https://www.pei.com/exchange-mailflow-send-connector/ which descibes exactly the issue that I'm seeing with the Send Connector configured to allow relaying emails from our internal infrastructure to mailboxes in Office 365 using the certificate for TLS.

Another article indicates that MSExchangeTransport service also needs to to be restarted and similar checks and changes are needed on the default receive connector.

Would it be possible (or even desirable) for win-acme to check the Exchange Send and Receive Connectors matching the FQDN of the certificate and update them, or should this be considered as a separate task for admins to create a scheduled task to update this?
I think the poweshell code in your script should be able to identify any send connectors using
$SendConnector = Get-SendConnector | where {$Certifcate.subject -like 'CN='+$_.fqdn}

if the returned value is not null then it can be updated using code adapted from the article I linked
$TLSCertificate = (‘<I>’+$Certificate.issuer+'<S>’+$Certificate.subject) Set-SendConnector -Identity $SendConnector.Identity -TLSCertificateName $TLSCertificate

I shall see if I can find time to do some modification of a copy of the ImportExchange script and further testing, as all looks to be possible with an amended or additional powershell script.

I'd welcome your thoughts.

[WouterTinus]

Would it be possible (or even desirable) for win-acme to check the Exchange Send and Receive Connectors matching the FQDN of the certificate and update them, or should this be considered as a separate task for admins to create a scheduled task to update this?

It goes beyond my knowledge of Exchange but I think it would make sense for the script to update the certificate whereever it's required, not just the most common services. If you have an Exchange lab available, would you mind also testing this change while your at it? #1372

I think the poweshell code in your script should be able to identify any send connectors using
$SendConnector = Get-SendConnector | where {$Certifcate.subject -like 'CN='+$_.fqdn}

Generally it a lot safer to work with certificate thumbprints than with the subject, because multiple certificates can share the same subject.

[mintylamb]

@WouterTinus thanks for your work on this great tool and taking the time to respond. Sorry I don't have an exchange lab, I've just been reviewing logs in a production environment. If I get one up and running for this I'll try to include looking at the other issue #1372

I will certainly look at adjusting my current setup to delete old certs as mentioned by @nsumner36 in #1372 as this may be worth trying before the powershell commands I've suggested as the SendConnector only references the cert by fqdn and Issuer so deleting expired certs may force it to use the new one and make this request redundant.

Generally it a lot safer to work with certificate thumbprints than with the subject, because multiple certificates can share the same subject

Whilst I agree in principle with your statement about using the thumbrint, we don't have the thumbprint of the old cert and the object returned by Get-SendConnector does not include the thumbprint of the old cert. According the the microsoft documentation for Set-SendConnector the method using -TLSCertificateName I listed above is the correct one to populate the Send-Connector for Exchange.

Extract from the documentation
The TlsCertificateName parameter specifies the X.509 certificate to use for TLS encryption. A valid value for this parameter is "<I>X.500Issuer<S>X.500Subject". The X.500Issuer value is found in the certificate's Issuer field, and the X.500Subject value is found in the certificate's Subject field. You can find these values by running the Get-ExchangeCertificate cmdlet. Or, after you run Get-ExchangeCertificate to find the thumbprint value of the certificate, run the command $TLSCert = Get-ExchangeCertificate -Thumbprint <Thumbprint>, run the command $TLSCertName = "<I>$($TLSCert.Issuer)<S>$($TLSCert.Subject)", and then use the value $TLSCertName for this parameter.

I can also see the possibility that more than one connector may share the same certificate as they may be scoped to send to different destinations, but would want to project the same identity.

[mintylamb]

Felt brave and FYI ran a manual with cleanup old certifcates in production which resulted in the following anonymised extract from the verbose output

Updating Exchange services...
Certificate set for the following services: IIS,SMTP,IMAP
Old Exchange certificates being cleaned up
Error: Remove-ExchangeCertificate : A special Rpc error occurs on server SERVER: These certificates are tagged with
Error: following Send Connectors : Outbound to Office 365. Removing and replacing certificates from Send Connector would
Error: break the mail flow. If you still want to proceed then replace or remove these certificates from Send Connector and
Error: then try this command.
Error: At C:\Program Files\win-acme\Scripts\ImportExchange.ps1:206 char:6
Error: + | Remove-ExchangeCertificate -Confirm:$false
Error: + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error: + CategoryInfo : InvalidArgument: (:) [Remove-ExchangeCertificate], InvalidOperationException
Error: + FullyQualifiedErrorId : [Server=SERVER,RequestId=b0d1eff2-3867-4b24-9036-7d811d610107,TimeStamp=29/01/2021 18
Error: :57:49] [FailureCategory=Cmdlet-InvalidOperationException] 23FA4F48,Microsoft.Exchange.Management.SystemConfigurat
Error: ionTasks.RemoveExchangeCertificate
2021-01-29 18:57:50.022 +00:00 [INF] Script finished
2021-01-29 18:57:50.294 +00:00 [VRB] Waiting for process to finish...
2021-01-29 18:57:50.299 +00:00 [INF] Uninstalling certificate from the certificate store
2021-01-29 18:57:50.306 +00:00 [DBG] Opened certificate store My
2021-01-29 18:57:50.311 +00:00 [INF] Removing certificate [Manual] mail.domain.com @ 2021/1/14 9:00:57 from store My
2021-01-29 18:57:50.317 +00:00 [DBG] Closing certificate store
2021-01-29 18:57:50.336 +00:00 [INF] Scheduled task looks healthy
2021-01-29 18:57:50.350 +00:00 [INF] Next renewal scheduled at 2021/3/25 18:57:10
2021-01-29 18:57:50.400 +00:00 [INF] Certificate [Manual] mail.domain.com created

I can confirm despite the error the old certificate was removed from the store. I'll try to verify that the Send Connector is actually using the new certificate, or still issues the old cert.

[mintylamb]

@WouterTinus took me a little while to track down how to correctly validate which certificate was in use on the send connector as there is no direct link to the thumbprint in the send connector.
I used this old Technet article which gave me

_Yes. If the Subject and Issuer are the EXACT same, then you just need to confirm in the SMTP protocol logs that the inbound and outbound connections are using that new cert as you have seen.

Make sure the chain is good on the new one and then remove the OLD cert via MMC and add the Certificate snap-in and delete it there from the personal store.

I like to then restart the transport service after removing the old cert. After that verify via the protocol logs and queue monitoring that things are moving._

Thankfully I already had verbose logging on for the Send Connector. I searched through the logs under "...\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpSend"
The Win-Acme result email went through this connector immediately after completing the manual certicate creation above and I could see the thumbprint of the new certificate in the logs, so the send connector immediately switches to the new cert because the old one is missing

It appears that the Exchange Notes for Win-Acme should suggest setting .PARAMETER LeaveOldExchangeCerts to 0 where used on a Hybrid Exchange server as these send connectors are part of the Office 365 configuration

--scriptparameters "'{CertThumbprint}' 'IIS,SMTP,IMAP' 0 '{CacheFile}' '{CachePassword}' '{CertFriendlyName}'"

Based on this I think that this request can be closed, unless you wish to consider updating the documentation before doing so.

Learnt a lot going through this - Thanks again for the great tool.

[nsumner36]

I had a second Exchange server running with script parameters of 1, I updated the script, and Win-Acme to the latest version (well the slightly modified script version). I also ran a new renewal request (deleted the old auto-renewal) and set the parameter to 0. It was unable to delete the old certs from Exchange because of the Office 365 error.

Script ./Scripts/ImportExchange.ps1 starting with parameters 'F5622135FB9615C3AEDCBF62090064335BE1DDA0' 'IIS,SMTP,IMAP' 0 'C:\ProgramData\win-acme\acme-v02.api.letsencrypt.org\Certificates\EfxKDEt0lEyHI-RNI30RzA-40c1561e6e600e7b40a4e57f6e431002e3575eec-temp.pfx' 'ZUjt437iHXiAL4yYn1pLqPJMRYmZFQxDnIr6AwHKfHE=' '[Manual] mail.nbn.org.il @ 2021/2/7 9:25:29'
Script error: A special Rpc error occurs on server MAIL16: These certificates are tagged with following Send Connectors :
Script error: Outbound to Office 365. Removing and replacing certificates from Send Connector would break the mail flow. If you
Script error: still want to proceed then replace or remove these certificates from Send Connector and then try this command.
Script error: + CategoryInfo : InvalidArgument: (:) [Remove-ExchangeCertificate], InvalidOperationException
Script error: + FullyQualifiedErrorId : [Server=JERMAIL16,RequestId=d5d543ca-f83b-469c-bc02-5baa0a50414f,TimeStamp=2/7/2021 7:28
Script error: :38 AM] [FailureCategory=Cmdlet-InvalidOperationException] 217FB65D,Microsoft.Exchange.Management.SystemConfigurat
Script error: ionTasks.RemoveExchangeCertificate
Script error: + PSComputerName : mail16.mydomain.lan
Script error: A special Rpc error occurs on server JERMAIL16: These certificates are tagged with following Send Connectors :
Script error: Outbound to Office 365. Removing and replacing certificates from Send Connector would break the mail flow. If you
Script error: still want to proceed then replace or remove these certificates from Send Connector and then try this command.
Script error: + CategoryInfo : InvalidArgument: (:) [Remove-ExchangeCertificate], InvalidOperationException
Script error: + FullyQualifiedErrorId : [Server=JERMAIL16,RequestId=d5d543ca-f83b-469c-bc02-5baa0a50414f,TimeStamp=2/7/2021 7:28
Script error: :38 AM] [FailureCategory=Cmdlet-InvalidOperationException] 217FB65D,Microsoft.Exchange.Management.SystemConfigurat
Script error: ionTasks.RemoveExchangeCertificate
Script error: + PSComputerName : jermail16.nbn.lan
Script finished
Adding Task Scheduler entry with the following settings

• Name win-acme renew (acme-v02.api.letsencrypt.org)
• Path C:\WACS
• Command wacs.exe --renew --baseuri "https://acme-v02.api.letsencrypt.org/"
• Start at 09:00:00
• Random delay 02:00:00
• Time limit 02:00:00
  Next renewal scheduled at 2021/4/3 10:28:28
  Certificate [Manual] mail.nbn.org.il created

I had to go and delete the certs using MMC and then they were happily gone from Exchange as well. So I still suspect we need a slight modification to the script. I will see if I can figure out any solution in the script...

[WouterTinus]

Whilst I agree in principle with your statement about using the thumbrint, we don't have the thumbprint of the old cert and the object returned by Get-SendConnector does not include the thumbprint of the old cert.

win-acme could send the thumbprint for the previously issued certificate to the script if you include {OldCertThumbprint} in the parameter string, but that might not be available or correct in all scenarios (e.g. first time setup, or manual intervention between runs).

I'd be happy to update the example scripts and/or docs once someone gets to the bottom of this. Also free to do a PR, the website is in a repo here: https://github.com/win-acme/win-acme.github.io

[WouterTinus]

Note to self to include this in docs:

It appears that the Exchange Notes for Win-Acme should suggest setting .PARAMETER LeaveOldExchangeCerts to 0 where used on a Hybrid Exchange server as these send connectors are part of the Office 365 configuration

[WouterTinus]

Released in 2.1.20