[O365] Connection fails with "no response"

[attie-argentum]

I've been using this to backup emails from outlook.office365.com for some time (thanks, great project!)

Unfortunately, a while ago, it started failing - looking in to it now, I run the setup subcommand, select the account, and use the "test connection" action... after a second or so it responds as follows:

Choose an action:
[...]
5. test connection
[...]
13. (q) return to main menu
?  5
No response
Press a key

Attempting to run a backup produces the following output:

I, [2022-10-07T16:22:28.218810 #9]  INFO -- : Running backup of account: xxx@xxx.xxx
/usr/local/lib/ruby/gems/3.1.0/gems/net-imap-0.2.3/lib/net/imap.rb:1245:in `get_tagged_response': LOGIN failed. (Net::IMAP::NoResponseError)
        from /usr/local/lib/ruby/gems/3.1.0/gems/net-imap-0.2.3/lib/net/imap.rb:1299:in `block in send_command'
        from /usr/local/lib/ruby/3.1.0/monitor.rb:202:in `synchronize'
        from /usr/local/lib/ruby/3.1.0/monitor.rb:202:in `mon_synchronize'
        from /usr/local/lib/ruby/gems/3.1.0/gems/net-imap-0.2.3/lib/net/imap.rb:1281:in `send_command'
        from /usr/local/lib/ruby/gems/3.1.0/gems/net-imap-0.2.3/lib/net/imap.rb:422:in `login'
        from /usr/local/lib/ruby/3.1.0/forwardable.rb:238:in `login'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/account/connection/client_factory.rb:31:in `block in run'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/retry_on_error.rb:4:in `retry_on_error'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/account/connection/client_factory.rb:19:in `run'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/account/connection.rb:114:in `client'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/account/connection.rb:37:in `run_backup'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/cli/backup.rb:17:in `block in run'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/cli/helpers.rb:91:in `block in each_connection'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/cli/helpers.rb:88:in `each'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/cli/helpers.rb:88:in `each_connection'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/cli/backup.rb:16:in `run'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/cli.rb:58:in `backup'
        from /usr/local/bundle/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /usr/local/bundle/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /usr/local/bundle/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /usr/local/bundle/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
        from /usr/local/bundle/gems/imap-backup-8.0.1/bin/imap-backup:12:in `block in <main>'
        from /usr/local/bundle/gems/imap-backup-8.0.1/lib/imap/backup/logger.rb:34:in `sanitize_stderr'
        from /usr/local/bundle/gems/imap-backup-8.0.1/bin/imap-backup:11:in `<main>'

I have updated to v8.0.1, and I've tried changing / confirming my password with no joy.

Any ideas? Happy to provide more info if it would be helpful!

[joeyates]

Hi @attie-argentum

imap-backup Checks

Please try running setup in verbose mode:

imap-backup setup --verbose

Then, go to your account and do 2 things:

1. Check the value of server, (I think in your case this should be outlook.office365.com),
2. Try 5. test connection again, and see if the debug output helps.

outlook.office365.com Checks

This article indicates a series of possible problems and solutions for outlook.office365.com

Edit: added link

[attie-argentum]

Please try running setup in verbose mode

Good idea - I didn't think for a --verbose flag...

I now get the following, so I think this probably isn't an issue with imap-backup?

Choose an action:
[...]
?  5
D, [2022-10-10T11:01:08.079075 #9] DEBUG -- : Creating IMAP instance: outlook.office365.com, options: {:port=>993, :ssl=>{:ssl_version=>:TLSv1_2}}
D, [2022-10-10T11:01:08.079409 #9] DEBUG -- : Logging in: xxx@xxx.xxx/xxxxxxxxxxxxxxxx
S: * OK The Microsoft Exchange IMAP4 service is ready. [xxx]
C: RUBY0001 LOGIN xxx@xxx.xxx [PASSWORD REDACTED]
[~1s delay here]
S: RUBY0001 NO LOGIN failed.
No response

This articles indicates a series of possible problems and solutions for outlook.office365.com

Did you mean to include a link? I've had a read around, and I suspect this may be out of my control...

[joeyates]

You're right, I didn't include the link. I've edited the comment. The link is https://support.microsoft.com/en-us/office/pop-imap-and-smtp-settings-for-outlook-com-d088b986-291d-42b8-9564-9c414e2aa040

[attie-argentum]

I think I'll close this, as I'm pretty confident it's actually an issue with Microsoft... thanks for the input!

[vielhuber]

Authentication with IMAP is not anymore possible with basic password auth.

Information:

• https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-deprecation-in-exchange-online-september/ba-p/3609437
• https://www.codewrecks.com/post/security/accessing-office-365-imap-with-oauth2/

Are you planning to integrate oauth2 into imap-backup?

Otherwise backuping office 365 / exchange online will not be possible anymore.

[joeyates]

Hi @vielhuber

It's a shame Office 365 is removing basic authentication. Implementing OAuth2 involves a lot of work. imap-backup previously had OAuth2 fro GMail, but I removed it as they are still supporting basic auth.

A PR would be welcome, but it would need to be well tested and documented.

[vielhuber]

You're right, perhaps a tool like https://emailengine.app/ could be an answer to this problem. Just also found the open source repo https://github.com/simonrob/email-oauth2-proxy.

On the other hand if you have successfully generated an access token (instead of a password), the implementation should be easy.

The steps I would see are:

• Complex documentation for the user, so that he can generate client_id, tenant_id, client_secret, object_id, all from the Azure Active Directory; Followed by PowerShell(!) commands for enabling the app for every mail address to backup. This is outside the scope of imap-backup, but needs to be documented well
• Integrating oauth as a switch in the imap-backup interface. You would not provide a password, but client_id, client_secret, tenant_id. This should be not hard
• imap-backup then generates an access token. This is a curl call to a specified route (based on the provider, e.g. for office 365 it is https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token)

I don't have the ability and experience to provide a PR for this. Any chance to support you in another way?

The question is if it is worth it - or outside of imap-backup's scope.

[vielhuber]

I've come to the conclusion that https://github.com/simonrob/email-oauth2-proxy is a very attractive solution to this problem.
I will test this out in future days, how it works together with imap-backup and post an update here.

[vielhuber]

I've successfully started the proxy and am trying now to connect via imap-backup:

config.json

    {
      "username": "xxx@xxx.xxx",
      "password": "xxx",
      "local_path": "xxx",
      "server": "localhost",
      "connection_options": {
        "port": 1993,
        "ssl": {
          "verify_mode": 0
        }
      }
    },

However, I get:

Unexpected error: SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:1993 state=error: wrong version number

I think this is totally unencrypted, is this possible with imap-backup at all to connect to a local proxy?

[vielhuber]

Got it running when setting up an encrypted proxy.
Seems that imap-backup does not support unencrypted connections.