ProtonMail Bridge Support

[bitterdev]

Could you add ProtonMail Brdige Support?
See more at jstedfast/MailKit@5b6032d

[gilleslamiral]

I've read the code you gave access but I couldn't figure out what it does to deal with ProtonMail.

A sync with imapsync ... --debugimap might be a better way to me:

imapsync ... --debugimap 

[bitterdev]

( use --no-modulesversion to turn off printing this Perl modules list )
Info: will resync flags for already transferred messages. Use --noresyncflags to not resync flags.
Host2: probing ssl on port 993 ( use --nosslcheck to avoid this ssl probe ) 
Host2: sslcheck detected open ssl port 993 so turning ssl on (use --nossl2 --notls2 to turn off SSL and TLS wizardry)
SSL debug mode level is --debugssl 1 (can be set from 0 meaning no debug to 4 meaning max debug)
Host2: SSL default mode is like --sslargs2 "SSL_verify_mode=0", meaning for host2 SSL_VERIFY_NONE, ie, do not check the certificate server.
Host2: Use --sslargs2 SSL_verify_mode=1 to have SSL_VERIFY_PEER, ie, check the certificate server of host2
Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1.
Host1: will try to use PLAIN authentication on host1
Host2: will try to use LOGIN authentication on host2
Host1: imap connection timeout is 120 seconds
Host2: imap connection timeout is 120 seconds
Host1: IMAP server [127.0.0.1] port [1143] user [fabianbitter@protonmail.com]
Host2: IMAP server [mx2f80.netcup.net] port [993] user [fabian@bitter.de]
Host1: connecting and login on host1 [127.0.0.1] port [1143] with user [fabianbitter@protonmail.com]
Connecting with IO::Socket::INET6 PeerAddr 127.0.0.1 PeerPort 1143 Proto tcp Timeout 120 Debug 1
Connected to 127.0.0.1 errno(Socket is already connected)
Read: 	* OK [CAPABILITY IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN ID APPENDLIMIT] IMAP4rev1 Service Ready
Host1 IP address: 127.0.0.1
Host1 banner: * OK [CAPABILITY IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN ID APPENDLIMIT] IMAP4rev1 Service Ready
Sending: 1 CAPABILITY
Sent 14 bytes
Read: 	* CAPABILITY IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN ID APPENDLIMIT
Read: 	1 OK CAPABILITY completed
Host1 capability before authentication: IMAP4rev1 STARTTLS AUTH=PLAIN AUTH=LOGIN ID APPENDLIMIT
Host1: 127.0.0.1 says it has CAPABILITY for AUTHENTICATE PLAIN
Sending: 2 AUTHENTICATE PLAIN
Sent 22 bytes
Read: 	+
Sending: [Redact: Count=2 Showcredentials=OFF]
Sent 106 bytes

[bitterdev]

The problem is that ProtonMail doesn't have public IMAP/POP3 interfaces... You need to install the Bridge. This is a local application that is in the middle between the local machine and the mail server and emulates an imap interface. But ProtonMail has some problems with some mail clients. But it would be really useful if ur application would support this because i want to move away from protonmail and want ot transfer all my emails

[gilleslamiral]

Well, I see AUTHENTICATE PLAIN so could you run a default run, no special option, and send the complete output except the credentials.

[bitterdev]

Same result :)

[gilleslamiral]

Ok, I hoped to see a different output using different parameters.
Well, I can't code blindly, I guess the result will stay the same or getting worse...

[RusticXploit]

This may not help bitterdev with his specific problem but I have used imapsync to move mail multiple times both to and from Protonmail. I am moving mail right now using Ubuntu 16.04, imapsync 1.882, and Protonmail bridge 1.1.0.

[gilleslamiral]

Thanks Jason! In other words it looks like it's not a ProtonMail Brigde issue nor an imapsync one. Did you use any special parameters or were they normal imapsync runs?

[bitterdev]

Hey, i have downloaded the beta of ProtonMails native email export client to export my emails. But yes your right. They have a very "special" implementation of IMAP and therefore it isn't working. Some other software vendors are patching there software only to support ProtonMail. The correct decission would be that ProtonMail will improve their Implementation and not otherwise. But thats the way it is.

[gilleslamiral]

It doesn't explain how Jason could successfully use imapsync with ProtonMail Bridge and how you can't.
I've also reread the patch of MailKit. imapsync doesn't use SPECIAL-USE so the patch is not very helpful.
If you give me access to a test account I'll be able to play with ProtonMail. Otherwise it's a blind work that will likely fail.

[bitterdev]

I don't know why it work's for Jason and not for me. Maybe a OS specific bug, maybe other versions of Bridge... Many possibilities. I was using using OS X latest + Bridge latest.
ProtonMail Bridge is only for paid users and i don't have test account. I just have my own account and this one is in use and private. So i can't provide you any credentials. Sorry.
But many imap clients have problems with the bridge solution. E.g. i wanted to use the AirMail client. imap isn't working too. It's definitly a problem the implementation within the Bridge Software.

[bombcheck]

Having this issue, too. imalsync 1.882 and Protonmail-Bride 1.1.3. imapsync do count the mails, but size is 0.
Error during the run (example):

• msg Trash/71 S[0] F[\Seen nonjunk] I[12-Mar-2019 17:14:00 +0100] has RFC822.SIZE null!

This does happen for all mails in the account. Any idea?

[gilleslamiral]

It means that the imap server tells imapsync that the message size is 0. It's not normal so imapsync reports it but it is not relevant for a good transfer since imapsync syncs messages whatever their proclaimed size.

[bombcheck]

But it does not transfer anything in the end.

[gilleslamiral]

So I need the next lines of output

[bombcheck]

Host1 folder [Archive] considering 534 messages
Host2 folder [Archive] considering 0 messages

• msg Archive/1 S[0] F[\Seen nonjunk] I[24-Feb-2018 21:53:05 +0100] has RFC822.SIZE null!
• msg Archive/1 skipped.
• msg Archive/2 S[0] F[\Seen nonjunk] I[24-Feb-2018 21:55:27 +0100] has RFC822.SIZE null!
• msg Archive/2 skipped.
• msg Archive/3 S[0] F[\Seen \Answered nonjunk] I[24-Feb-2018 21:55:47 +0100] has RFC822.SIZE null!
• msg Archive/3 skipped.
• msg Archive/4 S[0] F[\Seen nonjunk] I[25-Feb-2018 07:02:21 +0100] has RFC822.SIZE null!
• msg Archive/4 skipped.
• msg Archive/5 S[0] F[\Seen nonjunk] I[25-Feb-2018 07:29:07 +0100] has RFC822.SIZE null!
• msg Archive/5 skipped.
• msg Archive/6 S[0] F[\Seen \Answered nonjunk] I[25-Feb-2018 19:04:04 +0100] has RFC822.SIZE null!
• msg Archive/6 skipped.

And so on... Let my know if you need more.

[gilleslamiral]

Well, I spoke too fast... imapsync skips a messages when the server says it has a null size.
Search for or (!$h1_size) in the code, remove it and see what happens. It's inside sub message_for_host2

[bombcheck]

This seems to do the trick. Thx!

[gilleslamiral]

Wonderful!
I've also changed the code so that imapsync syncs messages whatever their proclaimed size are.
Fixed in imapsync 1.930 (not published at the time of this writing)

[gilleslamiral]

Fixed.

[exander77]

Hello, I am receiving could not append with empty error.

Communication:

 20 APPEND Folders/Test (\Seen) "09-Apr-2020 12:21:44 +0000" {3032}+ send literal
* 0 EXISTS
* OK [UNSEEN 0] 
* 1 FETCH (FLAGS (\Seen nonjunk) UID 6)
* 1 EXISTS
* OK [UNSEEN 0] 
20 OK [APPENDUID 5 6] APPEND successful

$new_id is 0, maybe wrong regexp in IMAP library?
my $ret = $data =~ m#\s+(\d+)\]# ? $1 : $self;

- msg Test/1 {3032} could not append ( Subject:[Test], Date:["09-Apr-2020 12:21:44 +0000"], Size:[3032], Flags:[\Seen] ) to folder Folders/Test:

Message is actually synced ok, but this error but has to put max errors to high number.

[gilleslamiral]

I don't know, all looks good. The $new_id should be 6 here I guess.
No clue.

[exander77]

I have created a test to replicate it as many times as I want to, will try to print some intermediate steps.

$data:

20 APPEND Folders/Test (\Seen) "09-Apr-2020 12:21:44 +0000" {3032}+ send literal
* 1 EXISTS
* OK [UNSEEN 0] 
* 1 FETCH (FLAGS (\Seen nonjunk) UID 7)
20 OK [APPENDUID 5 7] APPEND successful

my $ret = $data =~ m#\s+(\d+)\]# ? $1 : $self;

$ret:

0

It really seems like that regexp in IMAPClient is bad. As it produces 0.

I am not fluent in these Perl regular expressions, is it possible it takes 0 from [UNSEEN 0]?

[exander77]

Yes, it is that.

That regular is not what I would describe bulletproof, this fixes it:

my $ret = $data =~ m#\s+(\d+)\]\s+APPEND# ? $1 : $self;

What do you suggest to do about that? It is not exactly a bug in imapsync, but it affects its functionality significantly for me.

[exander77]

This can be workaround from within imapsync:

                # This is IMAPClient bug workaround.
                # Should be removed when fixed.
                my $data = join '', $mysync->{imap2}->Results;

                # look for something like return size or self if no size found:
                # <tag> OK [APPENDUID <uid> <size>] APPEND completed
                my $ret = $data =~ m#\s+(\d+)\s*\]\s+APPEND# ? $1 : undef;
                $new_id = $ret;
                # End of workaround.

Basically process the results in imapsync completely yourself (it may be sensible as IMAPClient seems to have some very poor handling of results).

I am also not sure if there should not be a check for defined:

                if ( ! $new_id){

Is zero an error here? IMAPClient is producing undef on error, also if $mysync->{imap2}->LastError is empty it suggests that no error actually occurred and it is a library error.

What are your thoughts?

[gilleslamiral]

My thoughts is that it I won't change easily such an important code to suit only ProtonMail behavior. This is the first time this bug is reported. APPEND messages is main imapsync job.
A quick fix in imapsync could be what you suggest:

if ( ! defined $new_id ) {

The previous worked till now since an UID can't be 0. So you end up with an uid 0 for imapsync. It is not very important unless --useuid or --usecache is used.

The correct fix in Mail::IMAPClient, according to the RFC, should be to match the client tag of APPEND in the response. Anyway, getting the new UID in the response is just sugar from the RFC point of view.
https://tools.ietf.org/html/rfc3501#section-2.2.2

For now, patch yours.

[exander77]

Studying all of the RFCs, I think that ProtonMail is complying with the RFC.

Would you accept this as a pull request?

if ( ! defined $new_id ) {

This fixes it for all clients (mind the special arguments you mentioned), affects all ProtonMail users (in positive way) and possibly others (in positive way) as well and should not break anything as the IMAPClient actually returns undef and maybe 0 is even legitimate id result.

[gilleslamiral]

Ok for if ( ! defined $new_id ) {

Yes, it looks ProtonMail is compliant with the RFC since
https://tools.ietf.org/html/rfc3501#section-2.2.2
"A client MUST be prepared to accept any server response at all times.
This includes server data that was not requested."

Anyway it is the one among a hundred to do what it does in response to an APPEND.

[exander77]

Yes, I read the same thing. The ProtonMail returns a lot of information which was not requested but actually can return anything it wants as per RFCs. I have created pull request, thank you!