Upon certificate issues, STARTTLS is ignored and the password sent in plaintext #15

Closed
devurandom opened this Issue Jan 19, 2014 · 12 comments

Comments

Projects
None yet
3 participants
@devurandom

Currently there seems to be no STARTTLS support in imapsync, is that correct? --tls1/2 on my system still results in (excerpt from output):

Info: will try to use LOGIN authentication on host1
Host1: IMAP server [$HOST1] port [$PORT1] user [$USER1]
Host1: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Host1: $HOST1 says it has NO CAPABILITY for AUTHENTICATE LOGIN
Failure: error login on [$HOST1] with user [$USER1] auth [LOGIN]: 4 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections.
@gilleslamiral

This comment has been minimized.

Show comment Hide comment
@gilleslamiral

gilleslamiral Jan 20, 2014

Contributor

Hello Dennis,

STARTTLS is supported.
Your paranoid imap server does not support plaintext authentication even with encryption.
The output should print what authentication it allows.

What are your command line options?

--authmd51 : Use MD5 authentification for host1.
--authmd52 : Use MD5 authentification for host2.
--authmech1 : Auth mechanism to use with host1:
PLAIN, LOGIN, CRAM-MD5 etc. Use UPPERCASE.
--authmech2 : Auth mechanism to use with host2. See --authmech1
--ssl1 : Use an SSL connection on host1.
--ssl2 : Use an SSL connection on host2.
--tls1 : Use an TLS connection on host1.
--tls2 : Use an TLS connection on host2.

On 19/01/2014 20:29, Dennis Schridde wrote:

Currently there seems to be no STARTTLS support in imapsync, is that correct? |--tls1/2| on my system still results in

|4 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections.
|


Reply to this email directly or view it on GitHub #15.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

Contributor

gilleslamiral commented Jan 20, 2014

Hello Dennis,

STARTTLS is supported.
Your paranoid imap server does not support plaintext authentication even with encryption.
The output should print what authentication it allows.

What are your command line options?

--authmd51 : Use MD5 authentification for host1.
--authmd52 : Use MD5 authentification for host2.
--authmech1 : Auth mechanism to use with host1:
PLAIN, LOGIN, CRAM-MD5 etc. Use UPPERCASE.
--authmech2 : Auth mechanism to use with host2. See --authmech1
--ssl1 : Use an SSL connection on host1.
--ssl2 : Use an SSL connection on host2.
--tls1 : Use an TLS connection on host1.
--tls2 : Use an TLS connection on host2.

On 19/01/2014 20:29, Dennis Schridde wrote:

Currently there seems to be no STARTTLS support in imapsync, is that correct? |--tls1/2| on my system still results in

|4 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections.
|


Reply to this email directly or view it on GitHub #15.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

@devurandom

This comment has been minimized.

Show comment Hide comment
@devurandom

devurandom Jan 20, 2014

Hello Gilles!

My server surely allows PLAIN and LOGIN over an encrypted connection. It authenticates against PAM, so there is no other way.

The commandline options were:

imapsync --dry --tls1 --host1 ... --user1 ... --password1 ... [host2=host1] --folder ... --delete2duplicates ----useheader Message-ID --usecache

Hello Gilles!

My server surely allows PLAIN and LOGIN over an encrypted connection. It authenticates against PAM, so there is no other way.

The commandline options were:

imapsync --dry --tls1 --host1 ... --user1 ... --password1 ... [host2=host1] --folder ... --delete2duplicates ----useheader Message-ID --usecache
@gilleslamiral

This comment has been minimized.

Show comment Hide comment
@gilleslamiral

gilleslamiral Jan 20, 2014

Contributor

Hi Dennis,

Ok but the error message

"NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections."

is given by the imap server, not by imapsync.

Try also

imapsync ... --ssl1

Then debug what's wrong with your server, maybe it wants certificates or something like that?

My server surely allows PLAIN and LOGIN over an encrypted connection. It authenticates against PAM, so there is no other way.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

Contributor

gilleslamiral commented Jan 20, 2014

Hi Dennis,

Ok but the error message

"NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections."

is given by the imap server, not by imapsync.

Try also

imapsync ... --ssl1

Then debug what's wrong with your server, maybe it wants certificates or something like that?

My server surely allows PLAIN and LOGIN over an encrypted connection. It authenticates against PAM, so there is no other way.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

@devurandom

This comment has been minimized.

Show comment Hide comment
@devurandom

devurandom Jan 21, 2014

The problem seems to be that imapsync cannot verify the server certificate (own CA). After STARTTLS fails locally, it tries to send CAPABILITY, which likely fails because the server expects the client to finish the STARTTLS sequence instead. Afterwards imapsync just reconnects, ignores the LOGINDISABLED capability and tries to LOGIN over a plaintext connection.

The major problems I see in this:

  1. The user is never notified of the certificate issue.
  2. imapsync ignores the --tls switch and sends my authentication plaintext. This should never ever happen.
Host1 connection
Connecting with IO::Socket::INET PeerAddr $HOST1 PeerPort $PORT1 Proto tcp Timeout 120 Debug 1
Connected to $HOST1
Read:   * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Host1: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Sending: 1 STARTTLS
Sent 12 bytes
Read:   1 OK Begin TLS negotiation now.
ERROR: Unable to start TLS: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 455.
[...]
Sending: 2 CAPABILITY
Sent 14 bytes
ERROR: socket closed while reading data from server at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 1629.
[...]
reconnecting to $HOST1, last error: socket closed while reading data from server
Connecting with IO::Socket::INET PeerAddr $HOST1 PeerPort $PORT1 Proto tcp Timeout 120 Debug 1
Connected to $HOST1
Read:   * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
reconnect success(1) on try #1/3
Sending: 3 CAPABILITY
Sent 14 bytes
Read:   * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED
        3 OK Pre-login capabilities listed, post-login capabilities have more.
Host1: $HOST1 says it has NO CAPABILITY for AUTHENTICATE LOGIN
Sending: 4 LOGIN $USER $PASS
Sent 28 bytes
Read:   * BAD [ALERT] Plaintext authentication not allowed without SSL/TLS, but your client did it anyway. If anyone was listening, the password was exposed.
        4 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections.
ERROR: 4 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections. at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 1353.
[...]

The problem seems to be that imapsync cannot verify the server certificate (own CA). After STARTTLS fails locally, it tries to send CAPABILITY, which likely fails because the server expects the client to finish the STARTTLS sequence instead. Afterwards imapsync just reconnects, ignores the LOGINDISABLED capability and tries to LOGIN over a plaintext connection.

The major problems I see in this:

  1. The user is never notified of the certificate issue.
  2. imapsync ignores the --tls switch and sends my authentication plaintext. This should never ever happen.
Host1 connection
Connecting with IO::Socket::INET PeerAddr $HOST1 PeerPort $PORT1 Proto tcp Timeout 120 Debug 1
Connected to $HOST1
Read:   * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Host1: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Sending: 1 STARTTLS
Sent 12 bytes
Read:   1 OK Begin TLS negotiation now.
ERROR: Unable to start TLS: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 455.
[...]
Sending: 2 CAPABILITY
Sent 14 bytes
ERROR: socket closed while reading data from server at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 1629.
[...]
reconnecting to $HOST1, last error: socket closed while reading data from server
Connecting with IO::Socket::INET PeerAddr $HOST1 PeerPort $PORT1 Proto tcp Timeout 120 Debug 1
Connected to $HOST1
Read:   * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
reconnect success(1) on try #1/3
Sending: 3 CAPABILITY
Sent 14 bytes
Read:   * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED
        3 OK Pre-login capabilities listed, post-login capabilities have more.
Host1: $HOST1 says it has NO CAPABILITY for AUTHENTICATE LOGIN
Sending: 4 LOGIN $USER $PASS
Sent 28 bytes
Read:   * BAD [ALERT] Plaintext authentication not allowed without SSL/TLS, but your client did it anyway. If anyone was listening, the password was exposed.
        4 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections.
ERROR: 4 NO [PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections. at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 1353.
[...]
@gilleslamiral

This comment has been minimized.

Show comment Hide comment
@gilleslamiral

gilleslamiral Jan 21, 2014

Contributor

Dear Dennis,

The problem seems to be that imapsync cannot verify the server certificate (own CA). After STARTTLS fails locally,
it tries to send CAPABILITY, which likely fails because the server expects the client to finish the STARTTLS sequence instead.

Ok. Bad imapsync.

Afterwards imapsync just reconnects, ignores the LOGINDISABLED capability and tries to LOGIN over a plaintext connection.
The major problems I see in this:

  1. The user is never notified of the certificate issue.
  2. imapsync ignores the --tls switch and sends my authentication plaintext. This should never ever happen.

Yes, you are twice right.
Shame on me, imapsync does not check the return code of the function starttls() it uses,
so it does not print the error either. It used to do it but let's forget history.
Plus, the automatic reconnect behavior is not a good idea in this scenario and then add exposure.

I'll fix that soon.

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Can you try with option --ssl1_SSL_version fixing the ssl version, for example

imapsync ... --ssl1 --ssl1_SSL_version "SSLv3"

Possibilities:
"SSLv3"
"SSLv2"
"SSLv23"
"SSLv23:!SSLv2"

|Host1 connection
Connecting with IO::Socket::INET PeerAddr $HOST1 PeerPort $PORT1 Proto tcp Timeout 120 Debug 1
Connected to $HOST1
Read: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Host1: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Sending: 1 STARTTLS
Sent 12 bytes
Read: 1 OK Begin TLS negotiation now.
ERROR: Unable to start TLS: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 455.
[...]
Sending: 2 CAPABILITY
Sent 14 bytes
ERROR: socket closed while reading data from server at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 1629.
[...]
reconnecting to $HOST1, last error: socket closed while reading data from server

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

Contributor

gilleslamiral commented Jan 21, 2014

Dear Dennis,

The problem seems to be that imapsync cannot verify the server certificate (own CA). After STARTTLS fails locally,
it tries to send CAPABILITY, which likely fails because the server expects the client to finish the STARTTLS sequence instead.

Ok. Bad imapsync.

Afterwards imapsync just reconnects, ignores the LOGINDISABLED capability and tries to LOGIN over a plaintext connection.
The major problems I see in this:

  1. The user is never notified of the certificate issue.
  2. imapsync ignores the --tls switch and sends my authentication plaintext. This should never ever happen.

Yes, you are twice right.
Shame on me, imapsync does not check the return code of the function starttls() it uses,
so it does not print the error either. It used to do it but let's forget history.
Plus, the automatic reconnect behavior is not a good idea in this scenario and then add exposure.

I'll fix that soon.

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Can you try with option --ssl1_SSL_version fixing the ssl version, for example

imapsync ... --ssl1 --ssl1_SSL_version "SSLv3"

Possibilities:
"SSLv3"
"SSLv2"
"SSLv23"
"SSLv23:!SSLv2"

|Host1 connection
Connecting with IO::Socket::INET PeerAddr $HOST1 PeerPort $PORT1 Proto tcp Timeout 120 Debug 1
Connected to $HOST1
Read: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Host1: * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS LOGINDISABLED] Dovecot ready.
Sending: 1 STARTTLS
Sent 12 bytes
Read: 1 OK Begin TLS negotiation now.
ERROR: Unable to start TLS: SSL connect attempt failed with unknown error error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 455.
[...]
Sending: 2 CAPABILITY
Sent 14 bytes
ERROR: socket closed while reading data from server at /usr/lib64/perl5/vendor_perl/5.18.1/Mail/IMAPClient.pm line 1629.
[...]
reconnecting to $HOST1, last error: socket closed while reading data from server

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

@devurandom

This comment has been minimized.

Show comment Hide comment
@devurandom

devurandom Jan 21, 2014

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Yes, the CA cert I added to /usr/local/share/ca-certificates was old and expired (i.e. not the one that signed the server's cert), so OpenSSL considered the chain untrusted. Rightly so, as there was a self-signed certificate in it. After I fixed that, imapsync worked like a charm.

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Yes, the CA cert I added to /usr/local/share/ca-certificates was old and expired (i.e. not the one that signed the server's cert), so OpenSSL considered the chain untrusted. Rightly so, as there was a self-signed certificate in it. After I fixed that, imapsync worked like a charm.

@gilleslamiral

This comment has been minimized.

Show comment Hide comment
@gilleslamiral

gilleslamiral Jan 24, 2014

Contributor

Dear Dennis,

I fixed this ugly bug in imapsync 1.582
It is hard to go in the same conditions as yours.
Now you fixed it on your side I supposed this won't interest you now.
I'll make it officially public later.
1.582 attached anyway.

On 22/01/2014 00:30, Dennis Schridde wrote:

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Yes, the CA cert I added to /usr/local/share/ca-certificates was old and expired (i.e. not the one that signed the server's cert), so OpenSSL considered the chain untrusted. Rightly so, as there was a self-signed certificate in it. After I fixed that, imapsync worked like a charm.

—
Reply to this email directly or view it on GitHub #15 (comment).

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06
#!/usr/bin/perl

structure

pod documentation

pragmas

main program

global variables initialisation

get_options( ) ;

default values

folder loop

subroutines

IMAPClient 3.xx ads

pod documentation

=pod

=head1 NAME

imapsync - IMAP synchronisation, sync, copy or migration tool.
Synchronises mailboxes between two imap servers.
Good at IMAP migration. More than 52 different IMAP server softwares
supported with success, few failures.

$Revision: 1.582 $

=head1 SYNOPSIS

To synchronize imap account "foo" on "imap.truc.org"
to imap account "bar" on "imap.trac.org"
with foo password "secret1"
and bar password "secret2":

imapsync
--host1 imap.truc.org --user1 foo --password1 secret1
--host2 imap.trac.org --user2 bar --password2 secret2

=head1 INSTALL

imapsync works fine under any Unix OS with perl.
imapsync works fine under Windows (2000, XP, Vista, Seven)
with Strawberry Perl (5.10, 5.12 or higher)
or as a standalone binary software imapsync.exe

imapsync can be available directly on the following distributions:
FreeBSD, Debian, Ubuntu, Gentoo, Fedora,
NetBSD, Darwin, Mandriva and OpenBSD.

Purchase latest imapsync at
http://imapsync.lamiral.info/

You'll receive a link to a compressed tarball called imapsync-x.xx.tgz
where x.xx is the version number. Untar the tarball where
you want (on Unix):

tar xzvf imapsync-x.xx.tgz

Go into the directory imapsync-x.xx and read the INSTALL file.
The INSTALL file is also at
http://imapsync.lamiral.info/INSTALL

The freecode (was freshmeat) record is at
http://freecode.com/projects/imapsync

=head1 USAGE

imapsync [options]

To get a description of each option just run imapsync like this:

imapsync --help
imapsync

The option list:

imapsync [--host1 server1] [--port1 ]
[--user1 ] [--passfile1 ]
[--host2 server2] [--port2 ]
[--user2 ] [--passfile2 ]
[--ssl1] [--ssl2]
[--tls1] [--tls2]
[--authmech1 ] [--authmech2 ]
[--proxyauth1] [--proxyauth2]
[--domain1] [--domain2]
[--authmd51] [--authmd52]
[--folder --folder ...]
[--folderrec --folderrec ...]
[--include ] [--exclude ]
[--prefix2 ] [--prefix1 ]
[--regextrans2 --regextrans2 ...]
[--sep1 ]
[--sep2 ]
[--justfolders] [--justfoldersizes] [--justconnect] [--justbanner]
[--syncinternaldates]
[--idatefromheader]
[--syncacls]
[--regexmess ] [--regexmess ]
[--maxsize ]
[--minsize ]
[--maxage ]
[--minage ]
[--search ]
[--search1 ]
[--search2 ]
[--skipheader ]
[--useheader ] [--useheader ]
[--nouid1] [--nouid2]
[--usecache]
[--skipsize] [--allowsizemismatch]
[--delete] [--delete2]
[--expunge] [--expunge1] [--expunge2] [--uidexpunge2]
[--delete2folders] [--delete2foldersonly] [--delete2foldersbutnot]
[--subscribed] [--subscribe] [--subscribe_all]
[--nofoldersizes] [--nofoldersizesatend]
[--dry]
[--debug] [--debugimap][--debugimap1][--debugimap2]
[--timeout ]
[--split1] [--split2]
[--reconnectretry1 ] [--reconnectretry2 ]
[--noreleasecheck]
[--pidfile ]
[--tmpdir ]
[--version] [--help]
[--tests] [--tests_debug]

=cut

comment

=pod

=head1 DESCRIPTION

The command imapsync is a tool allowing incremental and
recursive imap transfer from one mailbox to another.

By default all folders are transferred, recursively, all
possible flags (\Seen \Answered \Flagged etc.) are synced too.

We sometimes need to transfer mailboxes from one imap server to
another. This is called migration.

imapsync is a good tool because it reduces the amount
of data transferred by not transferring a given message
if it is already on both sides. Same headers
and the transfer is done only once. All flags are
preserved, unread will stay unread, read will stay read,
deleted will stay deleted. You can stop the transfer at any
time and restart it later, imapsync works well with bad
connections.

You can decide to delete the messages from the source mailbox
after a successful transfer, it can be a good feature when migrating
live mailboxes since messages will be only on one side.
In that case, use the --delete option. Option --delete implies
also option --expunge so all messages marked deleted on host1
will be really deleted.
(you can use --noexpunge to avoid this but I don't see any
good real world scenario for the combinaison --delete --noexpunge).

You can also just synchronize a mailbox B from another mailbox A
in case you just want to keep a "live" copy of A in B.
In that case --delete2 can be used, it deletes messages in host2
folder B that are not in host1 folder A.

imapsync is not adequate for maintaining two active imap accounts
in synchronization where the user plays independently on both sides.
Use offlineimap (written by John Goerzen) or mbsync (written by
Michael R. Elkins) for 2 ways synchronizations.

=head1 OPTIONS

To get a description of each option just invoke:

imapsync --help

=head1 HISTORY

I wrote imapsync because an enterprise (basystemes) paid me to install
a new imap server without losing huge old mailboxes located on a far
away remote imap server accessible by a low bandwidth link. The tool
imapcp (written in python) could not help me because I had to verify
every mailbox was well transferred and delete it after a good
transfer. imapsync started life as a copy_folder.pl patch.
The tool copy_folder.pl comes from the Mail-IMAPClient-2.1.3 perl
module tarball source (in the examples/ directory of the tarball).

=head1 EXAMPLE

While working on imapsync parameters please run imapsync in
dry mode (no modification induced) with the --dry
option. Nothing bad can be done this way.

To synchronize the imap account "buddy" (with password "secret1")
on host "imap.src.fr" to the imap account "max" (with password "secret2")
on host "imap.dest.fr":

imapsync --host1 imap.src.fr --user1 buddy --password1 secret1
--host2 imap.dest.fr --user2 max --password2 secret2

Then you will have max's mailbox updated from buddy's
mailbox.

=head1 SECURITY

You can use --passfile1 instead of --password1 to give the
password since it is safer. With --password1 option any user
on your host can see the password by using the 'ps auxwwww'
command. Using a variable (like $PASSWORD1) is also
dangerous because of the 'ps auxwwwwe' command. So, saving
the password in a well protected file (600 or rw-------) is
the best solution.

imasync is not totally protected against sniffers on the
network since passwords may be transferred in plain text
if CRAM-MD5 is not supported by your imap servers. Use
--ssl1 (or --tls1) and --ssl2 (or --tls2) to enable
encryption on host1 and host2.

You may authenticate as one user (typically an admin user),
but be authorized as someone else, which means you don't
need to know every user's personal password. Specify
--authuser1 "adminuser" to enable this on host1. In this
case, --authmech1 PLAIN will be used by default since it
is the only way to go for now. So don't use --authmech1 SOMETHING
with --authuser1 "adminuser", it will not work.
Same behavior with the --authuser2 option.
Authenticate with an admin account must be supported by your
imap server to work with imapsync.

When working on Sun/iPlanet/Netscape IMAP servers you must use
--proxyauth1 to enable administrative user to masquerade as another user.
Can also be used on destination server with --proxyauth2

You can authenticate with OAUTH when transfering from Google Apps.
The consumer key will be the domain part of the --user, and the
--password will be used as the consumer secret. It does not work
with Google Apps free edition.

=head1 EXIT STATUS

imapsync will exit with a 0 status (return code) if everything went good.
Otherwise, it exits with a non-zero status.

So if you have an unreliable internet connection, you can use this loop
in a Bourne shell:

    while ! imapsync ...; do 
          echo imapsync not complete
    done

=head1 LICENSE

imapsync is free, open, public but not always gratis software
cover by the NOLIMIT Public License.
See the LICENSE file included in the distribution or just read this
simple sentence as it is the licence text:
No limit to do anything with this work and this license.

=head1 MAILING-LIST

The public mailing-list may be the best way to get free support.

To write on the mailing-list, the address is:
imapsync@linux-france.org

To subscribe, send any message (even empty) to:
imapsync-subscribe@listes.linux-france.org
then just reply to the confirmation message.

To unsubscribe, send a message to:
imapsync-unsubscribe@listes.linux-france.org

To contact the person in charge for the list:
imapsync-request@listes.linux-france.org

The list archives are available at:
http://www.linux-france.org/prj/imapsync_list/
So consider that the list is public, anyone
can see your post. Use a pseudonym or do not
post to this list if you want to stay private.

Thank you for your participation.

=head1 AUTHOR

Gilles LAMIRAL gilles.lamiral@laposte.net

Feedback good or bad is very often welcome.

Gilles LAMIRAL earns his living by writing, installing,
configuring and teaching free, open and often gratis
softwares. It used to be "always gratis" but now it is
"often" because imapsync is sold by its author, a good
way to stay maintening and supporting free open public
softwares (see the license) over decades.

=head1 BUG REPORT GUIDELINES

Help us to help you: follow the following guidelines.

Report any bugs or feature requests to the public mailing-list
or to the author.

Before reporting bugs, read the FAQ, the README and the
TODO files. http://imapsync.lamiral.info/

Upgrade to last imapsync release, maybe the bug
is already fixed.

Upgrade to last Mail-IMAPClient Perl module.
http://search.cpan.org/dist/Mail-IMAPClient/
maybe the bug is already fixed.

Make a good title with word "imapsync" in it (my spam filter won't filter it),
Don't write an email title with just "imapsync" or "problem",
a good title is made of keywords summary, not too long (one visible line).

Don't write imapsync in uppercase in the email title, I'll
then know you run Windows and you haven't read this README yet.

Help us to help you: in your report, please include:

  • imapsync version.

  • output given with --debug --debugimap near the failure point.
    Isolate a message or two in a folder 'BUG' and use

    imapsync ... --folder 'BUG' --debug --debugimap

  • imap server software on both side and their version number.

  • imapsync with all the options you use, the full command line
    you use (except the passwords of course).

  • IMAPClient.pm version.

  • the run context. Do you run imapsync.exe, a unix binary
    or the perl script imapsync.

  • operating system running imapsync.

  • virtual software context (vmware, xen etc.)

  • operating systems on both sides and the third side in case
    you run imapsync on a foreign host from the both.

Most of those values can be found as a copy/paste at the begining of the output,
so a copy of the output is a very easy and very good debug report for me.

One time in your life, read the paper
"How To Ask Questions The Smart Way"
http://www.catb.org/~esr/faqs/smart-questions.html
and then forget it.

=head1 IMAP SERVERS

Failure stories reported with the following 3 imap servers:

  • MailEnable 1.54 (Proprietary) but MailEnable 4.23 is supported.
  • DBMail 0.9, 2.0.7 (GPL). But DBMail 1.2.1 is supported.
    Patient and confident testers are welcome.
  • Imail 7.04 (maybe).
  • (2011) MDaemon 12.0.3 as host2 but MDaemon is supported as host1.
    MDaemon is simply buggy with the APPEND IMAP command with
    any IMAP email client.
  • Hotmail since hotmail.com does not provide IMAP access
  • Outlook.com since outlook.com does not provide IMAP access

Success stories reported with the following 57 imap servers
(software names are in alphabetic order):

  • 1und1 H mimap1 84498 [host1] H mibap4 95231 [host1]
  • a1.net imap.a1.net IMAP4 Ready [host1]
  • Apple Server 10.6 Snow Leopard [host1]
  • Archiveopteryx 2.03, 2.04, 2.09, 2.10 [host2], 3.0.0 [host2](OSL 3.0) http://www.archiveopteryx.org/
  • Atmail 6.x [host1]
  • Axigen Mail Server Version 8.0.0
  • BincImap 1.2.3 (GPL) (http://www.bincimap.org/)
  • CommuniGatePro server (Redhat 8.0) (Solaris), CommuniGate Pro 5.2.17[host2](CentOS 5.4)
  • Courier IMAP 1.5.1, 2.2.0, 2.1.1, 2.2.1, 3.0.8, 3.0.3, 4.1.1 (GPL)
    (http://www.courier-mta.org/)
  • Critical Path (7.0.020)
  • Cyrus IMAP 1.5, 1.6,
    2.1, 2.1.15, 2.1.16, 2.1.18
    2.2.1, 2.2.2-BETA, 2.2.3, 2.2.6, 2.2.10, 2.2.12, 2.2.13,
    2.3-alpha (OSI Approved), 2.3.1, 2.3.7, 2.3.16
    (http://asg.web.cmu.edu/cyrus/)
  • David Tobit V8 (proprietary Message system).
  • Deerfield VisNetic MailServer 5.8.6 host1
  • DBMail 1.2.1, 2.0.4, 2.0.9, 2.2rc1 (GPL) (http://www.dbmail.org/).
    2.0.7 seems buggy.
  • DBOX 2.41 System host1.
  • Deerfield VisNetic MailServer 5.8.6 [host1]
  • dkimap4 [host1]
  • Domino (Notes) 4.61 [host1], 6.5 [host1], 5.0.6, 5.0.7, 7.0.2, 6.0.2CF1,
    7.0.1 [host1], 8.0.1 [host1], 8.5.2 [host2], 8.5.3 [host1]
  • Dovecot 0.99.10.4, 0.99.14, 0.99.14-8.fc4, 1.0-0.beta2.7,
    1.0.0 dest/source (http://www.dovecot.org/)
  • Eudora WorldMail v2
  • Fusemail imap.fusemail.net:143 (https://www.fusemail.com/).
  • Gimap (Gmail imap)
  • GMX IMAP4 StreamProxy.
  • Groupwise IMAP (Novell) 6.x and 7.0. Buggy so see the FAQ.
  • hMailServer 5.40-B1950 [host12], 5.3.3 [host2], 4.4.1 [host1](see FAQ)
  • IceWarp Server 10.4.5 host1
  • iPlanet Messaging server 4.15, 5.1, 5.2
  • IMail 7.15 (Ipswitch/Win2003), 8.12, 11.03 [host1]
  • Kerio 7.2.0 Patch 1 [host12], Kerio 8 [host1]
  • Mail2World IMAP4 Server 2.5 host1
  • MailEnable 4.23 [host1] [host2], 4.26 [host1][host2], 5 [host1]
  • MDaemon 7.0.1, 8.0.2, 8.1, 9.5.4 (Windows server 2003 R2 platform),
    9.6.5 [host1], 12 [host2], 12.0.3 [host1], 12.5.5 [host1],
  • Mercury 4.1 (Windows server 2000 platform)
  • Microsoft Exchange Server 5.5, 6.0.6249.0[host1], 6.0.6487.0[host1],
    6.5.7638.1 [host2], 6.5 [host1], Exchange 2007 SP1 (with Update Rollup 2),
    Exchange2007-EP-SP2,
    Exchange 2010 RTM (Release to Manufacturing) [host2],
    Exchange 2010 SP1 RU2[host2],
  • Mirapoint, 4.1.9-GA [host1]
  • Netscape Mail Server 3.6 (Wintel !)
  • Netscape Messaging Server 4.15 Patch 7
  • Office 365 [host1] [host2]
  • OpenMail IMAP server B.07.00.k0 (Samsung Contact ?)
  • OpenWave
  • Oracle Beehive [host1]
  • Parallels Plesk Panel 9.x [host2] 11.x host2
  • Qualcomm Worldmail (NT)
  • QQMail IMAP4Server [host1] [host2] https://en.mail.qq.com/
  • RackSpace hoster secure.emailsrvr.com:993 http://www.rackspace.com/
  • Rockliffe Mailsite 5.3.11, 4.5.6
  • Samsung Contact IMAP server 8.5.0
  • Scalix v10.1, 10.0.1.3, 11.0.0.431, 11.4.6
  • Sendmail Mail Store IMAP4rev1 (5.5.6/mstore-5-5-build-1874 [host1].
  • SmarterMail, Smarter Mail 5.0 Enterprise, Smarter Mail 5.5 [host1],
    SmarterMail Professional 10.2 [host1], Smarter Mail 11.7 [host1][host2].
  • Softalk Workgroup Mail 7.6.4 [host1].
  • SunONE Messaging server 5.2, 6.0 (SUN JES - Java Enterprise System)
  • Sun Java(tm) System Messaging Server 6.2-2.05, 6.2-7.05, 6.3
  • Surgemail 3.6f5-5, 6.3d-72 [host2]
  • UW-imap servers (imap-2000b) rijkkramer IMAP4rev1 2000.287
    (RedHat uses UW like 2003.338rh), v12.264 Solaris 5.7 (OSI Approved)
    (http://www.washington.edu/imap/)
  • UW - QMail v2.1
  • VMS, Imap part of TCP/IP suite of VMS 7.3.2
  • Yahoo [host1]
  • Zarafa 6,40,0,20653 host1
  • Zarafa ZCP 7.1.4 IMAP Gateway [host2]
  • Zimbra-IMAP 3.0.1 GA 160, 3.1.0 Build 279, 4.0.5, 4.5.2, 4.5.6,
    Zimbra 5.0.24_GA_3356.RHEL4 [host1], 5.5, 6.x

Please report to the author any success or bad story with
imapsync and do not forget to mention the IMAP server
software names and version on both sides. This will help
future users. To help the author maintaining this section
report the two lines at the begining of the output if they
are useful to know the softwares. Example:

Host1 software:* OK louloutte Cyrus IMAP4 v1.5.19 server ready
Host2 software:* OK Courier-IMAP ready

You can use option --justconnect to get those lines.
Example:

imapsync --host1 imap.troc.org --host2 imap.trac.org --justconnect

=head1 HUGE MIGRATION

Pay special attention to options
--subscribed
--subscribe
--delete
--delete2
--delete2folders
--maxage
--minage
--maxsize
--useuid
--usecache

If you have many mailboxes to migrate think about a little
shell program. Write a file called file.txt (for example)
containing users and passwords.
The separator used in this example is ';'

The file.txt file contains:

user001_1;password001_1;user001_2;password001_2
user002_1;password002_1;user002_2;password002_2
user003_1;password003_1;user003_2;password003_2
user004_1;password004_1;user004_2;password004_2
user005_1;password005_1;user005_2;password005_2
...

On Unix the shell program can be:

{ while IFS=';' read u1 p1 u2 p2; do
imapsync --host1 imap.side1.org --user1 "$u1" --password1 "$p1"
--host2 imap.side2.org --user2 "$u2" --password2 "$p2" ...
done ; } < file.txt

On Windows the batch program can be:

FOR /F "tokens=1,2,3,4 delims=; eol=#" %%G IN (file.txt) DO imapsync ^
--host1 imap.side1.org --user1 %%G --password1 %%H ^
--host2 imap.side2.org --user2 %%I --password2 %%J ...

The ... have to be replaced by nothing or any imapsync option.

Welcome in shell programming !

=head1 Hacking

Feel free to hack imapsync as the NOLIMIT license permits it.

=head1 Links

Entries for imapsync:
http://www.imap.org/products/showall.php

=head1 SIMILAR SOFTWARES

imap_tools : http://www.athensfbc.com/imap_tools
offlineimap : https://github.com/nicolas33/offlineimap
mbsync : http://isync.sourceforge.net/
mailsync : http://mailsync.sourceforge.net/
mailutil : http://www.washington.edu/imap/
part of the UW IMAP tookit.
imaprepl : http://www.bl0rg.net/software/
http://freecode.com/projects/imap-repl/
imapcopy : http://home.arcor.de/armin.diehl/imapcopy/imapcopy.html
migrationtool : http://sourceforge.net/projects/migrationtool/
imapmigrate : http://sourceforge.net/projects/cyrus-utils/
wonko_imapsync: http://wonko.com/article/554
see also file W/tools/wonko_ruby_imapsync
exchange-away : http://exchange-away.sourceforge.net/
pop2imap : http://www.linux-france.org/prj/pop2imap/

Feedback (good or bad) will often be welcome.

$Id: imapsync,v 1.582 2014/01/24 01:43:19 gilles Exp gilles $

=cut

pragmas

use strict ;
use warnings ;
++$| ;
use Carp ;
use Getopt::Long ;
use Mail::IMAPClient 3.29 ;
use Digest::MD5 qw( md5 md5_hex md5_base64 ) ;
use Digest::HMAC_SHA1 qw( hmac_sha1 ) ;
#use Term::ReadKey ;
#use IO::Socket::SSL ;
use MIME::Base64 ;
use English '-no_match_vars' ;
use File::Basename ;
use POSIX qw(uname SIGALRM) ;
use Fcntl ;
use File::Spec ;
use File::Path qw( mkpath rmtree ) ;
use File::Copy::Recursive ;
use IO::Socket qw(:crlf SOL_SOCKET SO_KEEPALIVE) ;
use Errno qw(EAGAIN EPIPE ECONNRESET) ;
use File::Glob qw( :glob ) ;
use IO::File ;
use Time::Local ;
use Time::HiRes qw( time sleep ) ;
use Test::More 'no_plan' ;
use IPC::Open3 'open3' ;
#use Unix::Sysexits ;

global variables

my(
$rcs, $pidfile, $pidfilelocking,
$debug, $debugimap, $debugimap1, $debugimap2, $debugcontent, $debugflags,
$debugLIST, $debugsleep, $debugdev, $debugmemory, $debugmaxlinelength,
$nb_errors,
$host1, $host2, $port1, $port2,
$user1, $user2, $domain1, $domain2,
$password1, $password2, $passfile1, $passfile2,
@folder, @include, @exclude, @folderrec,
@folderfirst, @folderlast,
$prefix1, $prefix2,
@regextrans2, @regexmess, @regexflag,
$flagsCase, $filterflags, $syncflagsaftercopy,
$sep1, $sep2,
$syncinternaldates,
$idatefromheader,
$syncacls,
$fastio1, $fastio2,
$maxsize, $minsize, $maxage, $minage,
$exitwhenover,
$search, $search1, $search2,
$skipheader, @useheader,
$skipsize, $allowsizemismatch, $foldersizes, $foldersizesatend, $buffersize,
$delete, $delete2, $delete2duplicates,
$expunge, $expunge1, $expunge2, $uidexpunge2, $dry,
$justfoldersizes,
$authmd5, $authmd51, $authmd52,
$subscribed, $subscribe, $subscribe_all,
$version, $help,
$justconnect, $justfolders, $justbanner,
$fast,

    $total_bytes_transferred,
    $total_bytes_skipped,
    $total_bytes_error,
    $nb_msg_transferred, 
$nb_msg_skipped, 
$nb_msg_skipped_dry_mode,
$h1_nb_msg_duplicate,
$h2_nb_msg_duplicate,
$h1_nb_msg_noheader,
$h2_nb_msg_noheader,
$h1_total_bytes_duplicate,
$h2_total_bytes_duplicate,
$h1_nb_msg_deleted,
$h2_nb_msg_deleted,

    $h1_bytes_processed, 
    $h1_nb_msg_processed,
    $h1_nb_msg_at_start, $h1_bytes_start,
    $h2_nb_msg_start, $h2_bytes_start, 
    $h1_nb_msg_end, $h1_bytes_end,
    $h2_nb_msg_end, $h2_bytes_end, 

    $timeout,
$timestart, $timestart_int, $timeend,
    $timebefore,
    $ssl1, $ssl2, 
    $ssl1_SSL_version, $ssl2_SSL_version,
$tls1, $tls2,
$uid1, $uid2,
    $authuser1, $authuser2,
    $proxyauth1, $proxyauth2,
    $authmech1, $authmech2,
    $split1, $split2,
    $reconnectretry1, $reconnectretry2,
$relogin1, $relogin2,
$tests, $test_builder, $tests_debug,
$allow3xx, $justlogin,
$tmpdir,
$releasecheck,
$max_msg_size_in_bytes,
$modules_version,
$delete2folders, $delete2foldersonly, $delete2foldersbutnot,
$usecache, $debugcache, $cacheaftercopy,
$wholeheaderifneeded, %h1_msgs_copy_by_uid, $useuid, $h2_uidguess,
    $addheader,
    %h1, %h2,
    $checkselectable, $checkmessageexists,
    $expungeaftereach,
    $abletosearch,
    $showpasswords,
    $fixslash2,
    $messageidnodomain,
    $fixInboxINBOX,
    $maxlinelength,
    $minmaxlinelength,
$uidnext_default,
    $fixcolonbug,
    $create_folder_old,
    $maxmessagespersecond,
    $maxbytespersecond,
    $skipcrossduplicates, $debugcrossduplicates, 

);

main program

global variables initialisation

$rcs = '$Id: imapsync,v 1.582 2014/01/24 01:43:19 gilles Exp gilles $ ';

$total_bytes_transferred = 0;
$total_bytes_skipped = 0;
$total_bytes_error = 0;
$nb_msg_transferred = 0;
$nb_msg_skipped = $nb_msg_skipped_dry_mode = 0;
$h1_nb_msg_deleted = $h2_nb_msg_deleted = 0;
$h1_nb_msg_duplicate = $h2_nb_msg_duplicate = 0;
$h1_nb_msg_noheader = $h2_nb_msg_noheader = 0;
$h1_total_bytes_duplicate = $h2_total_bytes_duplicate = 0;

$h1_nb_msg_at_start = $h1_bytes_start = 0 ;
$h2_nb_msg_start = $h2_bytes_start = 0 ;
$h1_nb_msg_processed = $h1_bytes_processed = 0 ;

$h1_nb_msg_end = $h1_bytes_end = 0 ;
$h2_nb_msg_end = $h2_bytes_end = 0 ;

$nb_errors = 0;
$max_msg_size_in_bytes = 0;

my %month_abrev = (
Jan => 0,
Feb => 1,
Mar => 2,
Apr => 3,
May => 4,
Jun => 5,
Jul => 6,
Aug => 7,
Sep => 8,
Oct => 9,
Nov => 10,
Dec => 11,
);

sub EX_USAGE {
# 64 on my linux box.
# See http://search.cpan.org/~jmates/Unix-Sysexits-0.02/lib/Unix/Sysexits.pm
return( 64 ) ;
}

@argv will be eat by get_options()

my @argv_copy = @argv;

get_options( ) ;

$SIG{ INT } = &catch_continue ;

local $SIG{ INT } = local $SIG{ QUIT } = local $SIG{ TERM } = &catch_exit ;

$timestart = time( );
$timestart_int = int( $timestart ) ;
$timebefore = $timestart ;

my $timestart_str = localtime( $timestart ) ;
print "Transfer started at $timestart_str\n" ;
print "PID is $PROCESS_ID\n" ;
$modules_version = defined( $modules_version ) ? $modules_version : 1 ;

$releasecheck = defined($releasecheck) ? $releasecheck : 1;
my $warn_release = ($releasecheck) ? check_last_release() : '';

default values

$tmpdir ||= File::Spec->tmpdir();
$pidfile ||= $tmpdir . '/imapsync.pid';

$pidfilelocking = defined( $pidfilelocking ) ? $pidfilelocking : 0 ;

allow Mail::IMAPClient 3.0.xx by default

$allow3xx = defined($allow3xx) ? $allow3xx : 1;

$wholeheaderifneeded = defined( $wholeheaderifneeded ) ? $wholeheaderifneeded : 1;

turn on RFC standard flags correction like \SEEN -> \Seen

$flagsCase = defined( $flagsCase ) ? $flagsCase : 1 ;

Use PERMANENTFLAGS if available

$filterflags = defined( $filterflags ) ? $filterflags : 1 ;

sync flags just after an APPEND, some servers ignore the flags given in the APPEND

like MailEnable IMAP server.

Off by default since it takes time.

$syncflagsaftercopy = defined( $syncflagsaftercopy ) ? $syncflagsaftercopy : 0 ;

turn on relogin 5 by default

$relogin1 = defined( $relogin1 ) ? $relogin1 : 5 ;
$relogin2 = defined( $relogin2 ) ? $relogin2 : 5 ;

if ( $fast ) {
# $useuid = 1 ;
# $foldersizes = 0 ;
# $foldersizesatend = 0 ;
}

Activate --usecache if --useuid is set and no --nousecache

$usecache = 1 if ( $useuid and ( ! defined( $usecache ) ) ) ;
$cacheaftercopy = 1 if ( $usecache and ( ! defined( $cacheaftercopy ) ) ) ;

$checkselectable = defined( $checkselectable ) ? $checkselectable : 1 ;
$checkmessageexists = defined( $checkmessageexists ) ? $checkmessageexists : 0 ;
$expungeaftereach = defined( $expungeaftereach ) ? $expungeaftereach : 1 ;
$abletosearch = defined( $abletosearch ) ? $abletosearch : 1 ;
$checkmessageexists = 0 if ( not $abletosearch ) ;
$showpasswords = defined( $showpasswords ) ? $showpasswords : 0 ;
$fixslash2 = defined( $fixslash2 ) ? $fixslash2 : 1 ;
$fixInboxINBOX = defined( $fixInboxINBOX ) ? $fixInboxINBOX : 1 ;
$create_folder_old = defined( $create_folder_old ) ? $create_folder_old : 0 ;

$delete2duplicates = 1 if ( $delete2 and ( ! defined( $delete2duplicates ) ) ) ;

$maxmessagespersecond = defined( $maxmessagespersecond ) ? $maxmessagespersecond : 0 ;
$maxbytespersecond = defined( $maxbytespersecond ) ? $maxbytespersecond : 0 ;

print banner_imapsync(@argv_copy);

print "Temp directory is $tmpdir\n";

is_valid_directory( $tmpdir ) ;
write_pidfile( $pidfile ) if ( $pidfile ) ;

$fixcolonbug = defined( $fixcolonbug ) ? $fixcolonbug : 1 ;

if ( $usecache and $fixcolonbug ) { tmpdir_fix_colon_bug( ) } ;

$modules_version and print "Modules version list:\n", modules_VERSION(), "\n";

check_lib_version() or
croak "imapsync needs perl lib Mail::IMAPClient release 3.25 or superior \n";

exit_clean( 0 ) if ( $justbanner ) ;

By default, 100 at a time, not more.

$split1 ||= 100;
$split2 ||= 100;

$host1 || missing_option("--host1") ;
$port1 ||= ( $ssl1 ) ? 993 : 143;

$host2 || missing_option("--host2") ;
$port2 ||= ( $ssl2 ) ? 993 : 143;

$debugimap1 = $debugimap2 = 1 if ( $debugimap ) ;
$debug = 1 if ( $debugimap1 or $debugimap2 ) ;

By default, don't take size to compare

$skipsize = (defined $skipsize) ? $skipsize : 1;

$uid1 = defined($uid1) ? $uid1 : 1;
$uid2 = defined($uid2) ? $uid2 : 1;

$subscribe = defined($subscribe) ? $subscribe : 1;

Allow size mismatch by default

$allowsizemismatch = defined($allowsizemismatch) ? $allowsizemismatch : 1;

$delete2folders = 1
if ( defined( $delete2foldersbutnot ) or defined( $delete2foldersonly ) ) ;

if ($justconnect) {
justconnect();
exit_clean(0);
}

$user1 || missing_option("--user1");
$user2 || missing_option("--user2");

$syncinternaldates = defined($syncinternaldates) ? $syncinternaldates : 1;

Turn on expunge if there is not explicit option --noexpunge and option

--delete is given.

Done because --delete --noexpunge is very dangerous on the second run:

the Deleted flag is then synced to all previously transfered messages.

So --delete implies --expunge is a better usability default behaviour.

if ($delete) {
if ( ! defined($expunge)) {
$expunge = 1;
}
}

if ( $uidexpunge2 and not Mail::IMAPClient->can( 'uidexpunge' ) ) {
print "Failure: uidexpunge not supported (IMAPClient release < 3.17), use --expunge2 instead\n" ;
exit_clean( 3 ) ;
}

if ( ( $delete2 or $delete2duplicates ) and not defined( $uidexpunge2 ) ) {
if ( Mail::IMAPClient->can( 'uidexpunge' ) ) {
print "Info: will act as --uidexpunge2\n" ;
$uidexpunge2 = 1 ;
}elsif ( not defined( $expunge2 ) ) {
print "Info: will act as --expunge2 (no uidexpunge support)\n" ;
$expunge2 = 1 ;
}
}

if ( $delete and $delete2 ) {
print "Warning: using --delete and --delete2 together is almost always a bad idea, exiting imapsync\n" ;
exit_clean( 4 ) ;
}

if ($idatefromheader) {
print "Turned ON idatefromheader, ",
"will set the internal dates on host2 from the 'Date:' header line.\n";
$syncinternaldates = 0;
}

if ($syncinternaldates) {
print "Info: turned ON syncinternaldates, ",
"will set the internal dates (arrival dates) on host2 same as host1.\n";
}else{
print "Info: turned OFF syncinternaldates\n";
}

if (defined($authmd5) and ($authmd5)) {
$authmd51 = 1 ;
$authmd52 = 1 ;
}

if (defined($authmd51) and ($authmd51)) {
$authmech1 ||= 'CRAM-MD5';
}
else{
$authmech1 ||= $authuser1 ? 'PLAIN' : 'LOGIN';
}

if (defined($authmd52) and ($authmd52)) {
$authmech2 ||= 'CRAM-MD5';
}
else{
$authmech2 ||= $authuser2 ? 'PLAIN' : 'LOGIN';
}

$authmech1 = uc($authmech1);
$authmech2 = uc($authmech2);

if (defined $proxyauth1 && !$authuser1) {
missing_option("With --proxyauth1, --authuser1");
}

if (defined $proxyauth2 && !$authuser2) {
missing_option("With --proxyauth2, --authuser2");
}

$authuser1 ||= $user1;
$authuser2 ||= $user2;

print "Info: will try to use $authmech1 authentication on host1\n";
print "Info: will try to use $authmech2 authentication on host2\n";

$timeout = defined( $timeout ) ? $timeout : 120 ;
print "Info: imap connexions timeout is $timeout seconds\n";

$syncacls = (defined($syncacls)) ? $syncacls : 0 ;
$foldersizes = (defined($foldersizes)) ? $foldersizes : 1 ;
$foldersizesatend = (defined($foldersizesatend)) ? $foldersizesatend : $foldersizes ;

$fastio1 = (defined($fastio1)) ? $fastio1 : 0;
$fastio2 = (defined($fastio2)) ? $fastio2 : 0;

$reconnectretry1 = (defined($reconnectretry1)) ? $reconnectretry1 : 3;
$reconnectretry2 = (defined($reconnectretry2)) ? $reconnectretry2 : 3;

Since select_msgs() returns no messages when uidnext does not return something

then $uidnext_default is never used. So I have to remove it.

$uidnext_default = 999999 ;

@useheader = ( "Message-Id", "Message-ID", "Received" ) unless ( @useheader ) ;

my %useheader ;

Make a hash %useheader of each --useheader 'key' in uppercase

for ( @useheader ) { $useheader{ uc( $_ ) } = undef } ;

#require Data::Dumper ;
#print Data::Dumper->Dump( [ %useheader ] ) ;
#exit ;

print "Host1: IMAP server [$host1] port [$port1] user [$user1]\n";
print "Host2: IMAP server [$host2] port [$port2] user [$user2]\n";

$password1 || $passfile1 || 'PREAUTH' eq $authmech1 || 'EXTERNAL' eq $authmech1 || do {
$password1 = ask_for_password( $authuser1 || $user1, $host1 ) ;
} ;

$password1 = ( defined( $passfile1 ) ) ? firstline ( $passfile1 ) : $password1 ;

$password2 || $passfile2 || 'PREAUTH' eq $authmech2 || 'EXTERNAL' eq $authmech2 || do {
$password2 = ask_for_password( $authuser2 || $user2, $host2 ) ;
} ;

$password2 = ( defined( $passfile2 ) ) ? firstline ( $passfile2 ) : $password2 ;

my $dry_message = '' ;
$dry_message = "\t(not really since --dry mode)" if $dry ;

$search1 ||= $search if ( $search ) ;
$search2 ||= $search if ( $search ) ;

if ( @regexmess ) {
my $string = regexmess( '' ) ;
# string undef means one of the eval regex was bad.
if ( not ( defined( $string ) ) ) {
die_clean( "Error: one of --regexmess option is bad, check it" ) ;
}
}

if ( @regexflag and not ( defined( flags_regex( '' ) ) ) ) {
die_clean( "Error: one of --regexmess option is bad, check it" ) ;
}

my $imap1 = ();
my $imap2 = ();

$debugimap1 and print "Host1 connection\n";
$imap1 = login_imap($host1, $port1, $user1, $domain1, $password1,
$debugimap1, $timeout, $fastio1, $ssl1, $tls1,
$authmech1, $authuser1, $reconnectretry1,
$proxyauth1, $uid1, $split1, 'Host1', $ssl1_SSL_version );

$debugimap2 and print "Host2 connection\n";
$imap2 = login_imap($host2, $port2, $user2, $domain2, $password2,
$debugimap2, $timeout, $fastio2, $ssl2, $tls2,
$authmech2, $authuser2, $reconnectretry2,
$proxyauth2, $uid2, $split2, 'Host2', $ssl2_SSL_version );

$debug and print "Host1 Buffer I/O: ", $imap1->Buffer(), "\n";
$debug and print "Host2 Buffer I/O: ", $imap2->Buffer(), "\n";

die_clean( 'Not authenticated on host1' ) unless $imap1->IsAuthenticated( ) ;
print "Host1: state Authenticated\n";
die_clean( 'Not authenticated on host2' ) unless $imap2->IsAuthenticated( ) ;
print "Host2: state Authenticated\n";

print "Host1 capability: ", join(" ", @{ $imap1->capability_update() || [] }), "\n";
print "Host2 capability: ", join(" ", @{ $imap2->capability_update() || [] }), "\n";

exit_clean(0) if ($justlogin);

Folder stuff

my (
@h1_folders_all, %h1_folders_all, @h1_folders_wanted, %requested_folder,
%h1_subscribed_folder, %h2_subscribed_folder,
@h2_folders_all, %h2_folders_all,
@h2_folders_from_1_wanted, %h2_folders_from_1_wanted,
%h2_folders_from_1_several,
%h2_folders_from_1_all,
);

Make a hash of subscribed folders in both servers.

for ( $imap1->subscribed( ) ) { $h1_subscribed_folder{ $_ } = 1 } ;
for ( $imap2->subscribed( ) ) { $h2_subscribed_folder{ $_ } = 1 } ;

All folders on host1 and host2

@h1_folders_all = sort $imap1->folders();
@h2_folders_all = sort $imap2->folders();

for ( @h1_folders_all ) { $h1_folders_all{ $_ } = 1 } ;
for ( @h2_folders_all ) { $h2_folders_all{ $_ } = 1 } ;

if ( $fixInboxINBOX and ( my $reg = fix_Inbox_INBOX_mapping( %h1_folders_all, %h2_folders_all ) ) ) {
#print "RRRRRR $reg\n" ;
push( @regextrans2, $reg ) ;
}

if (scalar(@folder) or $subscribed or scalar(@folderrec)) {
# folders given by option --folder
if (scalar(@folder)) {
add_to_requested_folders(@folder);
}

# option --subscribed
if ( $subscribed ) {
    add_to_requested_folders( keys ( %h1_subscribed_folder ) ) ;
}

# option --folderrec
if (scalar(@folderrec)) {
    foreach my $folderrec (@folderrec) {
        add_to_requested_folders($imap1->folders($folderrec));
    }
}

}
else {
# no include, no folder/subscribed/folderrec options => all folders
if (not scalar(@include)) {
add_to_requested_folders(@h1_folders_all);
}
}

consider (optional) includes and excludes

if ( scalar( @include ) ) {
foreach my $include ( @include ) {
my @included_folders = grep { /$include/x } @h1_folders_all ;
add_to_requested_folders( @included_folders ) ;
my $included_folders = join( " ", map( "[$_]", @included_folders ) ) ;
print "Including folders matching pattern '$include': " . $included_folders . "\n" ;
}
}

if ( scalar( @exclude ) ) {
foreach my $exclude ( @exclude ) {
my @requested_folder = sort( keys( %requested_folder ) ) ;
my @excluded_folders = grep { /$exclude/x } @requested_folder ;
remove_from_requested_folders( @excluded_folders ) ;
my $excluded_folders = join( " ", map( "[$_]", @excluded_folders ) ) ;
print "Excluding folders matching pattern '$exclude': " . $excluded_folders . "\n" ;
}
}

Remove no selectable folders

$checkselectable and do {
foreach my $folder (keys(%requested_folder)) {
if ( not $imap1->selectable($folder)) {
print "Warning: ignoring folder $folder because it is not selectable\n";
remove_from_requested_folders($folder);
}
}
} ;

@h1_folders_wanted = sort_requested_folders( ) ;

#my $h1_namespace = $imap1->namespace() ;
#my $h2_namespace = $imap2->namespace() ;
#require Data::Dumper ;
#$debug and print "Host1 namespace:\n", Data::Dumper->Dump([$h1_namespace]) ;
#$debug and print "Host2 namespace:\n", Data::Dumper->Dump([$h2_namespace]) ;

my($h1_sep,$h2_sep);

what are the private folders separators for each server ?

$debug and print "Getting separators\n";
$h1_sep = get_separator($imap1, $sep1, "--sep1");
$h2_sep = get_separator($imap2, $sep2, "--sep2");

my($h1_prefix,$h2_prefix);
$h1_prefix = get_prefix($imap1, $prefix1, "--prefix1");
$h2_prefix = get_prefix($imap2, $prefix2, "--prefix2");

print "Host1 separator and prefix: [$h1_sep][$h1_prefix]\n";
print "Host2 separator and prefix: [$h2_sep][$h2_prefix]\n";

#my $h1_xlist_folders = $imap1->xlist_folders( ) ;
#my $h2_xlist_folders = $imap2->xlist_folders( ) ;
#require Data::Dumper ;
#print "Host1 xlist:\n", Data::Dumper->Dump([$h1_xlist_folders]) ;
#print "Host2 xlist:\n", Data::Dumper->Dump([$h2_xlist_folders]) ;

#exit ;

foreach my $h1_fold ( @h1_folders_wanted ) {
my $h2_fold ;
$h2_fold = imap2_folder_name( $h1_fold ) ;
$h2_folders_from_1_wanted{ $h2_fold }++ ;
if ( 1 < $h2_folders_from_1_wanted{ $h2_fold } ) {
$h2_folders_from_1_several{ $h2_fold }++ ;
}
}
@h2_folders_from_1_wanted = sort keys(%h2_folders_from_1_wanted);

foreach my $h1_fold (@h1_folders_all) {
my $h2_fold;
$h2_fold = imap2_folder_name($h1_fold);
$h2_folders_from_1_all{$h2_fold}++;
}

if ( $foldersizes ) {
( $h1_nb_msg_at_start, $h1_bytes_start ) = foldersizes( "Host1", $imap1, $search1, @h1_folders_wanted ) ;
( $h2_nb_msg_start, $h2_bytes_start ) = foldersizes( "Host2", $imap2, $search2, @h2_folders_from_1_wanted ) ;
$fast or sleep( 2 ) ;
}

exit_clean(0) if ($justfoldersizes);

print
"++++ Listing folders\n",
"Host1 folders list:\n", map( { "[$]\n" } @h1_folders_all ), "\n",
"Host2 folders list:\n", map( { "[$
]\n" } @h2_folders_all ), "\n" ;

print
"Host1 subscribed folders list: ",
map( { "[$_] " } sort keys( %h1_subscribed_folder ) ), "\n"
if ( $subscribed ) ;

my @h2_folders_not_in_1;
@h2_folders_not_in_1 = list_folders_in_2_not_in_1();

print "Folders in host2 not in host1:\n",
map( { "[$_]\n" } @h2_folders_not_in_1 ), "\n" ;

delete_folders_in_2_not_in_1() if $delete2folders;

folder loop

print "++++ Looping on each folder\n";

my $begin_transfer_time = time ;

my %uid_candidate_for_deletion ;
my %uid_candidate_no_deletion ;

my %h2_folders_of_md5 = ( ) ;

FOLDER: foreach my $h1_fold ( @h1_folders_wanted ) {

    last FOLDER if $imap1->IsUnconnected();
    last FOLDER if $imap2->IsUnconnected();

my $h2_fold = imap2_folder_name( $h1_fold ) ;
#relogin1(  ) if ( $relogin1 ) ;
printf( "%-35s -> %-35s\n", "[$h1_fold]", "[$h2_fold]" ) ;

# host1 can not be fetched read only, select is needed because of expunge.
select_folder( $imap1, $h1_fold, 'Host1' ) or next FOLDER ;
#examine_folder( $imap1, $h1_fold, 'Host1' ) or next FOLDER ;


if ( ! exists( $h2_folders_all{ $h2_fold } ) ) {
    create_folder( $imap2, $h2_fold, $h1_fold ) or next FOLDER ;
}

acls_sync( $h1_fold, $h2_fold ) ;

    # Sometimes the folder on host2 is listed (it exists) but is
    # not selectable but becomes selectable by a create (Gmail)
select_folder( $imap2, $h2_fold, 'Host2' ) 
    or ( create_folder( $imap2, $h2_fold, $h1_fold ) 
         and select_folder( $imap2, $h2_fold, 'Host2' ) )
    or next FOLDER ;
my @select_results = $imap2->Results(  ) ;

#print "%%% @select_results\n" ;
my $permanentflags2 = permanentflags( @select_results ) ;
( $debug or $debugflags ) and print "permanentflags: $permanentflags2\n" ;

if ( $expunge or $expunge1 ){
    print "Expunging host1 $h1_fold $dry_message\n" ;
    unless($dry) { $imap1->expunge() } ;
    #print "Expunging host2 $h2_fold\n" ;
    #unless($dry) { $imap2->expunge() } ;
}

if ( ( ( $subscribe and exists $h1_subscribed_folder{ $h1_fold } ) or $subscribe_all )
         and not exists( $h2_subscribed_folder{ $h2_fold } ) ) {
    print "Subscribing to folder $h2_fold on destination server\n" ;
    unless( $dry ) { $imap2->subscribe( $h2_fold ) } ;
}

next FOLDER if ($justfolders);

    last FOLDER if $imap1->IsUnconnected();
    last FOLDER if $imap2->IsUnconnected();

    my $h1_msgs_all_hash_ref = {  } ;
my @h1_msgs = select_msgs( $imap1, $h1_msgs_all_hash_ref, $search1, $h1_fold );
last FOLDER if $imap1->IsUnconnected();

    my $h1_msgs_nb = scalar( @h1_msgs ) ;
    $h1{ $h1_fold }{ 'messages_nb' } = $h1_msgs_nb ;

( $debug or $debugLIST ) and print "Host1 LIST: $h1_msgs_nb messages [@h1_msgs]\n" ;
    $debug and print "Host1 selecting messages of folder [$h1_fold] took ", timenext(), " s\n";

    my $h2_msgs_all_hash_ref = {  } ;
my @h2_msgs = select_msgs( $imap2, $h2_msgs_all_hash_ref, $search2, $h2_fold ) ;
last FOLDER if $imap2->IsUnconnected();

    my $h2_msgs_nb = scalar( @h2_msgs ) ;
    $h2{ $h2_fold }{ 'messages_nb' } = $h2_msgs_nb ;

( $debug or $debugLIST ) and print "Host2 LIST: $h2_msgs_nb messages [@h2_msgs]\n";
    $debug and print "Host2 selecting messages of folder [$h2_fold] took ", timenext(), " s\n";

my $cache_base = "$tmpdir/imapsync_cache/" ;
my $cache_dir = cache_folder( $cache_base, "$host1/$user1/$host2/$user2", $h1_fold, $h2_fold ) ;
my ( $cache_1_2_ref, $cache_2_1_ref ) = ( {}, {} ) ;

my $h1_uidvalidity = $imap1->uidvalidity(  ) || '' ;
my $h2_uidvalidity = $imap2->uidvalidity(  ) || '' ;

    last FOLDER if $imap1->IsUnconnected() ;
    last FOLDER if $imap2->IsUnconnected() ;

if ( $usecache ) {
    print "cache directory: $cache_dir\n" ;
    mkpath( "$cache_dir" ) ;
    ( $cache_1_2_ref, $cache_2_1_ref ) 
            = get_cache( $cache_dir, \@h1_msgs, \@h2_msgs, $h1_msgs_all_hash_ref, $h2_msgs_all_hash_ref ) ;
    print "CACHE h1 h2: ", scalar( keys %$cache_1_2_ref ), " files\n" ; 
    $debug and print '[',
        map ( { "$_->$cache_1_2_ref->{$_} " } keys %$cache_1_2_ref ), " ]\n";
}

my %h1_hash = () ;
my %h2_hash = () ;

my ( %h1_msgs, %h2_msgs ) ;
@h1_msgs{ @h1_msgs } = ();
@h2_msgs{ @h2_msgs } = ();

my @h1_msgs_in_cache = sort { $a <=> $b } keys %$cache_1_2_ref ;
my @h2_msgs_in_cache = keys %$cache_2_1_ref ;

my ( %h1_msgs_not_in_cache, %h2_msgs_not_in_cache ) ;
%h1_msgs_not_in_cache = %h1_msgs ;
%h2_msgs_not_in_cache = %h2_msgs ;
delete @h1_msgs_not_in_cache{ @h1_msgs_in_cache } ;
delete @h2_msgs_not_in_cache{ @h2_msgs_in_cache } ;

my @h1_msgs_not_in_cache = keys %h1_msgs_not_in_cache ;
#print "h1_msgs_not_in_cache: [@h1_msgs_not_in_cache]\n" ;
my @h2_msgs_not_in_cache = keys %h2_msgs_not_in_cache ;

my @h2_msgs_delete2_not_in_cache = () ;
%h1_msgs_copy_by_uid = (  ) ;

if ( $useuid ) {
    # use uid so we have to avoid getting header
    @h1_msgs_copy_by_uid{ @h1_msgs_not_in_cache } = (  ) ;
    @h2_msgs_delete2_not_in_cache = @h2_msgs_not_in_cache if $usecache ;
    @h1_msgs_not_in_cache = (  ) ;
    @h2_msgs_not_in_cache = (  ) ;

    #print "delete2: @h2_msgs_delete2_not_in_cache\n";
}

$debug and print "Host1 parsing headers of folder [$h1_fold]\n";

my ($h1_heads_ref, $h1_fir_ref) = ({}, {});
$h1_heads_ref = $imap1->parse_headers([@h1_msgs_not_in_cache], @useheader) if (@h1_msgs_not_in_cache);
$debug and print "Host1 parsing headers of folder [$h1_fold] took ", timenext(), " s\n";

@$h1_fir_ref{@h1_msgs} = (undef);

$debug and print "Host1 getting flags idate and sizes of folder [$h1_fold]\n" ;
    if ( $abletosearch ) {
    $h1_fir_ref = $imap1->fetch_hash( \@h1_msgs, "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h1_fir_ref )
    if ( @h1_msgs ) ;
    }else{
    my $uidnext = $imap1->uidnext( $h1_fold ) || $uidnext_default ;
    $h1_fir_ref = $imap1->fetch_hash( "1:$uidnext", "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h1_fir_ref )
    if ( @h1_msgs ) ;
    }
$debug and print "Host1 getting flags idate and sizes of folder [$h1_fold] took ", timenext(), " s\n";
unless ($h1_fir_ref) {
    print
    "Host1 folder $h1_fold: Could not fetch_hash ",
    scalar(@h1_msgs), " msgs: ", $imap1->LastError || '', "\n";
    $nb_errors++;
    next FOLDER;
}

my @h1_msgs_duplicate;
foreach my $m (@h1_msgs_not_in_cache) {
    my $rc = parse_header_msg($imap1, $m, $h1_heads_ref, $h1_fir_ref, 'Host1', \%h1_hash);
    if (! defined($rc)) {
        my $h1_size = $h1_fir_ref->{$m}->{"RFC822.SIZE"} || 0;
        print "Host1 $h1_fold/$m size $h1_size ignored (no wanted headers so we ignore this message. To solve this: use --addheader)\n" ;
        $total_bytes_skipped += $h1_size;
        $nb_msg_skipped += 1;
        $h1_nb_msg_noheader +=1;
                    $h1_nb_msg_processed +=1 ;
    } elsif(0 == $rc) {
        # duplicate
        push(@h1_msgs_duplicate, $m);
        # duplicate, same id same size?
        my $h1_size = $h1_fir_ref->{$m}->{"RFC822.SIZE"} || 0;
        $nb_msg_skipped += 1;
        $h1_total_bytes_duplicate += $h1_size;
        $h1_nb_msg_duplicate += 1;
                    $h1_nb_msg_processed +=1 ;
    }
}
    my $h1_msgs_duplicate_nb = scalar( @h1_msgs_duplicate ) ;
    $h1{ $h1_fold }{ 'duplicates_nb' } = $h1_msgs_duplicate_nb ;

    $debug and print "Host1 selected: $h1_msgs_nb  duplicates: $h1_msgs_duplicate_nb\n" ;
$debug and print "Host1 whole time parsing headers took ", timenext(), " s\n";

$debug and print "Host2 parsing headers of folder [$h2_fold]\n";

my ($h2_heads_ref, $h2_fir_ref) = ( {}, {} );
$h2_heads_ref =   $imap2->parse_headers([@h2_msgs_not_in_cache], @useheader) if (@h2_msgs_not_in_cache);
$debug and print "Host2 parsing headers of folder [$h2_fold] took ", timenext(), " s\n" ;

$debug and print "Host2 getting flags idate and sizes of folder [$h2_fold]\n" ;
@$h2_fir_ref{@h2_msgs} = (  ); # fetch_hash can select by uid with last arg as ref


    if ( $abletosearch ) {
    $h2_fir_ref = $imap2->fetch_hash( \@h2_msgs, "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h2_fir_ref)
    if (@h2_msgs) ;
    }else{
    my $uidnext = $imap2->uidnext( $h2_fold ) || $uidnext_default ;
    $h2_fir_ref = $imap2->fetch_hash( "1:$uidnext", "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h2_fir_ref )
    if ( @h2_msgs ) ;
    }

$debug and print "Host2 getting flags idate and sizes of folder [$h2_fold] took ", timenext(), " s\n" ;

my @h2_msgs_duplicate;
foreach my $m (@h2_msgs_not_in_cache) {
    my $rc = parse_header_msg($imap2, $m, $h2_heads_ref, $h2_fir_ref, 'Host2', \%h2_hash);
    my $h2_size = $h2_fir_ref->{$m}->{"RFC822.SIZE"} || 0;
    if (! defined($rc)) {
                    print "Host2 $h2_fold/$m size $h2_size ignored (no wanted headers so we ignore this message)\n" ;
        $h2_nb_msg_noheader += 1 ;
    } elsif(0 == $rc) {
        # duplicate
        $h2_nb_msg_duplicate += 1;
        $h2_total_bytes_duplicate += $h2_size;
        push(@h2_msgs_duplicate, $m);
    }
}

    # %h2_folders_of_md5
    foreach my $md5 (  keys( %h2_hash ) ) {
        $h2_folders_of_md5{ $md5 }->{ $h2_fold } ++ ;
    }


    my $h2_msgs_duplicate_nb = scalar( @h2_msgs_duplicate ) ;
    $h2{ $h2_fold }{ 'duplicates_nb' } = $h2_msgs_duplicate_nb ;

    print "Host2 folder $h2_fold selected: $h2_msgs_nb messages,  duplicates: $h2_msgs_duplicate_nb\n" 
        if ( $debug or $delete2duplicates or $h2_msgs_duplicate_nb ) ;
$debug and print "Host2 whole time parsing headers took ", timenext(), " s\n";

$debug and print "++++ Verifying [$h1_fold] -> [$h2_fold]\n";
# messages in host1 that are not in host2

my @h1_hash_keys_sorted_by_uid 
  = sort {$h1_hash{$a}{'m'} <=> $h1_hash{$b}{'m'}} keys(%h1_hash);

#print map { $h1_hash{$_}{'m'} . " "} @h1_hash_keys_sorted_by_uid;

my @h2_hash_keys_sorted_by_uid 
  = sort {$h2_hash{$a}{'m'} <=> $h2_hash{$b}{'m'}} keys(%h2_hash);


if( $delete2duplicates and not exists( $h2_folders_from_1_several{ $h2_fold } ) ) {
    my @h2_expunge ;

    foreach my $h2_msg ( @h2_msgs_duplicate ) {
        print "msg $h2_fold/$h2_msg marked \\Deleted [duplicate] on host2 $dry_message\n" ;
        push( @h2_expunge, $h2_msg ) if $uidexpunge2 ;
        unless ( $dry ) {
            $imap2->delete_message( $h2_msg ) ;
            $h2_nb_msg_deleted += 1 ;
        }
    }
    my $cnt = scalar @h2_expunge ;
    if( @h2_expunge ) {
        print "uidexpunge $cnt message(s) $dry_message\n" ;
        $imap2->uidexpunge( \@h2_expunge ) if ! $dry ;
    }
        if ( $expunge2 ){
                print "Expunging host2 folder $h2_fold $dry_message\n" ;
                $imap2->expunge(  ) if ! $dry ;
        }
}

if( $delete2 and not exists( $h2_folders_from_1_several{ $h2_fold } ) ) {
        # No host1 folders f1a f1b ... going all to same f2 (via --regextrans2)
    my @h2_expunge;
    foreach my $m_id (@h2_hash_keys_sorted_by_uid) {
        #print "$m_id ";
        unless (exists($h1_hash{$m_id})) {
            my $h2_msg  = $h2_hash{$m_id}{'m'};
            my $h2_flags  = $h2_hash{$m_id}{'F'} || "";
            my $isdel  = $h2_flags =~ /\B\\Deleted\b/x ? 1 : 0;
            print "msg $h2_fold/$h2_msg marked \\Deleted on host2 [$m_id] $dry_message\n"
              if ! $isdel;
            push(@h2_expunge, $h2_msg) if $uidexpunge2;
            unless ($dry or $isdel) {
                $imap2->delete_message($h2_msg);
                $h2_nb_msg_deleted += 1;
            }
        }
    }
    foreach my $h2_msg ( @h2_msgs_delete2_not_in_cache ) {
        print "msg $h2_fold/$h2_msg marked \\Deleted [not in cache] on host2 $dry_message\n";
                    push(@h2_expunge, $h2_msg) if $uidexpunge2;
        unless ($dry) {
            $imap2->delete_message($h2_msg);
            $h2_nb_msg_deleted += 1;
        }
    }
    my $cnt = scalar @h2_expunge ;
    if( @h2_expunge ) {
        print "uidexpunge $cnt message(s) $dry_message\n" ;
        $imap2->uidexpunge( \@h2_expunge ) if ! $dry ;
    }
        if ($expunge2){
                print "Expunging host2 folder $h2_fold $dry_message\n" ;
                $imap2->expunge(  ) if ! $dry ;
        }
}

if( $delete2 and exists( $h2_folders_from_1_several{ $h2_fold } ) ) {
        print "Host2 folder $h2_fold $h2_folders_from_1_several{ $h2_fold } folders left to sync there\n" ;
    my @h2_expunge;
    foreach my $m_id ( @h2_hash_keys_sorted_by_uid ) {
                my $h2_msg  = $h2_hash{ $m_id }{ 'm' } ;
        unless ( exists( $h1_hash{ $m_id } ) ) {
            my $h2_flags  = $h2_hash{ $m_id }{ 'F' } || "" ;
            my $isdel  = $h2_flags =~ /\B\\Deleted\b/x ? 1 : 0 ;
            unless ( $isdel ) {
                                $debug and print "msg $h2_fold/$h2_msg candidate for deletion on host2 [$m_id]\n" ;
                $uid_candidate_for_deletion{ $h2_fold }{ $h2_msg }++ ;
            }
        }else{
                        $debug and print "msg $h2_fold/$h2_msg will cancel deletion on host2 [$m_id]\n" ;
                        $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
                    }
    }
    foreach my $h2_msg ( @h2_msgs_delete2_not_in_cache ) {
        print "msg $h2_fold/$h2_msg candidate for deletion [not in cache] on host2\n";
                    $uid_candidate_for_deletion{ $h2_fold }{ $h2_msg }++ ;
    }

    foreach my $h2_msg ( @h2_msgs_in_cache ) {
        print "msg $h2_fold/$h2_msg will cancel deletion [in cache] on host2\n";
                    $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
    }


            if ( 0 == $h2_folders_from_1_several{ $h2_fold } ) {
                # last host1 folder going to $h2_fold
                    print "Last host1 folder going to $h2_fold\n" ;
                    foreach my $h2_msg ( keys %{ $uid_candidate_for_deletion{ $h2_fold } } ) {
                        $debug and print "msg $h2_fold/$h2_msg candidate for deletion on host2\n" ;
                            if ( exists( $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg } ) ) {
                                $debug and print "msg $h2_fold/$h2_msg canceled deletion on host2\n" ;
                            }else{
                                print "msg $h2_fold/$h2_msg marked \\Deleted on host2 $dry_message\n";
                                    push( @h2_expunge, $h2_msg ) if $uidexpunge2 ;
                                    unless ( $dry ) {
                                        $imap2->delete_message( $h2_msg ) ;
                                        $h2_nb_msg_deleted += 1 ;
                                    }
                            }
                    }
            }

    my $cnt = scalar @h2_expunge ;
    if( @h2_expunge ) {
        print "uidexpunge $cnt message(s) $dry_message\n" ;
        $imap2->uidexpunge( \@h2_expunge ) if ! $dry ;
    }
        if ( $expunge2 ) {
                print "Expunging host2 folder $h2_fold $dry_message\n" ;
                $imap2->expunge(  ) if ! $dry ;
        }

            $h2_folders_from_1_several{ $h2_fold }-- ;                
}


my $h2_uidnext = $imap2->uidnext( $h2_fold ) ;
    $debug and print "Host2 uidnext: $h2_uidnext\n" ;
$h2_uidguess = $h2_uidnext ;
MESS: foreach my $m_id (@h1_hash_keys_sorted_by_uid) {
        last FOLDER if $imap1->IsUnconnected() ;
            last FOLDER if $imap2->IsUnconnected() ;

    #print "h1_nb_msg_processed: $h1_nb_msg_processed\n" ;
    my $h1_size  = $h1_hash{$m_id}{'s'};
    my $h1_msg   = $h1_hash{$m_id}{'m'};
    my $h1_idate = $h1_hash{$m_id}{'D'};

    if ( ( not exists( $h2_hash{ $m_id } ) ) 
                and ( not exists( $h2_folders_of_md5{ $m_id } )
                          or not $skipcrossduplicates ) ) {
        # copy
        my $h2_msg = copy_message( $h1_msg, $h1_fold, $h2_fold, $h1_fir_ref, $permanentflags2, $cache_dir ) ;
                    $h2_folders_of_md5{ $m_id }->{ $h2_fold } ++ ;
                    if( $delete2 and exists( $h2_folders_from_1_several{ $h2_fold } ) and $h2_msg ) {
                        print "msg $h2_fold/$h2_msg will cancel deletion [fresh copy] on host2\n" ;
                        $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
                    }
                    last FOLDER if total_bytes_max_reached(  ) ;
        next MESS;
    }
    else{
            # already on host2
                    if ( exists( $h2_hash{ $m_id } ) ) {
            my $h2_msg   = $h2_hash{$m_id}{'m'} ;
            $debug and print "Host1 found msg $h1_fold/$h1_msg equals Host2 $h2_fold/$h2_msg\n" ;
                            if ( $usecache ) {
                                $debugcache and print "touch $cache_dir/${h1_msg}_$h2_msg\n" ;
                                touch( "$cache_dir/${h1_msg}_$h2_msg" ) 
                                    or croak( "Couldn't touch $cache_dir/${h1_msg}_$h2_msg" ) ;
                            }
                    }elsif( exists( $h2_folders_of_md5{ $m_id } ) ) {
                        my @folders_dup = keys( %{ $h2_folders_of_md5{ $m_id } } ) ;
                        ( $debug or $debugcrossduplicates ) and print "Host1 found msg $h1_fold/$h1_msg is also in Host2 folders @folders_dup\n" ;
                    }
        $total_bytes_skipped += $h1_size ;
        $nb_msg_skipped += 1 ;
                    $h1_nb_msg_processed +=1 ;
            }

            if ( exists( $h2_hash{ $m_id } ) ) {
        #$debug and print "MESSAGE $m_id\n";
        my $h2_msg  = $h2_hash{$m_id}{'m'};

                sync_flags_fir( $h1_fold, $h1_msg, $h2_fold, $h2_msg, $permanentflags2, $h1_fir_ref, $h2_fir_ref ) ;
            # Good
        my $h2_size = $h2_hash{$m_id}{'s'};
        $debug and print
        "Host1 size  msg $h1_fold/$h1_msg = $h1_size <> $h2_size = Host2 $h2_fold/$h2_msg\n";
    }
            last FOLDER if $imap2->IsUnconnected() ;

    if( $delete ) {
                    my $expunge_message = '' ;
                    $expunge_message = "and expunged" if ( $expungeaftereach and ( $expunge or $expunge1 ) ) ;
        print "Host1 msg $h1_fold/$h1_msg marked deleted $expunge_message $dry_message\n" ;
        unless( $dry ) {
            $imap1->delete_message( $h1_msg ) ;
            $h1_nb_msg_deleted += 1 ;
            $imap1->expunge() if ( $expungeaftereach and ( $expunge or $expunge1 ) ) ;
        }
    }
}
# END MESS: loop
    last FOLDER if $imap1->IsUnconnected();
    last FOLDER if $imap2->IsUnconnected();
MESS_IN_CACHE: foreach my $h1_msg ( @h1_msgs_in_cache ) {
    my $h2_msg = $cache_1_2_ref->{ $h1_msg } ;
    $debugcache and print "cache messages update flags $h1_msg->$h2_msg\n";
    sync_flags_fir( $h1_fold, $h1_msg, $h2_fold, $h2_msg, $permanentflags2, $h1_fir_ref, $h2_fir_ref ) ;
    my $h1_size = $h1_fir_ref->{ $h1_msg }->{ 'RFC822.SIZE' } || 0 ;
    $total_bytes_skipped += $h1_size;
    $nb_msg_skipped += 1;
            $h1_nb_msg_processed +=1 ;
            last FOLDER if $imap2->IsUnconnected();                
}

#print "Messages by uid: ", map { "$_ " } keys %h1_msgs_copy_by_uid, "\n" ;
MESS_BY_UID: foreach my $h1_msg ( sort { $a <=> $b } keys %h1_msgs_copy_by_uid ) {
    # 
    $debug and print "Copy by uid $h1_fold/$h1_msg\n" ;
            last FOLDER if $imap1->IsUnconnected();
            last FOLDER if $imap2->IsUnconnected();
    my $h2_msg = copy_message( $h1_msg, $h1_fold, $h2_fold, $h1_fir_ref, $permanentflags2, $cache_dir ) ;
            if( $delete2 and exists( $h2_folders_from_1_several{ $h2_fold } ) and $h2_msg ) {
                print "msg $h2_fold/$h2_msg will cancel deletion [fresh copy] on host2\n" ;
                $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
            }
    last FOLDER if total_bytes_max_reached(  ) ;
}

if ($expunge or $expunge1){
    print "Expunging host1 folder $h1_fold $dry_message\n";
    unless($dry) { $imap1->expunge() };
}
if ($expunge2){
    print "Expunging host2 folder $h2_fold $dry_message\n";
    unless($dry) { $imap2->expunge() };
}

$debug and print "Time: ", timenext(), " s\n";

}

sub total_bytes_max_reached {

return( 0 ) if not $exitwhenover ;
if ( $total_bytes_transferred >= $exitwhenover ) {
        print "Maximum bytes transfered reached, $total_bytes_transferred >= $exitwhenover, ending sync\n" ;
        return( 1 ) ;
    }  

}

print "++++ End looping on each folder\n";
$debug and print "Time: ", timenext(), " s\n";

#print memory_consumption();

if ( $foldersizesatend ) {
timenext() ;
# Get all folders on host2 again since new were created
@h2_folders_all = sort $imap2->folders();
for ( @h2_folders_all ) { $h2_folders_all{ $_ } = 1 } ;
( $h1_nb_msg_end, $h1_bytes_end ) = foldersizes( "Host1", $imap1, $search1, @h1_folders_wanted ) ;
( $h2_nb_msg_end, $h2_bytes_end ) = foldersizes( "Host2", $imap2, $search2, @h2_folders_from_1_wanted ) ;
}

$imap1->logout( ) unless lost_connection($imap1, "for host1 [$host1]");
$imap2->logout( ) unless lost_connection($imap2, "for host2 [$host2]");

stats( ) ;
exit_clean( 1 ) if ( $nb_errors ) ;
exit_clean( 0 ) ;

END of main program

subroutines

sub size_filtered_flag {
my $h1_size = shift ;

if (defined $maxsize and $h1_size >= $maxsize) {
    return( 1 ) ;
}
if (defined $minsize and $h1_size <= $minsize) {
    return( 1 ) ;
}
return( 0 ) ;

}

sub sync_flags_fir {
my ( $h1_fold, $h1_msg, $h2_fold, $h2_msg, $permanentflags2, $h1_fir_ref, $h2_fir_ref ) = @_ ;

if ( not defined( $h1_msg ) ) { return(  ) } ;
if ( not defined( $h2_msg ) ) { return(  ) } ;

my $h1_size = $h1_fir_ref->{$h1_msg}->{"RFC822.SIZE"} ;
return(  ) if size_filtered_flag( $h1_size ) ;

# used cached flag values for efficiency
my $h1_flags = $h1_fir_ref->{ $h1_msg }->{ "FLAGS" } || '' ;
my $h2_flags = $h2_fir_ref->{ $h2_msg }->{ "FLAGS" } || '' ;

sync_flags( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $h2_flags, $permanentflags2 ) ;

    return(  ) ;

}

sub sync_flags_after_copy {
my( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $permanentflags2 ) = @_ ;

    my @h2_flags = $imap2->flags( $h2_msg ) ;
    my $h2_flags = "@h2_flags" ;
    ( $debug or $debugflags ) and print "Host2 flags before resync by STORE on msg $h2_msg: $h2_flags\n" ;
sync_flags( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $h2_flags, $permanentflags2 ) ;
    return(  ) ;

}

sub sync_flags {
my( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $h2_flags, $permanentflags2 ) = @_ ;

( $debug or $debugflags ) and 
    print "Host1 flags init msg $h1_fold/$h1_msg flags( $h1_flags ) Host2 $h2_fold/$h2_msg flags( $h2_flags )\n" ;

$h1_flags = flags_for_host2( $h1_flags, $permanentflags2 ) ;

$h2_flags = flagsCase( $h2_flags ) ;

( $debug or $debugflags ) and 
    print "Host1 flags filt msg $h1_fold/$h1_msg flags( $h1_flags ) Host2 $h2_fold/$h2_msg flags( $h2_flags )\n" ;


# compare flags - set flags if there a difference
my @h1_flags = sort split(' ', $h1_flags );
my @h2_flags = sort split(' ', $h2_flags );
my $diff = compare_lists( \@h1_flags, \@h2_flags );

$diff and ( $debug or $debugflags ) 
    and     print "Host2 flags msg $h2_fold/$h2_msg replacing h2 flags( $h2_flags ) with h1 flags( $h1_flags )\n";
# This sets flags so flags can be removed with this
# When you remove a \Seen flag on host1 you want to it
# to be removed on host2. Just add flags is not what 
# we need most of the time.

if ( not $dry and $diff and not $imap2->store( $h2_msg, "FLAGS.SILENT (@h1_flags)" ) ) {
    print "Host2 flags msg $h2_fold/$h2_msg could not add flags [@h1_flags]: ",
      $imap2->LastError || '', "\n" ;
    #$nb_errors++ ;
}

    return(  ) ;

}

sub _filter {
my $str = shift or return "";
my $sz = 64;
my $len = length($str);
if ( not $debug and $len > $sz*2 ) {
my $beg = substr($str, 0, $sz);
my $end = substr($str, -$sz, $sz);
$str = $beg . "..." . $end;
}
$str =~ s/\012?\015$//x;
return "(len=$len) " . $str;
}

sub lost_connection {
my($imap, $error_message) = @_;
if ( $imap->IsUnconnected() ) {
$nb_errors++;
my $lcomm = $imap->LastIMAPCommand || "";
my $einfo = $imap->LastError || @{$imap->History}[-1] || "";

            # if string is long try reduce to a more reasonable size
            $lcomm = _filter($lcomm);
            $einfo = _filter($einfo);
            print("Failure: last command: $lcomm\n") if ($debug && $lcomm);
            print("Failure: lost connection $error_message: ", $einfo, "\n");
            return(1);
    }
    else{
        return(0);
    }

}

sub max {
my @list = @_ ;
return( undef ) if ( 0 == scalar( @list ) ) ;
my @sorted = sort { $a <=> $b } @list ;
return( pop( @sorted ) ) ;
}

sub tests_max {
ok( 0 == max(0), "max 0");
ok( 1 == max(1), "max 1");
ok( -1 == max(-1), "max -1");
ok( not ( defined( max( ) ) ), "max no arg" ) ;
ok( 100 == max( 1, 100 ), "max 1 100" ) ;
ok( 100 == max( 100, 1 ), "max 100 1") ;
ok( 100 == max( 100, 42, 1 ), "max 100 42 1") ;
ok( 100 == max( 100, "42", 1 ), "max 100 42 1") ;
ok( 100 == max( "100", "42", 1 ), "max 100 42 1") ;
#ok( 100 == max( 100, "haha", 1 ), "max 100 42 1") ;
return( ) ;
}

sub keyval {
my %hash = @_ ;
return( join( " ", map( { "$_ => " . $hash{ $_ } } keys %hash ) ) . "\n" ) ;
}

sub check_lib_version {
$debug and print "IMAPClient $Mail::IMAPClient::VERSION\n";
if ($Mail::IMAPClient::VERSION eq '2.2.9') {
print "imapsync no longer supports Mail::IMAPClient 2.2.9, upgrade it" ;
return( 0 ) ;
}
else{
# 3.x.x is no longer buggy with imapsync.
return( 1 ) ;
}
}

sub module_version_str {
my( $module_name, $module_version ) = @_ ;
my $str = sprintf( "%-20s %s\n", $module_name, $module_version ) ;
return( $str ) ;
}

sub modules_VERSION {

my @list_version;

my $v ;
eval { require Mail::IMAPClient; $v = $Mail::IMAPClient::VERSION } or $v = "?" ;
push ( @list_version, module_version_str( 'Mail::IMAPClient', $v ) ) ;

eval { require IO::Socket; $v = $IO::Socket::VERSION } or $v = "?" ;
push ( @list_version, module_version_str( 'IO::Socket', $v ) ) ;

eval { require IO::Socket::IP; $v = $IO::Socket::IP::VERSION } or $v = "?"
Contributor

gilleslamiral commented Jan 24, 2014

Dear Dennis,

I fixed this ugly bug in imapsync 1.582
It is hard to go in the same conditions as yours.
Now you fixed it on your side I supposed this won't interest you now.
I'll make it officially public later.
1.582 attached anyway.

On 22/01/2014 00:30, Dennis Schridde wrote:

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Yes, the CA cert I added to /usr/local/share/ca-certificates was old and expired (i.e. not the one that signed the server's cert), so OpenSSL considered the chain untrusted. Rightly so, as there was a self-signed certificate in it. After I fixed that, imapsync worked like a charm.

—
Reply to this email directly or view it on GitHub #15 (comment).

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06
#!/usr/bin/perl

structure

pod documentation

pragmas

main program

global variables initialisation

get_options( ) ;

default values

folder loop

subroutines

IMAPClient 3.xx ads

pod documentation

=pod

=head1 NAME

imapsync - IMAP synchronisation, sync, copy or migration tool.
Synchronises mailboxes between two imap servers.
Good at IMAP migration. More than 52 different IMAP server softwares
supported with success, few failures.

$Revision: 1.582 $

=head1 SYNOPSIS

To synchronize imap account "foo" on "imap.truc.org"
to imap account "bar" on "imap.trac.org"
with foo password "secret1"
and bar password "secret2":

imapsync
--host1 imap.truc.org --user1 foo --password1 secret1
--host2 imap.trac.org --user2 bar --password2 secret2

=head1 INSTALL

imapsync works fine under any Unix OS with perl.
imapsync works fine under Windows (2000, XP, Vista, Seven)
with Strawberry Perl (5.10, 5.12 or higher)
or as a standalone binary software imapsync.exe

imapsync can be available directly on the following distributions:
FreeBSD, Debian, Ubuntu, Gentoo, Fedora,
NetBSD, Darwin, Mandriva and OpenBSD.

Purchase latest imapsync at
http://imapsync.lamiral.info/

You'll receive a link to a compressed tarball called imapsync-x.xx.tgz
where x.xx is the version number. Untar the tarball where
you want (on Unix):

tar xzvf imapsync-x.xx.tgz

Go into the directory imapsync-x.xx and read the INSTALL file.
The INSTALL file is also at
http://imapsync.lamiral.info/INSTALL

The freecode (was freshmeat) record is at
http://freecode.com/projects/imapsync

=head1 USAGE

imapsync [options]

To get a description of each option just run imapsync like this:

imapsync --help
imapsync

The option list:

imapsync [--host1 server1] [--port1 ]
[--user1 ] [--passfile1 ]
[--host2 server2] [--port2 ]
[--user2 ] [--passfile2 ]
[--ssl1] [--ssl2]
[--tls1] [--tls2]
[--authmech1 ] [--authmech2 ]
[--proxyauth1] [--proxyauth2]
[--domain1] [--domain2]
[--authmd51] [--authmd52]
[--folder --folder ...]
[--folderrec --folderrec ...]
[--include ] [--exclude ]
[--prefix2 ] [--prefix1 ]
[--regextrans2 --regextrans2 ...]
[--sep1 ]
[--sep2 ]
[--justfolders] [--justfoldersizes] [--justconnect] [--justbanner]
[--syncinternaldates]
[--idatefromheader]
[--syncacls]
[--regexmess ] [--regexmess ]
[--maxsize ]
[--minsize ]
[--maxage ]
[--minage ]
[--search ]
[--search1 ]
[--search2 ]
[--skipheader ]
[--useheader ] [--useheader ]
[--nouid1] [--nouid2]
[--usecache]
[--skipsize] [--allowsizemismatch]
[--delete] [--delete2]
[--expunge] [--expunge1] [--expunge2] [--uidexpunge2]
[--delete2folders] [--delete2foldersonly] [--delete2foldersbutnot]
[--subscribed] [--subscribe] [--subscribe_all]
[--nofoldersizes] [--nofoldersizesatend]
[--dry]
[--debug] [--debugimap][--debugimap1][--debugimap2]
[--timeout ]
[--split1] [--split2]
[--reconnectretry1 ] [--reconnectretry2 ]
[--noreleasecheck]
[--pidfile ]
[--tmpdir ]
[--version] [--help]
[--tests] [--tests_debug]

=cut

comment

=pod

=head1 DESCRIPTION

The command imapsync is a tool allowing incremental and
recursive imap transfer from one mailbox to another.

By default all folders are transferred, recursively, all
possible flags (\Seen \Answered \Flagged etc.) are synced too.

We sometimes need to transfer mailboxes from one imap server to
another. This is called migration.

imapsync is a good tool because it reduces the amount
of data transferred by not transferring a given message
if it is already on both sides. Same headers
and the transfer is done only once. All flags are
preserved, unread will stay unread, read will stay read,
deleted will stay deleted. You can stop the transfer at any
time and restart it later, imapsync works well with bad
connections.

You can decide to delete the messages from the source mailbox
after a successful transfer, it can be a good feature when migrating
live mailboxes since messages will be only on one side.
In that case, use the --delete option. Option --delete implies
also option --expunge so all messages marked deleted on host1
will be really deleted.
(you can use --noexpunge to avoid this but I don't see any
good real world scenario for the combinaison --delete --noexpunge).

You can also just synchronize a mailbox B from another mailbox A
in case you just want to keep a "live" copy of A in B.
In that case --delete2 can be used, it deletes messages in host2
folder B that are not in host1 folder A.

imapsync is not adequate for maintaining two active imap accounts
in synchronization where the user plays independently on both sides.
Use offlineimap (written by John Goerzen) or mbsync (written by
Michael R. Elkins) for 2 ways synchronizations.

=head1 OPTIONS

To get a description of each option just invoke:

imapsync --help

=head1 HISTORY

I wrote imapsync because an enterprise (basystemes) paid me to install
a new imap server without losing huge old mailboxes located on a far
away remote imap server accessible by a low bandwidth link. The tool
imapcp (written in python) could not help me because I had to verify
every mailbox was well transferred and delete it after a good
transfer. imapsync started life as a copy_folder.pl patch.
The tool copy_folder.pl comes from the Mail-IMAPClient-2.1.3 perl
module tarball source (in the examples/ directory of the tarball).

=head1 EXAMPLE

While working on imapsync parameters please run imapsync in
dry mode (no modification induced) with the --dry
option. Nothing bad can be done this way.

To synchronize the imap account "buddy" (with password "secret1")
on host "imap.src.fr" to the imap account "max" (with password "secret2")
on host "imap.dest.fr":

imapsync --host1 imap.src.fr --user1 buddy --password1 secret1
--host2 imap.dest.fr --user2 max --password2 secret2

Then you will have max's mailbox updated from buddy's
mailbox.

=head1 SECURITY

You can use --passfile1 instead of --password1 to give the
password since it is safer. With --password1 option any user
on your host can see the password by using the 'ps auxwwww'
command. Using a variable (like $PASSWORD1) is also
dangerous because of the 'ps auxwwwwe' command. So, saving
the password in a well protected file (600 or rw-------) is
the best solution.

imasync is not totally protected against sniffers on the
network since passwords may be transferred in plain text
if CRAM-MD5 is not supported by your imap servers. Use
--ssl1 (or --tls1) and --ssl2 (or --tls2) to enable
encryption on host1 and host2.

You may authenticate as one user (typically an admin user),
but be authorized as someone else, which means you don't
need to know every user's personal password. Specify
--authuser1 "adminuser" to enable this on host1. In this
case, --authmech1 PLAIN will be used by default since it
is the only way to go for now. So don't use --authmech1 SOMETHING
with --authuser1 "adminuser", it will not work.
Same behavior with the --authuser2 option.
Authenticate with an admin account must be supported by your
imap server to work with imapsync.

When working on Sun/iPlanet/Netscape IMAP servers you must use
--proxyauth1 to enable administrative user to masquerade as another user.
Can also be used on destination server with --proxyauth2

You can authenticate with OAUTH when transfering from Google Apps.
The consumer key will be the domain part of the --user, and the
--password will be used as the consumer secret. It does not work
with Google Apps free edition.

=head1 EXIT STATUS

imapsync will exit with a 0 status (return code) if everything went good.
Otherwise, it exits with a non-zero status.

So if you have an unreliable internet connection, you can use this loop
in a Bourne shell:

    while ! imapsync ...; do 
          echo imapsync not complete
    done

=head1 LICENSE

imapsync is free, open, public but not always gratis software
cover by the NOLIMIT Public License.
See the LICENSE file included in the distribution or just read this
simple sentence as it is the licence text:
No limit to do anything with this work and this license.

=head1 MAILING-LIST

The public mailing-list may be the best way to get free support.

To write on the mailing-list, the address is:
imapsync@linux-france.org

To subscribe, send any message (even empty) to:
imapsync-subscribe@listes.linux-france.org
then just reply to the confirmation message.

To unsubscribe, send a message to:
imapsync-unsubscribe@listes.linux-france.org

To contact the person in charge for the list:
imapsync-request@listes.linux-france.org

The list archives are available at:
http://www.linux-france.org/prj/imapsync_list/
So consider that the list is public, anyone
can see your post. Use a pseudonym or do not
post to this list if you want to stay private.

Thank you for your participation.

=head1 AUTHOR

Gilles LAMIRAL gilles.lamiral@laposte.net

Feedback good or bad is very often welcome.

Gilles LAMIRAL earns his living by writing, installing,
configuring and teaching free, open and often gratis
softwares. It used to be "always gratis" but now it is
"often" because imapsync is sold by its author, a good
way to stay maintening and supporting free open public
softwares (see the license) over decades.

=head1 BUG REPORT GUIDELINES

Help us to help you: follow the following guidelines.

Report any bugs or feature requests to the public mailing-list
or to the author.

Before reporting bugs, read the FAQ, the README and the
TODO files. http://imapsync.lamiral.info/

Upgrade to last imapsync release, maybe the bug
is already fixed.

Upgrade to last Mail-IMAPClient Perl module.
http://search.cpan.org/dist/Mail-IMAPClient/
maybe the bug is already fixed.

Make a good title with word "imapsync" in it (my spam filter won't filter it),
Don't write an email title with just "imapsync" or "problem",
a good title is made of keywords summary, not too long (one visible line).

Don't write imapsync in uppercase in the email title, I'll
then know you run Windows and you haven't read this README yet.

Help us to help you: in your report, please include:

  • imapsync version.

  • output given with --debug --debugimap near the failure point.
    Isolate a message or two in a folder 'BUG' and use

    imapsync ... --folder 'BUG' --debug --debugimap

  • imap server software on both side and their version number.

  • imapsync with all the options you use, the full command line
    you use (except the passwords of course).

  • IMAPClient.pm version.

  • the run context. Do you run imapsync.exe, a unix binary
    or the perl script imapsync.

  • operating system running imapsync.

  • virtual software context (vmware, xen etc.)

  • operating systems on both sides and the third side in case
    you run imapsync on a foreign host from the both.

Most of those values can be found as a copy/paste at the begining of the output,
so a copy of the output is a very easy and very good debug report for me.

One time in your life, read the paper
"How To Ask Questions The Smart Way"
http://www.catb.org/~esr/faqs/smart-questions.html
and then forget it.

=head1 IMAP SERVERS

Failure stories reported with the following 3 imap servers:

  • MailEnable 1.54 (Proprietary) but MailEnable 4.23 is supported.
  • DBMail 0.9, 2.0.7 (GPL). But DBMail 1.2.1 is supported.
    Patient and confident testers are welcome.
  • Imail 7.04 (maybe).
  • (2011) MDaemon 12.0.3 as host2 but MDaemon is supported as host1.
    MDaemon is simply buggy with the APPEND IMAP command with
    any IMAP email client.
  • Hotmail since hotmail.com does not provide IMAP access
  • Outlook.com since outlook.com does not provide IMAP access

Success stories reported with the following 57 imap servers
(software names are in alphabetic order):

  • 1und1 H mimap1 84498 [host1] H mibap4 95231 [host1]
  • a1.net imap.a1.net IMAP4 Ready [host1]
  • Apple Server 10.6 Snow Leopard [host1]
  • Archiveopteryx 2.03, 2.04, 2.09, 2.10 [host2], 3.0.0 [host2](OSL 3.0) http://www.archiveopteryx.org/
  • Atmail 6.x [host1]
  • Axigen Mail Server Version 8.0.0
  • BincImap 1.2.3 (GPL) (http://www.bincimap.org/)
  • CommuniGatePro server (Redhat 8.0) (Solaris), CommuniGate Pro 5.2.17[host2](CentOS 5.4)
  • Courier IMAP 1.5.1, 2.2.0, 2.1.1, 2.2.1, 3.0.8, 3.0.3, 4.1.1 (GPL)
    (http://www.courier-mta.org/)
  • Critical Path (7.0.020)
  • Cyrus IMAP 1.5, 1.6,
    2.1, 2.1.15, 2.1.16, 2.1.18
    2.2.1, 2.2.2-BETA, 2.2.3, 2.2.6, 2.2.10, 2.2.12, 2.2.13,
    2.3-alpha (OSI Approved), 2.3.1, 2.3.7, 2.3.16
    (http://asg.web.cmu.edu/cyrus/)
  • David Tobit V8 (proprietary Message system).
  • Deerfield VisNetic MailServer 5.8.6 host1
  • DBMail 1.2.1, 2.0.4, 2.0.9, 2.2rc1 (GPL) (http://www.dbmail.org/).
    2.0.7 seems buggy.
  • DBOX 2.41 System host1.
  • Deerfield VisNetic MailServer 5.8.6 [host1]
  • dkimap4 [host1]
  • Domino (Notes) 4.61 [host1], 6.5 [host1], 5.0.6, 5.0.7, 7.0.2, 6.0.2CF1,
    7.0.1 [host1], 8.0.1 [host1], 8.5.2 [host2], 8.5.3 [host1]
  • Dovecot 0.99.10.4, 0.99.14, 0.99.14-8.fc4, 1.0-0.beta2.7,
    1.0.0 dest/source (http://www.dovecot.org/)
  • Eudora WorldMail v2
  • Fusemail imap.fusemail.net:143 (https://www.fusemail.com/).
  • Gimap (Gmail imap)
  • GMX IMAP4 StreamProxy.
  • Groupwise IMAP (Novell) 6.x and 7.0. Buggy so see the FAQ.
  • hMailServer 5.40-B1950 [host12], 5.3.3 [host2], 4.4.1 [host1](see FAQ)
  • IceWarp Server 10.4.5 host1
  • iPlanet Messaging server 4.15, 5.1, 5.2
  • IMail 7.15 (Ipswitch/Win2003), 8.12, 11.03 [host1]
  • Kerio 7.2.0 Patch 1 [host12], Kerio 8 [host1]
  • Mail2World IMAP4 Server 2.5 host1
  • MailEnable 4.23 [host1] [host2], 4.26 [host1][host2], 5 [host1]
  • MDaemon 7.0.1, 8.0.2, 8.1, 9.5.4 (Windows server 2003 R2 platform),
    9.6.5 [host1], 12 [host2], 12.0.3 [host1], 12.5.5 [host1],
  • Mercury 4.1 (Windows server 2000 platform)
  • Microsoft Exchange Server 5.5, 6.0.6249.0[host1], 6.0.6487.0[host1],
    6.5.7638.1 [host2], 6.5 [host1], Exchange 2007 SP1 (with Update Rollup 2),
    Exchange2007-EP-SP2,
    Exchange 2010 RTM (Release to Manufacturing) [host2],
    Exchange 2010 SP1 RU2[host2],
  • Mirapoint, 4.1.9-GA [host1]
  • Netscape Mail Server 3.6 (Wintel !)
  • Netscape Messaging Server 4.15 Patch 7
  • Office 365 [host1] [host2]
  • OpenMail IMAP server B.07.00.k0 (Samsung Contact ?)
  • OpenWave
  • Oracle Beehive [host1]
  • Parallels Plesk Panel 9.x [host2] 11.x host2
  • Qualcomm Worldmail (NT)
  • QQMail IMAP4Server [host1] [host2] https://en.mail.qq.com/
  • RackSpace hoster secure.emailsrvr.com:993 http://www.rackspace.com/
  • Rockliffe Mailsite 5.3.11, 4.5.6
  • Samsung Contact IMAP server 8.5.0
  • Scalix v10.1, 10.0.1.3, 11.0.0.431, 11.4.6
  • Sendmail Mail Store IMAP4rev1 (5.5.6/mstore-5-5-build-1874 [host1].
  • SmarterMail, Smarter Mail 5.0 Enterprise, Smarter Mail 5.5 [host1],
    SmarterMail Professional 10.2 [host1], Smarter Mail 11.7 [host1][host2].
  • Softalk Workgroup Mail 7.6.4 [host1].
  • SunONE Messaging server 5.2, 6.0 (SUN JES - Java Enterprise System)
  • Sun Java(tm) System Messaging Server 6.2-2.05, 6.2-7.05, 6.3
  • Surgemail 3.6f5-5, 6.3d-72 [host2]
  • UW-imap servers (imap-2000b) rijkkramer IMAP4rev1 2000.287
    (RedHat uses UW like 2003.338rh), v12.264 Solaris 5.7 (OSI Approved)
    (http://www.washington.edu/imap/)
  • UW - QMail v2.1
  • VMS, Imap part of TCP/IP suite of VMS 7.3.2
  • Yahoo [host1]
  • Zarafa 6,40,0,20653 host1
  • Zarafa ZCP 7.1.4 IMAP Gateway [host2]
  • Zimbra-IMAP 3.0.1 GA 160, 3.1.0 Build 279, 4.0.5, 4.5.2, 4.5.6,
    Zimbra 5.0.24_GA_3356.RHEL4 [host1], 5.5, 6.x

Please report to the author any success or bad story with
imapsync and do not forget to mention the IMAP server
software names and version on both sides. This will help
future users. To help the author maintaining this section
report the two lines at the begining of the output if they
are useful to know the softwares. Example:

Host1 software:* OK louloutte Cyrus IMAP4 v1.5.19 server ready
Host2 software:* OK Courier-IMAP ready

You can use option --justconnect to get those lines.
Example:

imapsync --host1 imap.troc.org --host2 imap.trac.org --justconnect

=head1 HUGE MIGRATION

Pay special attention to options
--subscribed
--subscribe
--delete
--delete2
--delete2folders
--maxage
--minage
--maxsize
--useuid
--usecache

If you have many mailboxes to migrate think about a little
shell program. Write a file called file.txt (for example)
containing users and passwords.
The separator used in this example is ';'

The file.txt file contains:

user001_1;password001_1;user001_2;password001_2
user002_1;password002_1;user002_2;password002_2
user003_1;password003_1;user003_2;password003_2
user004_1;password004_1;user004_2;password004_2
user005_1;password005_1;user005_2;password005_2
...

On Unix the shell program can be:

{ while IFS=';' read u1 p1 u2 p2; do
imapsync --host1 imap.side1.org --user1 "$u1" --password1 "$p1"
--host2 imap.side2.org --user2 "$u2" --password2 "$p2" ...
done ; } < file.txt

On Windows the batch program can be:

FOR /F "tokens=1,2,3,4 delims=; eol=#" %%G IN (file.txt) DO imapsync ^
--host1 imap.side1.org --user1 %%G --password1 %%H ^
--host2 imap.side2.org --user2 %%I --password2 %%J ...

The ... have to be replaced by nothing or any imapsync option.

Welcome in shell programming !

=head1 Hacking

Feel free to hack imapsync as the NOLIMIT license permits it.

=head1 Links

Entries for imapsync:
http://www.imap.org/products/showall.php

=head1 SIMILAR SOFTWARES

imap_tools : http://www.athensfbc.com/imap_tools
offlineimap : https://github.com/nicolas33/offlineimap
mbsync : http://isync.sourceforge.net/
mailsync : http://mailsync.sourceforge.net/
mailutil : http://www.washington.edu/imap/
part of the UW IMAP tookit.
imaprepl : http://www.bl0rg.net/software/
http://freecode.com/projects/imap-repl/
imapcopy : http://home.arcor.de/armin.diehl/imapcopy/imapcopy.html
migrationtool : http://sourceforge.net/projects/migrationtool/
imapmigrate : http://sourceforge.net/projects/cyrus-utils/
wonko_imapsync: http://wonko.com/article/554
see also file W/tools/wonko_ruby_imapsync
exchange-away : http://exchange-away.sourceforge.net/
pop2imap : http://www.linux-france.org/prj/pop2imap/

Feedback (good or bad) will often be welcome.

$Id: imapsync,v 1.582 2014/01/24 01:43:19 gilles Exp gilles $

=cut

pragmas

use strict ;
use warnings ;
++$| ;
use Carp ;
use Getopt::Long ;
use Mail::IMAPClient 3.29 ;
use Digest::MD5 qw( md5 md5_hex md5_base64 ) ;
use Digest::HMAC_SHA1 qw( hmac_sha1 ) ;
#use Term::ReadKey ;
#use IO::Socket::SSL ;
use MIME::Base64 ;
use English '-no_match_vars' ;
use File::Basename ;
use POSIX qw(uname SIGALRM) ;
use Fcntl ;
use File::Spec ;
use File::Path qw( mkpath rmtree ) ;
use File::Copy::Recursive ;
use IO::Socket qw(:crlf SOL_SOCKET SO_KEEPALIVE) ;
use Errno qw(EAGAIN EPIPE ECONNRESET) ;
use File::Glob qw( :glob ) ;
use IO::File ;
use Time::Local ;
use Time::HiRes qw( time sleep ) ;
use Test::More 'no_plan' ;
use IPC::Open3 'open3' ;
#use Unix::Sysexits ;

global variables

my(
$rcs, $pidfile, $pidfilelocking,
$debug, $debugimap, $debugimap1, $debugimap2, $debugcontent, $debugflags,
$debugLIST, $debugsleep, $debugdev, $debugmemory, $debugmaxlinelength,
$nb_errors,
$host1, $host2, $port1, $port2,
$user1, $user2, $domain1, $domain2,
$password1, $password2, $passfile1, $passfile2,
@folder, @include, @exclude, @folderrec,
@folderfirst, @folderlast,
$prefix1, $prefix2,
@regextrans2, @regexmess, @regexflag,
$flagsCase, $filterflags, $syncflagsaftercopy,
$sep1, $sep2,
$syncinternaldates,
$idatefromheader,
$syncacls,
$fastio1, $fastio2,
$maxsize, $minsize, $maxage, $minage,
$exitwhenover,
$search, $search1, $search2,
$skipheader, @useheader,
$skipsize, $allowsizemismatch, $foldersizes, $foldersizesatend, $buffersize,
$delete, $delete2, $delete2duplicates,
$expunge, $expunge1, $expunge2, $uidexpunge2, $dry,
$justfoldersizes,
$authmd5, $authmd51, $authmd52,
$subscribed, $subscribe, $subscribe_all,
$version, $help,
$justconnect, $justfolders, $justbanner,
$fast,

    $total_bytes_transferred,
    $total_bytes_skipped,
    $total_bytes_error,
    $nb_msg_transferred, 
$nb_msg_skipped, 
$nb_msg_skipped_dry_mode,
$h1_nb_msg_duplicate,
$h2_nb_msg_duplicate,
$h1_nb_msg_noheader,
$h2_nb_msg_noheader,
$h1_total_bytes_duplicate,
$h2_total_bytes_duplicate,
$h1_nb_msg_deleted,
$h2_nb_msg_deleted,

    $h1_bytes_processed, 
    $h1_nb_msg_processed,
    $h1_nb_msg_at_start, $h1_bytes_start,
    $h2_nb_msg_start, $h2_bytes_start, 
    $h1_nb_msg_end, $h1_bytes_end,
    $h2_nb_msg_end, $h2_bytes_end, 

    $timeout,
$timestart, $timestart_int, $timeend,
    $timebefore,
    $ssl1, $ssl2, 
    $ssl1_SSL_version, $ssl2_SSL_version,
$tls1, $tls2,
$uid1, $uid2,
    $authuser1, $authuser2,
    $proxyauth1, $proxyauth2,
    $authmech1, $authmech2,
    $split1, $split2,
    $reconnectretry1, $reconnectretry2,
$relogin1, $relogin2,
$tests, $test_builder, $tests_debug,
$allow3xx, $justlogin,
$tmpdir,
$releasecheck,
$max_msg_size_in_bytes,
$modules_version,
$delete2folders, $delete2foldersonly, $delete2foldersbutnot,
$usecache, $debugcache, $cacheaftercopy,
$wholeheaderifneeded, %h1_msgs_copy_by_uid, $useuid, $h2_uidguess,
    $addheader,
    %h1, %h2,
    $checkselectable, $checkmessageexists,
    $expungeaftereach,
    $abletosearch,
    $showpasswords,
    $fixslash2,
    $messageidnodomain,
    $fixInboxINBOX,
    $maxlinelength,
    $minmaxlinelength,
$uidnext_default,
    $fixcolonbug,
    $create_folder_old,
    $maxmessagespersecond,
    $maxbytespersecond,
    $skipcrossduplicates, $debugcrossduplicates, 

);

main program

global variables initialisation

$rcs = '$Id: imapsync,v 1.582 2014/01/24 01:43:19 gilles Exp gilles $ ';

$total_bytes_transferred = 0;
$total_bytes_skipped = 0;
$total_bytes_error = 0;
$nb_msg_transferred = 0;
$nb_msg_skipped = $nb_msg_skipped_dry_mode = 0;
$h1_nb_msg_deleted = $h2_nb_msg_deleted = 0;
$h1_nb_msg_duplicate = $h2_nb_msg_duplicate = 0;
$h1_nb_msg_noheader = $h2_nb_msg_noheader = 0;
$h1_total_bytes_duplicate = $h2_total_bytes_duplicate = 0;

$h1_nb_msg_at_start = $h1_bytes_start = 0 ;
$h2_nb_msg_start = $h2_bytes_start = 0 ;
$h1_nb_msg_processed = $h1_bytes_processed = 0 ;

$h1_nb_msg_end = $h1_bytes_end = 0 ;
$h2_nb_msg_end = $h2_bytes_end = 0 ;

$nb_errors = 0;
$max_msg_size_in_bytes = 0;

my %month_abrev = (
Jan => 0,
Feb => 1,
Mar => 2,
Apr => 3,
May => 4,
Jun => 5,
Jul => 6,
Aug => 7,
Sep => 8,
Oct => 9,
Nov => 10,
Dec => 11,
);

sub EX_USAGE {
# 64 on my linux box.
# See http://search.cpan.org/~jmates/Unix-Sysexits-0.02/lib/Unix/Sysexits.pm
return( 64 ) ;
}

@argv will be eat by get_options()

my @argv_copy = @argv;

get_options( ) ;

$SIG{ INT } = &catch_continue ;

local $SIG{ INT } = local $SIG{ QUIT } = local $SIG{ TERM } = &catch_exit ;

$timestart = time( );
$timestart_int = int( $timestart ) ;
$timebefore = $timestart ;

my $timestart_str = localtime( $timestart ) ;
print "Transfer started at $timestart_str\n" ;
print "PID is $PROCESS_ID\n" ;
$modules_version = defined( $modules_version ) ? $modules_version : 1 ;

$releasecheck = defined($releasecheck) ? $releasecheck : 1;
my $warn_release = ($releasecheck) ? check_last_release() : '';

default values

$tmpdir ||= File::Spec->tmpdir();
$pidfile ||= $tmpdir . '/imapsync.pid';

$pidfilelocking = defined( $pidfilelocking ) ? $pidfilelocking : 0 ;

allow Mail::IMAPClient 3.0.xx by default

$allow3xx = defined($allow3xx) ? $allow3xx : 1;

$wholeheaderifneeded = defined( $wholeheaderifneeded ) ? $wholeheaderifneeded : 1;

turn on RFC standard flags correction like \SEEN -> \Seen

$flagsCase = defined( $flagsCase ) ? $flagsCase : 1 ;

Use PERMANENTFLAGS if available

$filterflags = defined( $filterflags ) ? $filterflags : 1 ;

sync flags just after an APPEND, some servers ignore the flags given in the APPEND

like MailEnable IMAP server.

Off by default since it takes time.

$syncflagsaftercopy = defined( $syncflagsaftercopy ) ? $syncflagsaftercopy : 0 ;

turn on relogin 5 by default

$relogin1 = defined( $relogin1 ) ? $relogin1 : 5 ;
$relogin2 = defined( $relogin2 ) ? $relogin2 : 5 ;

if ( $fast ) {
# $useuid = 1 ;
# $foldersizes = 0 ;
# $foldersizesatend = 0 ;
}

Activate --usecache if --useuid is set and no --nousecache

$usecache = 1 if ( $useuid and ( ! defined( $usecache ) ) ) ;
$cacheaftercopy = 1 if ( $usecache and ( ! defined( $cacheaftercopy ) ) ) ;

$checkselectable = defined( $checkselectable ) ? $checkselectable : 1 ;
$checkmessageexists = defined( $checkmessageexists ) ? $checkmessageexists : 0 ;
$expungeaftereach = defined( $expungeaftereach ) ? $expungeaftereach : 1 ;
$abletosearch = defined( $abletosearch ) ? $abletosearch : 1 ;
$checkmessageexists = 0 if ( not $abletosearch ) ;
$showpasswords = defined( $showpasswords ) ? $showpasswords : 0 ;
$fixslash2 = defined( $fixslash2 ) ? $fixslash2 : 1 ;
$fixInboxINBOX = defined( $fixInboxINBOX ) ? $fixInboxINBOX : 1 ;
$create_folder_old = defined( $create_folder_old ) ? $create_folder_old : 0 ;

$delete2duplicates = 1 if ( $delete2 and ( ! defined( $delete2duplicates ) ) ) ;

$maxmessagespersecond = defined( $maxmessagespersecond ) ? $maxmessagespersecond : 0 ;
$maxbytespersecond = defined( $maxbytespersecond ) ? $maxbytespersecond : 0 ;

print banner_imapsync(@argv_copy);

print "Temp directory is $tmpdir\n";

is_valid_directory( $tmpdir ) ;
write_pidfile( $pidfile ) if ( $pidfile ) ;

$fixcolonbug = defined( $fixcolonbug ) ? $fixcolonbug : 1 ;

if ( $usecache and $fixcolonbug ) { tmpdir_fix_colon_bug( ) } ;

$modules_version and print "Modules version list:\n", modules_VERSION(), "\n";

check_lib_version() or
croak "imapsync needs perl lib Mail::IMAPClient release 3.25 or superior \n";

exit_clean( 0 ) if ( $justbanner ) ;

By default, 100 at a time, not more.

$split1 ||= 100;
$split2 ||= 100;

$host1 || missing_option("--host1") ;
$port1 ||= ( $ssl1 ) ? 993 : 143;

$host2 || missing_option("--host2") ;
$port2 ||= ( $ssl2 ) ? 993 : 143;

$debugimap1 = $debugimap2 = 1 if ( $debugimap ) ;
$debug = 1 if ( $debugimap1 or $debugimap2 ) ;

By default, don't take size to compare

$skipsize = (defined $skipsize) ? $skipsize : 1;

$uid1 = defined($uid1) ? $uid1 : 1;
$uid2 = defined($uid2) ? $uid2 : 1;

$subscribe = defined($subscribe) ? $subscribe : 1;

Allow size mismatch by default

$allowsizemismatch = defined($allowsizemismatch) ? $allowsizemismatch : 1;

$delete2folders = 1
if ( defined( $delete2foldersbutnot ) or defined( $delete2foldersonly ) ) ;

if ($justconnect) {
justconnect();
exit_clean(0);
}

$user1 || missing_option("--user1");
$user2 || missing_option("--user2");

$syncinternaldates = defined($syncinternaldates) ? $syncinternaldates : 1;

Turn on expunge if there is not explicit option --noexpunge and option

--delete is given.

Done because --delete --noexpunge is very dangerous on the second run:

the Deleted flag is then synced to all previously transfered messages.

So --delete implies --expunge is a better usability default behaviour.

if ($delete) {
if ( ! defined($expunge)) {
$expunge = 1;
}
}

if ( $uidexpunge2 and not Mail::IMAPClient->can( 'uidexpunge' ) ) {
print "Failure: uidexpunge not supported (IMAPClient release < 3.17), use --expunge2 instead\n" ;
exit_clean( 3 ) ;
}

if ( ( $delete2 or $delete2duplicates ) and not defined( $uidexpunge2 ) ) {
if ( Mail::IMAPClient->can( 'uidexpunge' ) ) {
print "Info: will act as --uidexpunge2\n" ;
$uidexpunge2 = 1 ;
}elsif ( not defined( $expunge2 ) ) {
print "Info: will act as --expunge2 (no uidexpunge support)\n" ;
$expunge2 = 1 ;
}
}

if ( $delete and $delete2 ) {
print "Warning: using --delete and --delete2 together is almost always a bad idea, exiting imapsync\n" ;
exit_clean( 4 ) ;
}

if ($idatefromheader) {
print "Turned ON idatefromheader, ",
"will set the internal dates on host2 from the 'Date:' header line.\n";
$syncinternaldates = 0;
}

if ($syncinternaldates) {
print "Info: turned ON syncinternaldates, ",
"will set the internal dates (arrival dates) on host2 same as host1.\n";
}else{
print "Info: turned OFF syncinternaldates\n";
}

if (defined($authmd5) and ($authmd5)) {
$authmd51 = 1 ;
$authmd52 = 1 ;
}

if (defined($authmd51) and ($authmd51)) {
$authmech1 ||= 'CRAM-MD5';
}
else{
$authmech1 ||= $authuser1 ? 'PLAIN' : 'LOGIN';
}

if (defined($authmd52) and ($authmd52)) {
$authmech2 ||= 'CRAM-MD5';
}
else{
$authmech2 ||= $authuser2 ? 'PLAIN' : 'LOGIN';
}

$authmech1 = uc($authmech1);
$authmech2 = uc($authmech2);

if (defined $proxyauth1 && !$authuser1) {
missing_option("With --proxyauth1, --authuser1");
}

if (defined $proxyauth2 && !$authuser2) {
missing_option("With --proxyauth2, --authuser2");
}

$authuser1 ||= $user1;
$authuser2 ||= $user2;

print "Info: will try to use $authmech1 authentication on host1\n";
print "Info: will try to use $authmech2 authentication on host2\n";

$timeout = defined( $timeout ) ? $timeout : 120 ;
print "Info: imap connexions timeout is $timeout seconds\n";

$syncacls = (defined($syncacls)) ? $syncacls : 0 ;
$foldersizes = (defined($foldersizes)) ? $foldersizes : 1 ;
$foldersizesatend = (defined($foldersizesatend)) ? $foldersizesatend : $foldersizes ;

$fastio1 = (defined($fastio1)) ? $fastio1 : 0;
$fastio2 = (defined($fastio2)) ? $fastio2 : 0;

$reconnectretry1 = (defined($reconnectretry1)) ? $reconnectretry1 : 3;
$reconnectretry2 = (defined($reconnectretry2)) ? $reconnectretry2 : 3;

Since select_msgs() returns no messages when uidnext does not return something

then $uidnext_default is never used. So I have to remove it.

$uidnext_default = 999999 ;

@useheader = ( "Message-Id", "Message-ID", "Received" ) unless ( @useheader ) ;

my %useheader ;

Make a hash %useheader of each --useheader 'key' in uppercase

for ( @useheader ) { $useheader{ uc( $_ ) } = undef } ;

#require Data::Dumper ;
#print Data::Dumper->Dump( [ %useheader ] ) ;
#exit ;

print "Host1: IMAP server [$host1] port [$port1] user [$user1]\n";
print "Host2: IMAP server [$host2] port [$port2] user [$user2]\n";

$password1 || $passfile1 || 'PREAUTH' eq $authmech1 || 'EXTERNAL' eq $authmech1 || do {
$password1 = ask_for_password( $authuser1 || $user1, $host1 ) ;
} ;

$password1 = ( defined( $passfile1 ) ) ? firstline ( $passfile1 ) : $password1 ;

$password2 || $passfile2 || 'PREAUTH' eq $authmech2 || 'EXTERNAL' eq $authmech2 || do {
$password2 = ask_for_password( $authuser2 || $user2, $host2 ) ;
} ;

$password2 = ( defined( $passfile2 ) ) ? firstline ( $passfile2 ) : $password2 ;

my $dry_message = '' ;
$dry_message = "\t(not really since --dry mode)" if $dry ;

$search1 ||= $search if ( $search ) ;
$search2 ||= $search if ( $search ) ;

if ( @regexmess ) {
my $string = regexmess( '' ) ;
# string undef means one of the eval regex was bad.
if ( not ( defined( $string ) ) ) {
die_clean( "Error: one of --regexmess option is bad, check it" ) ;
}
}

if ( @regexflag and not ( defined( flags_regex( '' ) ) ) ) {
die_clean( "Error: one of --regexmess option is bad, check it" ) ;
}

my $imap1 = ();
my $imap2 = ();

$debugimap1 and print "Host1 connection\n";
$imap1 = login_imap($host1, $port1, $user1, $domain1, $password1,
$debugimap1, $timeout, $fastio1, $ssl1, $tls1,
$authmech1, $authuser1, $reconnectretry1,
$proxyauth1, $uid1, $split1, 'Host1', $ssl1_SSL_version );

$debugimap2 and print "Host2 connection\n";
$imap2 = login_imap($host2, $port2, $user2, $domain2, $password2,
$debugimap2, $timeout, $fastio2, $ssl2, $tls2,
$authmech2, $authuser2, $reconnectretry2,
$proxyauth2, $uid2, $split2, 'Host2', $ssl2_SSL_version );

$debug and print "Host1 Buffer I/O: ", $imap1->Buffer(), "\n";
$debug and print "Host2 Buffer I/O: ", $imap2->Buffer(), "\n";

die_clean( 'Not authenticated on host1' ) unless $imap1->IsAuthenticated( ) ;
print "Host1: state Authenticated\n";
die_clean( 'Not authenticated on host2' ) unless $imap2->IsAuthenticated( ) ;
print "Host2: state Authenticated\n";

print "Host1 capability: ", join(" ", @{ $imap1->capability_update() || [] }), "\n";
print "Host2 capability: ", join(" ", @{ $imap2->capability_update() || [] }), "\n";

exit_clean(0) if ($justlogin);

Folder stuff

my (
@h1_folders_all, %h1_folders_all, @h1_folders_wanted, %requested_folder,
%h1_subscribed_folder, %h2_subscribed_folder,
@h2_folders_all, %h2_folders_all,
@h2_folders_from_1_wanted, %h2_folders_from_1_wanted,
%h2_folders_from_1_several,
%h2_folders_from_1_all,
);

Make a hash of subscribed folders in both servers.

for ( $imap1->subscribed( ) ) { $h1_subscribed_folder{ $_ } = 1 } ;
for ( $imap2->subscribed( ) ) { $h2_subscribed_folder{ $_ } = 1 } ;

All folders on host1 and host2

@h1_folders_all = sort $imap1->folders();
@h2_folders_all = sort $imap2->folders();

for ( @h1_folders_all ) { $h1_folders_all{ $_ } = 1 } ;
for ( @h2_folders_all ) { $h2_folders_all{ $_ } = 1 } ;

if ( $fixInboxINBOX and ( my $reg = fix_Inbox_INBOX_mapping( %h1_folders_all, %h2_folders_all ) ) ) {
#print "RRRRRR $reg\n" ;
push( @regextrans2, $reg ) ;
}

if (scalar(@folder) or $subscribed or scalar(@folderrec)) {
# folders given by option --folder
if (scalar(@folder)) {
add_to_requested_folders(@folder);
}

# option --subscribed
if ( $subscribed ) {
    add_to_requested_folders( keys ( %h1_subscribed_folder ) ) ;
}

# option --folderrec
if (scalar(@folderrec)) {
    foreach my $folderrec (@folderrec) {
        add_to_requested_folders($imap1->folders($folderrec));
    }
}

}
else {
# no include, no folder/subscribed/folderrec options => all folders
if (not scalar(@include)) {
add_to_requested_folders(@h1_folders_all);
}
}

consider (optional) includes and excludes

if ( scalar( @include ) ) {
foreach my $include ( @include ) {
my @included_folders = grep { /$include/x } @h1_folders_all ;
add_to_requested_folders( @included_folders ) ;
my $included_folders = join( " ", map( "[$_]", @included_folders ) ) ;
print "Including folders matching pattern '$include': " . $included_folders . "\n" ;
}
}

if ( scalar( @exclude ) ) {
foreach my $exclude ( @exclude ) {
my @requested_folder = sort( keys( %requested_folder ) ) ;
my @excluded_folders = grep { /$exclude/x } @requested_folder ;
remove_from_requested_folders( @excluded_folders ) ;
my $excluded_folders = join( " ", map( "[$_]", @excluded_folders ) ) ;
print "Excluding folders matching pattern '$exclude': " . $excluded_folders . "\n" ;
}
}

Remove no selectable folders

$checkselectable and do {
foreach my $folder (keys(%requested_folder)) {
if ( not $imap1->selectable($folder)) {
print "Warning: ignoring folder $folder because it is not selectable\n";
remove_from_requested_folders($folder);
}
}
} ;

@h1_folders_wanted = sort_requested_folders( ) ;

#my $h1_namespace = $imap1->namespace() ;
#my $h2_namespace = $imap2->namespace() ;
#require Data::Dumper ;
#$debug and print "Host1 namespace:\n", Data::Dumper->Dump([$h1_namespace]) ;
#$debug and print "Host2 namespace:\n", Data::Dumper->Dump([$h2_namespace]) ;

my($h1_sep,$h2_sep);

what are the private folders separators for each server ?

$debug and print "Getting separators\n";
$h1_sep = get_separator($imap1, $sep1, "--sep1");
$h2_sep = get_separator($imap2, $sep2, "--sep2");

my($h1_prefix,$h2_prefix);
$h1_prefix = get_prefix($imap1, $prefix1, "--prefix1");
$h2_prefix = get_prefix($imap2, $prefix2, "--prefix2");

print "Host1 separator and prefix: [$h1_sep][$h1_prefix]\n";
print "Host2 separator and prefix: [$h2_sep][$h2_prefix]\n";

#my $h1_xlist_folders = $imap1->xlist_folders( ) ;
#my $h2_xlist_folders = $imap2->xlist_folders( ) ;
#require Data::Dumper ;
#print "Host1 xlist:\n", Data::Dumper->Dump([$h1_xlist_folders]) ;
#print "Host2 xlist:\n", Data::Dumper->Dump([$h2_xlist_folders]) ;

#exit ;

foreach my $h1_fold ( @h1_folders_wanted ) {
my $h2_fold ;
$h2_fold = imap2_folder_name( $h1_fold ) ;
$h2_folders_from_1_wanted{ $h2_fold }++ ;
if ( 1 < $h2_folders_from_1_wanted{ $h2_fold } ) {
$h2_folders_from_1_several{ $h2_fold }++ ;
}
}
@h2_folders_from_1_wanted = sort keys(%h2_folders_from_1_wanted);

foreach my $h1_fold (@h1_folders_all) {
my $h2_fold;
$h2_fold = imap2_folder_name($h1_fold);
$h2_folders_from_1_all{$h2_fold}++;
}

if ( $foldersizes ) {
( $h1_nb_msg_at_start, $h1_bytes_start ) = foldersizes( "Host1", $imap1, $search1, @h1_folders_wanted ) ;
( $h2_nb_msg_start, $h2_bytes_start ) = foldersizes( "Host2", $imap2, $search2, @h2_folders_from_1_wanted ) ;
$fast or sleep( 2 ) ;
}

exit_clean(0) if ($justfoldersizes);

print
"++++ Listing folders\n",
"Host1 folders list:\n", map( { "[$]\n" } @h1_folders_all ), "\n",
"Host2 folders list:\n", map( { "[$
]\n" } @h2_folders_all ), "\n" ;

print
"Host1 subscribed folders list: ",
map( { "[$_] " } sort keys( %h1_subscribed_folder ) ), "\n"
if ( $subscribed ) ;

my @h2_folders_not_in_1;
@h2_folders_not_in_1 = list_folders_in_2_not_in_1();

print "Folders in host2 not in host1:\n",
map( { "[$_]\n" } @h2_folders_not_in_1 ), "\n" ;

delete_folders_in_2_not_in_1() if $delete2folders;

folder loop

print "++++ Looping on each folder\n";

my $begin_transfer_time = time ;

my %uid_candidate_for_deletion ;
my %uid_candidate_no_deletion ;

my %h2_folders_of_md5 = ( ) ;

FOLDER: foreach my $h1_fold ( @h1_folders_wanted ) {

    last FOLDER if $imap1->IsUnconnected();
    last FOLDER if $imap2->IsUnconnected();

my $h2_fold = imap2_folder_name( $h1_fold ) ;
#relogin1(  ) if ( $relogin1 ) ;
printf( "%-35s -> %-35s\n", "[$h1_fold]", "[$h2_fold]" ) ;

# host1 can not be fetched read only, select is needed because of expunge.
select_folder( $imap1, $h1_fold, 'Host1' ) or next FOLDER ;
#examine_folder( $imap1, $h1_fold, 'Host1' ) or next FOLDER ;


if ( ! exists( $h2_folders_all{ $h2_fold } ) ) {
    create_folder( $imap2, $h2_fold, $h1_fold ) or next FOLDER ;
}

acls_sync( $h1_fold, $h2_fold ) ;

    # Sometimes the folder on host2 is listed (it exists) but is
    # not selectable but becomes selectable by a create (Gmail)
select_folder( $imap2, $h2_fold, 'Host2' ) 
    or ( create_folder( $imap2, $h2_fold, $h1_fold ) 
         and select_folder( $imap2, $h2_fold, 'Host2' ) )
    or next FOLDER ;
my @select_results = $imap2->Results(  ) ;

#print "%%% @select_results\n" ;
my $permanentflags2 = permanentflags( @select_results ) ;
( $debug or $debugflags ) and print "permanentflags: $permanentflags2\n" ;

if ( $expunge or $expunge1 ){
    print "Expunging host1 $h1_fold $dry_message\n" ;
    unless($dry) { $imap1->expunge() } ;
    #print "Expunging host2 $h2_fold\n" ;
    #unless($dry) { $imap2->expunge() } ;
}

if ( ( ( $subscribe and exists $h1_subscribed_folder{ $h1_fold } ) or $subscribe_all )
         and not exists( $h2_subscribed_folder{ $h2_fold } ) ) {
    print "Subscribing to folder $h2_fold on destination server\n" ;
    unless( $dry ) { $imap2->subscribe( $h2_fold ) } ;
}

next FOLDER if ($justfolders);

    last FOLDER if $imap1->IsUnconnected();
    last FOLDER if $imap2->IsUnconnected();

    my $h1_msgs_all_hash_ref = {  } ;
my @h1_msgs = select_msgs( $imap1, $h1_msgs_all_hash_ref, $search1, $h1_fold );
last FOLDER if $imap1->IsUnconnected();

    my $h1_msgs_nb = scalar( @h1_msgs ) ;
    $h1{ $h1_fold }{ 'messages_nb' } = $h1_msgs_nb ;

( $debug or $debugLIST ) and print "Host1 LIST: $h1_msgs_nb messages [@h1_msgs]\n" ;
    $debug and print "Host1 selecting messages of folder [$h1_fold] took ", timenext(), " s\n";

    my $h2_msgs_all_hash_ref = {  } ;
my @h2_msgs = select_msgs( $imap2, $h2_msgs_all_hash_ref, $search2, $h2_fold ) ;
last FOLDER if $imap2->IsUnconnected();

    my $h2_msgs_nb = scalar( @h2_msgs ) ;
    $h2{ $h2_fold }{ 'messages_nb' } = $h2_msgs_nb ;

( $debug or $debugLIST ) and print "Host2 LIST: $h2_msgs_nb messages [@h2_msgs]\n";
    $debug and print "Host2 selecting messages of folder [$h2_fold] took ", timenext(), " s\n";

my $cache_base = "$tmpdir/imapsync_cache/" ;
my $cache_dir = cache_folder( $cache_base, "$host1/$user1/$host2/$user2", $h1_fold, $h2_fold ) ;
my ( $cache_1_2_ref, $cache_2_1_ref ) = ( {}, {} ) ;

my $h1_uidvalidity = $imap1->uidvalidity(  ) || '' ;
my $h2_uidvalidity = $imap2->uidvalidity(  ) || '' ;

    last FOLDER if $imap1->IsUnconnected() ;
    last FOLDER if $imap2->IsUnconnected() ;

if ( $usecache ) {
    print "cache directory: $cache_dir\n" ;
    mkpath( "$cache_dir" ) ;
    ( $cache_1_2_ref, $cache_2_1_ref ) 
            = get_cache( $cache_dir, \@h1_msgs, \@h2_msgs, $h1_msgs_all_hash_ref, $h2_msgs_all_hash_ref ) ;
    print "CACHE h1 h2: ", scalar( keys %$cache_1_2_ref ), " files\n" ; 
    $debug and print '[',
        map ( { "$_->$cache_1_2_ref->{$_} " } keys %$cache_1_2_ref ), " ]\n";
}

my %h1_hash = () ;
my %h2_hash = () ;

my ( %h1_msgs, %h2_msgs ) ;
@h1_msgs{ @h1_msgs } = ();
@h2_msgs{ @h2_msgs } = ();

my @h1_msgs_in_cache = sort { $a <=> $b } keys %$cache_1_2_ref ;
my @h2_msgs_in_cache = keys %$cache_2_1_ref ;

my ( %h1_msgs_not_in_cache, %h2_msgs_not_in_cache ) ;
%h1_msgs_not_in_cache = %h1_msgs ;
%h2_msgs_not_in_cache = %h2_msgs ;
delete @h1_msgs_not_in_cache{ @h1_msgs_in_cache } ;
delete @h2_msgs_not_in_cache{ @h2_msgs_in_cache } ;

my @h1_msgs_not_in_cache = keys %h1_msgs_not_in_cache ;
#print "h1_msgs_not_in_cache: [@h1_msgs_not_in_cache]\n" ;
my @h2_msgs_not_in_cache = keys %h2_msgs_not_in_cache ;

my @h2_msgs_delete2_not_in_cache = () ;
%h1_msgs_copy_by_uid = (  ) ;

if ( $useuid ) {
    # use uid so we have to avoid getting header
    @h1_msgs_copy_by_uid{ @h1_msgs_not_in_cache } = (  ) ;
    @h2_msgs_delete2_not_in_cache = @h2_msgs_not_in_cache if $usecache ;
    @h1_msgs_not_in_cache = (  ) ;
    @h2_msgs_not_in_cache = (  ) ;

    #print "delete2: @h2_msgs_delete2_not_in_cache\n";
}

$debug and print "Host1 parsing headers of folder [$h1_fold]\n";

my ($h1_heads_ref, $h1_fir_ref) = ({}, {});
$h1_heads_ref = $imap1->parse_headers([@h1_msgs_not_in_cache], @useheader) if (@h1_msgs_not_in_cache);
$debug and print "Host1 parsing headers of folder [$h1_fold] took ", timenext(), " s\n";

@$h1_fir_ref{@h1_msgs} = (undef);

$debug and print "Host1 getting flags idate and sizes of folder [$h1_fold]\n" ;
    if ( $abletosearch ) {
    $h1_fir_ref = $imap1->fetch_hash( \@h1_msgs, "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h1_fir_ref )
    if ( @h1_msgs ) ;
    }else{
    my $uidnext = $imap1->uidnext( $h1_fold ) || $uidnext_default ;
    $h1_fir_ref = $imap1->fetch_hash( "1:$uidnext", "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h1_fir_ref )
    if ( @h1_msgs ) ;
    }
$debug and print "Host1 getting flags idate and sizes of folder [$h1_fold] took ", timenext(), " s\n";
unless ($h1_fir_ref) {
    print
    "Host1 folder $h1_fold: Could not fetch_hash ",
    scalar(@h1_msgs), " msgs: ", $imap1->LastError || '', "\n";
    $nb_errors++;
    next FOLDER;
}

my @h1_msgs_duplicate;
foreach my $m (@h1_msgs_not_in_cache) {
    my $rc = parse_header_msg($imap1, $m, $h1_heads_ref, $h1_fir_ref, 'Host1', \%h1_hash);
    if (! defined($rc)) {
        my $h1_size = $h1_fir_ref->{$m}->{"RFC822.SIZE"} || 0;
        print "Host1 $h1_fold/$m size $h1_size ignored (no wanted headers so we ignore this message. To solve this: use --addheader)\n" ;
        $total_bytes_skipped += $h1_size;
        $nb_msg_skipped += 1;
        $h1_nb_msg_noheader +=1;
                    $h1_nb_msg_processed +=1 ;
    } elsif(0 == $rc) {
        # duplicate
        push(@h1_msgs_duplicate, $m);
        # duplicate, same id same size?
        my $h1_size = $h1_fir_ref->{$m}->{"RFC822.SIZE"} || 0;
        $nb_msg_skipped += 1;
        $h1_total_bytes_duplicate += $h1_size;
        $h1_nb_msg_duplicate += 1;
                    $h1_nb_msg_processed +=1 ;
    }
}
    my $h1_msgs_duplicate_nb = scalar( @h1_msgs_duplicate ) ;
    $h1{ $h1_fold }{ 'duplicates_nb' } = $h1_msgs_duplicate_nb ;

    $debug and print "Host1 selected: $h1_msgs_nb  duplicates: $h1_msgs_duplicate_nb\n" ;
$debug and print "Host1 whole time parsing headers took ", timenext(), " s\n";

$debug and print "Host2 parsing headers of folder [$h2_fold]\n";

my ($h2_heads_ref, $h2_fir_ref) = ( {}, {} );
$h2_heads_ref =   $imap2->parse_headers([@h2_msgs_not_in_cache], @useheader) if (@h2_msgs_not_in_cache);
$debug and print "Host2 parsing headers of folder [$h2_fold] took ", timenext(), " s\n" ;

$debug and print "Host2 getting flags idate and sizes of folder [$h2_fold]\n" ;
@$h2_fir_ref{@h2_msgs} = (  ); # fetch_hash can select by uid with last arg as ref


    if ( $abletosearch ) {
    $h2_fir_ref = $imap2->fetch_hash( \@h2_msgs, "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h2_fir_ref)
    if (@h2_msgs) ;
    }else{
    my $uidnext = $imap2->uidnext( $h2_fold ) || $uidnext_default ;
    $h2_fir_ref = $imap2->fetch_hash( "1:$uidnext", "FLAGS", "INTERNALDATE", "RFC822.SIZE", $h2_fir_ref )
    if ( @h2_msgs ) ;
    }

$debug and print "Host2 getting flags idate and sizes of folder [$h2_fold] took ", timenext(), " s\n" ;

my @h2_msgs_duplicate;
foreach my $m (@h2_msgs_not_in_cache) {
    my $rc = parse_header_msg($imap2, $m, $h2_heads_ref, $h2_fir_ref, 'Host2', \%h2_hash);
    my $h2_size = $h2_fir_ref->{$m}->{"RFC822.SIZE"} || 0;
    if (! defined($rc)) {
                    print "Host2 $h2_fold/$m size $h2_size ignored (no wanted headers so we ignore this message)\n" ;
        $h2_nb_msg_noheader += 1 ;
    } elsif(0 == $rc) {
        # duplicate
        $h2_nb_msg_duplicate += 1;
        $h2_total_bytes_duplicate += $h2_size;
        push(@h2_msgs_duplicate, $m);
    }
}

    # %h2_folders_of_md5
    foreach my $md5 (  keys( %h2_hash ) ) {
        $h2_folders_of_md5{ $md5 }->{ $h2_fold } ++ ;
    }


    my $h2_msgs_duplicate_nb = scalar( @h2_msgs_duplicate ) ;
    $h2{ $h2_fold }{ 'duplicates_nb' } = $h2_msgs_duplicate_nb ;

    print "Host2 folder $h2_fold selected: $h2_msgs_nb messages,  duplicates: $h2_msgs_duplicate_nb\n" 
        if ( $debug or $delete2duplicates or $h2_msgs_duplicate_nb ) ;
$debug and print "Host2 whole time parsing headers took ", timenext(), " s\n";

$debug and print "++++ Verifying [$h1_fold] -> [$h2_fold]\n";
# messages in host1 that are not in host2

my @h1_hash_keys_sorted_by_uid 
  = sort {$h1_hash{$a}{'m'} <=> $h1_hash{$b}{'m'}} keys(%h1_hash);

#print map { $h1_hash{$_}{'m'} . " "} @h1_hash_keys_sorted_by_uid;

my @h2_hash_keys_sorted_by_uid 
  = sort {$h2_hash{$a}{'m'} <=> $h2_hash{$b}{'m'}} keys(%h2_hash);


if( $delete2duplicates and not exists( $h2_folders_from_1_several{ $h2_fold } ) ) {
    my @h2_expunge ;

    foreach my $h2_msg ( @h2_msgs_duplicate ) {
        print "msg $h2_fold/$h2_msg marked \\Deleted [duplicate] on host2 $dry_message\n" ;
        push( @h2_expunge, $h2_msg ) if $uidexpunge2 ;
        unless ( $dry ) {
            $imap2->delete_message( $h2_msg ) ;
            $h2_nb_msg_deleted += 1 ;
        }
    }
    my $cnt = scalar @h2_expunge ;
    if( @h2_expunge ) {
        print "uidexpunge $cnt message(s) $dry_message\n" ;
        $imap2->uidexpunge( \@h2_expunge ) if ! $dry ;
    }
        if ( $expunge2 ){
                print "Expunging host2 folder $h2_fold $dry_message\n" ;
                $imap2->expunge(  ) if ! $dry ;
        }
}

if( $delete2 and not exists( $h2_folders_from_1_several{ $h2_fold } ) ) {
        # No host1 folders f1a f1b ... going all to same f2 (via --regextrans2)
    my @h2_expunge;
    foreach my $m_id (@h2_hash_keys_sorted_by_uid) {
        #print "$m_id ";
        unless (exists($h1_hash{$m_id})) {
            my $h2_msg  = $h2_hash{$m_id}{'m'};
            my $h2_flags  = $h2_hash{$m_id}{'F'} || "";
            my $isdel  = $h2_flags =~ /\B\\Deleted\b/x ? 1 : 0;
            print "msg $h2_fold/$h2_msg marked \\Deleted on host2 [$m_id] $dry_message\n"
              if ! $isdel;
            push(@h2_expunge, $h2_msg) if $uidexpunge2;
            unless ($dry or $isdel) {
                $imap2->delete_message($h2_msg);
                $h2_nb_msg_deleted += 1;
            }
        }
    }
    foreach my $h2_msg ( @h2_msgs_delete2_not_in_cache ) {
        print "msg $h2_fold/$h2_msg marked \\Deleted [not in cache] on host2 $dry_message\n";
                    push(@h2_expunge, $h2_msg) if $uidexpunge2;
        unless ($dry) {
            $imap2->delete_message($h2_msg);
            $h2_nb_msg_deleted += 1;
        }
    }
    my $cnt = scalar @h2_expunge ;
    if( @h2_expunge ) {
        print "uidexpunge $cnt message(s) $dry_message\n" ;
        $imap2->uidexpunge( \@h2_expunge ) if ! $dry ;
    }
        if ($expunge2){
                print "Expunging host2 folder $h2_fold $dry_message\n" ;
                $imap2->expunge(  ) if ! $dry ;
        }
}

if( $delete2 and exists( $h2_folders_from_1_several{ $h2_fold } ) ) {
        print "Host2 folder $h2_fold $h2_folders_from_1_several{ $h2_fold } folders left to sync there\n" ;
    my @h2_expunge;
    foreach my $m_id ( @h2_hash_keys_sorted_by_uid ) {
                my $h2_msg  = $h2_hash{ $m_id }{ 'm' } ;
        unless ( exists( $h1_hash{ $m_id } ) ) {
            my $h2_flags  = $h2_hash{ $m_id }{ 'F' } || "" ;
            my $isdel  = $h2_flags =~ /\B\\Deleted\b/x ? 1 : 0 ;
            unless ( $isdel ) {
                                $debug and print "msg $h2_fold/$h2_msg candidate for deletion on host2 [$m_id]\n" ;
                $uid_candidate_for_deletion{ $h2_fold }{ $h2_msg }++ ;
            }
        }else{
                        $debug and print "msg $h2_fold/$h2_msg will cancel deletion on host2 [$m_id]\n" ;
                        $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
                    }
    }
    foreach my $h2_msg ( @h2_msgs_delete2_not_in_cache ) {
        print "msg $h2_fold/$h2_msg candidate for deletion [not in cache] on host2\n";
                    $uid_candidate_for_deletion{ $h2_fold }{ $h2_msg }++ ;
    }

    foreach my $h2_msg ( @h2_msgs_in_cache ) {
        print "msg $h2_fold/$h2_msg will cancel deletion [in cache] on host2\n";
                    $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
    }


            if ( 0 == $h2_folders_from_1_several{ $h2_fold } ) {
                # last host1 folder going to $h2_fold
                    print "Last host1 folder going to $h2_fold\n" ;
                    foreach my $h2_msg ( keys %{ $uid_candidate_for_deletion{ $h2_fold } } ) {
                        $debug and print "msg $h2_fold/$h2_msg candidate for deletion on host2\n" ;
                            if ( exists( $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg } ) ) {
                                $debug and print "msg $h2_fold/$h2_msg canceled deletion on host2\n" ;
                            }else{
                                print "msg $h2_fold/$h2_msg marked \\Deleted on host2 $dry_message\n";
                                    push( @h2_expunge, $h2_msg ) if $uidexpunge2 ;
                                    unless ( $dry ) {
                                        $imap2->delete_message( $h2_msg ) ;
                                        $h2_nb_msg_deleted += 1 ;
                                    }
                            }
                    }
            }

    my $cnt = scalar @h2_expunge ;
    if( @h2_expunge ) {
        print "uidexpunge $cnt message(s) $dry_message\n" ;
        $imap2->uidexpunge( \@h2_expunge ) if ! $dry ;
    }
        if ( $expunge2 ) {
                print "Expunging host2 folder $h2_fold $dry_message\n" ;
                $imap2->expunge(  ) if ! $dry ;
        }

            $h2_folders_from_1_several{ $h2_fold }-- ;                
}


my $h2_uidnext = $imap2->uidnext( $h2_fold ) ;
    $debug and print "Host2 uidnext: $h2_uidnext\n" ;
$h2_uidguess = $h2_uidnext ;
MESS: foreach my $m_id (@h1_hash_keys_sorted_by_uid) {
        last FOLDER if $imap1->IsUnconnected() ;
            last FOLDER if $imap2->IsUnconnected() ;

    #print "h1_nb_msg_processed: $h1_nb_msg_processed\n" ;
    my $h1_size  = $h1_hash{$m_id}{'s'};
    my $h1_msg   = $h1_hash{$m_id}{'m'};
    my $h1_idate = $h1_hash{$m_id}{'D'};

    if ( ( not exists( $h2_hash{ $m_id } ) ) 
                and ( not exists( $h2_folders_of_md5{ $m_id } )
                          or not $skipcrossduplicates ) ) {
        # copy
        my $h2_msg = copy_message( $h1_msg, $h1_fold, $h2_fold, $h1_fir_ref, $permanentflags2, $cache_dir ) ;
                    $h2_folders_of_md5{ $m_id }->{ $h2_fold } ++ ;
                    if( $delete2 and exists( $h2_folders_from_1_several{ $h2_fold } ) and $h2_msg ) {
                        print "msg $h2_fold/$h2_msg will cancel deletion [fresh copy] on host2\n" ;
                        $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
                    }
                    last FOLDER if total_bytes_max_reached(  ) ;
        next MESS;
    }
    else{
            # already on host2
                    if ( exists( $h2_hash{ $m_id } ) ) {
            my $h2_msg   = $h2_hash{$m_id}{'m'} ;
            $debug and print "Host1 found msg $h1_fold/$h1_msg equals Host2 $h2_fold/$h2_msg\n" ;
                            if ( $usecache ) {
                                $debugcache and print "touch $cache_dir/${h1_msg}_$h2_msg\n" ;
                                touch( "$cache_dir/${h1_msg}_$h2_msg" ) 
                                    or croak( "Couldn't touch $cache_dir/${h1_msg}_$h2_msg" ) ;
                            }
                    }elsif( exists( $h2_folders_of_md5{ $m_id } ) ) {
                        my @folders_dup = keys( %{ $h2_folders_of_md5{ $m_id } } ) ;
                        ( $debug or $debugcrossduplicates ) and print "Host1 found msg $h1_fold/$h1_msg is also in Host2 folders @folders_dup\n" ;
                    }
        $total_bytes_skipped += $h1_size ;
        $nb_msg_skipped += 1 ;
                    $h1_nb_msg_processed +=1 ;
            }

            if ( exists( $h2_hash{ $m_id } ) ) {
        #$debug and print "MESSAGE $m_id\n";
        my $h2_msg  = $h2_hash{$m_id}{'m'};

                sync_flags_fir( $h1_fold, $h1_msg, $h2_fold, $h2_msg, $permanentflags2, $h1_fir_ref, $h2_fir_ref ) ;
            # Good
        my $h2_size = $h2_hash{$m_id}{'s'};
        $debug and print
        "Host1 size  msg $h1_fold/$h1_msg = $h1_size <> $h2_size = Host2 $h2_fold/$h2_msg\n";
    }
            last FOLDER if $imap2->IsUnconnected() ;

    if( $delete ) {
                    my $expunge_message = '' ;
                    $expunge_message = "and expunged" if ( $expungeaftereach and ( $expunge or $expunge1 ) ) ;
        print "Host1 msg $h1_fold/$h1_msg marked deleted $expunge_message $dry_message\n" ;
        unless( $dry ) {
            $imap1->delete_message( $h1_msg ) ;
            $h1_nb_msg_deleted += 1 ;
            $imap1->expunge() if ( $expungeaftereach and ( $expunge or $expunge1 ) ) ;
        }
    }
}
# END MESS: loop
    last FOLDER if $imap1->IsUnconnected();
    last FOLDER if $imap2->IsUnconnected();
MESS_IN_CACHE: foreach my $h1_msg ( @h1_msgs_in_cache ) {
    my $h2_msg = $cache_1_2_ref->{ $h1_msg } ;
    $debugcache and print "cache messages update flags $h1_msg->$h2_msg\n";
    sync_flags_fir( $h1_fold, $h1_msg, $h2_fold, $h2_msg, $permanentflags2, $h1_fir_ref, $h2_fir_ref ) ;
    my $h1_size = $h1_fir_ref->{ $h1_msg }->{ 'RFC822.SIZE' } || 0 ;
    $total_bytes_skipped += $h1_size;
    $nb_msg_skipped += 1;
            $h1_nb_msg_processed +=1 ;
            last FOLDER if $imap2->IsUnconnected();                
}

#print "Messages by uid: ", map { "$_ " } keys %h1_msgs_copy_by_uid, "\n" ;
MESS_BY_UID: foreach my $h1_msg ( sort { $a <=> $b } keys %h1_msgs_copy_by_uid ) {
    # 
    $debug and print "Copy by uid $h1_fold/$h1_msg\n" ;
            last FOLDER if $imap1->IsUnconnected();
            last FOLDER if $imap2->IsUnconnected();
    my $h2_msg = copy_message( $h1_msg, $h1_fold, $h2_fold, $h1_fir_ref, $permanentflags2, $cache_dir ) ;
            if( $delete2 and exists( $h2_folders_from_1_several{ $h2_fold } ) and $h2_msg ) {
                print "msg $h2_fold/$h2_msg will cancel deletion [fresh copy] on host2\n" ;
                $uid_candidate_no_deletion{ $h2_fold }{ $h2_msg }++ ;
            }
    last FOLDER if total_bytes_max_reached(  ) ;
}

if ($expunge or $expunge1){
    print "Expunging host1 folder $h1_fold $dry_message\n";
    unless($dry) { $imap1->expunge() };
}
if ($expunge2){
    print "Expunging host2 folder $h2_fold $dry_message\n";
    unless($dry) { $imap2->expunge() };
}

$debug and print "Time: ", timenext(), " s\n";

}

sub total_bytes_max_reached {

return( 0 ) if not $exitwhenover ;
if ( $total_bytes_transferred >= $exitwhenover ) {
        print "Maximum bytes transfered reached, $total_bytes_transferred >= $exitwhenover, ending sync\n" ;
        return( 1 ) ;
    }  

}

print "++++ End looping on each folder\n";
$debug and print "Time: ", timenext(), " s\n";

#print memory_consumption();

if ( $foldersizesatend ) {
timenext() ;
# Get all folders on host2 again since new were created
@h2_folders_all = sort $imap2->folders();
for ( @h2_folders_all ) { $h2_folders_all{ $_ } = 1 } ;
( $h1_nb_msg_end, $h1_bytes_end ) = foldersizes( "Host1", $imap1, $search1, @h1_folders_wanted ) ;
( $h2_nb_msg_end, $h2_bytes_end ) = foldersizes( "Host2", $imap2, $search2, @h2_folders_from_1_wanted ) ;
}

$imap1->logout( ) unless lost_connection($imap1, "for host1 [$host1]");
$imap2->logout( ) unless lost_connection($imap2, "for host2 [$host2]");

stats( ) ;
exit_clean( 1 ) if ( $nb_errors ) ;
exit_clean( 0 ) ;

END of main program

subroutines

sub size_filtered_flag {
my $h1_size = shift ;

if (defined $maxsize and $h1_size >= $maxsize) {
    return( 1 ) ;
}
if (defined $minsize and $h1_size <= $minsize) {
    return( 1 ) ;
}
return( 0 ) ;

}

sub sync_flags_fir {
my ( $h1_fold, $h1_msg, $h2_fold, $h2_msg, $permanentflags2, $h1_fir_ref, $h2_fir_ref ) = @_ ;

if ( not defined( $h1_msg ) ) { return(  ) } ;
if ( not defined( $h2_msg ) ) { return(  ) } ;

my $h1_size = $h1_fir_ref->{$h1_msg}->{"RFC822.SIZE"} ;
return(  ) if size_filtered_flag( $h1_size ) ;

# used cached flag values for efficiency
my $h1_flags = $h1_fir_ref->{ $h1_msg }->{ "FLAGS" } || '' ;
my $h2_flags = $h2_fir_ref->{ $h2_msg }->{ "FLAGS" } || '' ;

sync_flags( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $h2_flags, $permanentflags2 ) ;

    return(  ) ;

}

sub sync_flags_after_copy {
my( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $permanentflags2 ) = @_ ;

    my @h2_flags = $imap2->flags( $h2_msg ) ;
    my $h2_flags = "@h2_flags" ;
    ( $debug or $debugflags ) and print "Host2 flags before resync by STORE on msg $h2_msg: $h2_flags\n" ;
sync_flags( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $h2_flags, $permanentflags2 ) ;
    return(  ) ;

}

sub sync_flags {
my( $h1_fold, $h1_msg, $h1_flags, $h2_fold, $h2_msg, $h2_flags, $permanentflags2 ) = @_ ;

( $debug or $debugflags ) and 
    print "Host1 flags init msg $h1_fold/$h1_msg flags( $h1_flags ) Host2 $h2_fold/$h2_msg flags( $h2_flags )\n" ;

$h1_flags = flags_for_host2( $h1_flags, $permanentflags2 ) ;

$h2_flags = flagsCase( $h2_flags ) ;

( $debug or $debugflags ) and 
    print "Host1 flags filt msg $h1_fold/$h1_msg flags( $h1_flags ) Host2 $h2_fold/$h2_msg flags( $h2_flags )\n" ;


# compare flags - set flags if there a difference
my @h1_flags = sort split(' ', $h1_flags );
my @h2_flags = sort split(' ', $h2_flags );
my $diff = compare_lists( \@h1_flags, \@h2_flags );

$diff and ( $debug or $debugflags ) 
    and     print "Host2 flags msg $h2_fold/$h2_msg replacing h2 flags( $h2_flags ) with h1 flags( $h1_flags )\n";
# This sets flags so flags can be removed with this
# When you remove a \Seen flag on host1 you want to it
# to be removed on host2. Just add flags is not what 
# we need most of the time.

if ( not $dry and $diff and not $imap2->store( $h2_msg, "FLAGS.SILENT (@h1_flags)" ) ) {
    print "Host2 flags msg $h2_fold/$h2_msg could not add flags [@h1_flags]: ",
      $imap2->LastError || '', "\n" ;
    #$nb_errors++ ;
}

    return(  ) ;

}

sub _filter {
my $str = shift or return "";
my $sz = 64;
my $len = length($str);
if ( not $debug and $len > $sz*2 ) {
my $beg = substr($str, 0, $sz);
my $end = substr($str, -$sz, $sz);
$str = $beg . "..." . $end;
}
$str =~ s/\012?\015$//x;
return "(len=$len) " . $str;
}

sub lost_connection {
my($imap, $error_message) = @_;
if ( $imap->IsUnconnected() ) {
$nb_errors++;
my $lcomm = $imap->LastIMAPCommand || "";
my $einfo = $imap->LastError || @{$imap->History}[-1] || "";

            # if string is long try reduce to a more reasonable size
            $lcomm = _filter($lcomm);
            $einfo = _filter($einfo);
            print("Failure: last command: $lcomm\n") if ($debug && $lcomm);
            print("Failure: lost connection $error_message: ", $einfo, "\n");
            return(1);
    }
    else{
        return(0);
    }

}

sub max {
my @list = @_ ;
return( undef ) if ( 0 == scalar( @list ) ) ;
my @sorted = sort { $a <=> $b } @list ;
return( pop( @sorted ) ) ;
}

sub tests_max {
ok( 0 == max(0), "max 0");
ok( 1 == max(1), "max 1");
ok( -1 == max(-1), "max -1");
ok( not ( defined( max( ) ) ), "max no arg" ) ;
ok( 100 == max( 1, 100 ), "max 1 100" ) ;
ok( 100 == max( 100, 1 ), "max 100 1") ;
ok( 100 == max( 100, 42, 1 ), "max 100 42 1") ;
ok( 100 == max( 100, "42", 1 ), "max 100 42 1") ;
ok( 100 == max( "100", "42", 1 ), "max 100 42 1") ;
#ok( 100 == max( 100, "haha", 1 ), "max 100 42 1") ;
return( ) ;
}

sub keyval {
my %hash = @_ ;
return( join( " ", map( { "$_ => " . $hash{ $_ } } keys %hash ) ) . "\n" ) ;
}

sub check_lib_version {
$debug and print "IMAPClient $Mail::IMAPClient::VERSION\n";
if ($Mail::IMAPClient::VERSION eq '2.2.9') {
print "imapsync no longer supports Mail::IMAPClient 2.2.9, upgrade it" ;
return( 0 ) ;
}
else{
# 3.x.x is no longer buggy with imapsync.
return( 1 ) ;
}
}

sub module_version_str {
my( $module_name, $module_version ) = @_ ;
my $str = sprintf( "%-20s %s\n", $module_name, $module_version ) ;
return( $str ) ;
}

sub modules_VERSION {

my @list_version;

my $v ;
eval { require Mail::IMAPClient; $v = $Mail::IMAPClient::VERSION } or $v = "?" ;
push ( @list_version, module_version_str( 'Mail::IMAPClient', $v ) ) ;

eval { require IO::Socket; $v = $IO::Socket::VERSION } or $v = "?" ;
push ( @list_version, module_version_str( 'IO::Socket', $v ) ) ;

eval { require IO::Socket::IP; $v = $IO::Socket::IP::VERSION } or $v = "?"
@gilleslamiral

This comment has been minimized.

Show comment Hide comment
@gilleslamiral

gilleslamiral Jan 24, 2014

Contributor

Dear Dennis,

I fixed this ugly bug in imapsync 1.582
It is hard to go in the same conditions as yours.
Now you fixed it on your side I supposed this won't interest you now.
I'll make it officially public later.

On 22/01/2014 00:30, Dennis Schridde wrote:

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Yes, the CA cert I added to /usr/local/share/ca-certificates was old and expired (i.e. not the one that signed the server's cert), so OpenSSL considered the chain untrusted. Rightly so, as there was a self-signed certificate in it. After I fixed that, imapsync worked like a charm.


Reply to this email directly or view it on GitHub #15 (comment).

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

Contributor

gilleslamiral commented Jan 24, 2014

Dear Dennis,

I fixed this ugly bug in imapsync 1.582
It is hard to go in the same conditions as yours.
Now you fixed it on your side I supposed this won't interest you now.
I'll make it officially public later.

On 22/01/2014 00:30, Dennis Schridde wrote:

Now did you find why you get "SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"?

Yes, the CA cert I added to /usr/local/share/ca-certificates was old and expired (i.e. not the one that signed the server's cert), so OpenSSL considered the chain untrusted. Rightly so, as there was a self-signed certificate in it. After I fixed that, imapsync worked like a charm.


Reply to this email directly or view it on GitHub #15 (comment).

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

@devurandom

This comment has been minimized.

Show comment Hide comment
@devurandom

devurandom Jan 24, 2014

Thanks!

My server is configured to listen only on port 143 and to require starttls before allowing login - if that helps you reproduce it. If you want, I can send you a copy of my dovecot config.

Thanks!

My server is configured to listen only on port 143 and to require starttls before allowing login - if that helps you reproduce it. If you want, I can send you a copy of my dovecot config.

@gilleslamiral

This comment has been minimized.

Show comment Hide comment
@gilleslamiral

gilleslamiral Jan 24, 2014

Contributor

Hi Dennis,

My server is configured to listen only on port 143 and to require starttls before allowing login -
if that helps you reproduce it. If you want, I can send you a copy of my dovecot config.

I already have tls servers.
What I don't have is tls servers with obsolete/broken certificates and I'm lazy
to build one. Anyway the fix should be ok since the issue was clear, the
output clearly showed the starttls() function detected it and now
imapsync check the exit status of starttls().
So we'll wait users that used to fall into the bug, imapsync working faking tls
on broken certificates, and the next release exiting in that case.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

Contributor

gilleslamiral commented Jan 24, 2014

Hi Dennis,

My server is configured to listen only on port 143 and to require starttls before allowing login -
if that helps you reproduce it. If you want, I can send you a copy of my dovecot config.

I already have tls servers.
What I don't have is tls servers with obsolete/broken certificates and I'm lazy
to build one. Anyway the fix should be ok since the issue was clear, the
output clearly showed the starttls() function detected it and now
imapsync check the exit status of starttls().
So we'll wait users that used to fall into the bug, imapsync working faking tls
on broken certificates, and the next release exiting in that case.

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

@devurandom

This comment has been minimized.

Show comment Hide comment
@devurandom

devurandom Jan 24, 2014

What I don't have is tls servers with obsolete/broken certificates

It's easier than that. If you have a server with a valid certificate that was signed by a local CA, and you do not put the CA cert in /usr/local/share/ca-certificates, you'll be able to reproduce. CAcert will probably work as well, if it is not automatically trusted by your distribution.

What I don't have is tls servers with obsolete/broken certificates

It's easier than that. If you have a server with a valid certificate that was signed by a local CA, and you do not put the CA cert in /usr/local/share/ca-certificates, you'll be able to reproduce. CAcert will probably work as well, if it is not automatically trusted by your distribution.

@gilleslamiral

This comment has been minimized.

Show comment Hide comment
@gilleslamiral

gilleslamiral Jan 28, 2014

Contributor

Hallo Dennis,

In fact I had nothing tp prepare,
Patch sounds working since some previous fake tls tests started to break.

But it more complicated, there's still something weird, since I found that:

--ssl1 --tls2 fails on host2 login with "Unable to start TLS: Cannot determine peer hostname for verification"
--tls1 --tls2 succeeds
--tls2 succeeds
--tls1 --ssl2 succeeds

I search.

2 ll_dev_reconnect_ssl_tls

Transfer started at Tue Jan 28 15:35:55 2014
PID is 13974
$RCSfile: imapsync,v $ $Revision: 1.582 $ $Date: 2014/01/24 01:43:19 $
Here is a [linux] system (Linux mail.cardio-sfc.org 2.6.32-54-generic #116-Ubuntu SMP Tue Nov 12 19:27:09 UTC 2013 i686)
With perl 5.10.1 Mail::IMAPClient 3.35
Command line used:
./imapsync --host1 localhost --ssl1 --user1 tata --passfile1 ../../var/pass/secret.tata --host2 localhost --tls2 --user2 titi --passfile2 ../../var/pass/secret.titi --folder INBOX --useuid --delete2
Temp directory is /tmp
PID file is /tmp/imapsync.pid
Modules version list:
Mail::IMAPClient 3.35
IO::Socket 1.31
IO::Socket::IP ?
IO::Socket::INET 1.31
IO::Socket::SSL 1.31
Net::SSLeay 1.35
Digest::MD5 2.39
Digest::HMAC_MD5 1.01
Digest::HMAC_SHA1 1.01
Term::ReadKey 2.30
Authen::NTLM 1.09
File::Spec 3.31
Time::HiRes 1.9719
URI::Escape 3.29
Data::Uniqid 0.12

Info: will act as --uidexpunge2
Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1.
Info: will try to use LOGIN authentication on host1
Info: will try to use LOGIN authentication on host2
Info: imap connexions timeout is 120 seconds
Host1: IMAP server [localhost] port [993] user [tata]
Host2: IMAP server [localhost] port [143] user [titi]
Host1: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information.
Host1: localhost says it has NO CAPABILITY for AUTHENTICATE LOGIN
Host1: success login on [localhost] with user [tata] auth [LOGIN]
Host2: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information.
Can not go to tls encryption on [localhost]:Unable to start TLS: Cannot determine peer hostname for verificationerror:00000000:lib(0):func(0):reason(0)
at ./imapsync line 2370
main::die_clean('Can not go to tls encryption on [localhost]:', 'Unable to start TLS: Cannot determine peer hostname for verif...', '\x{a}') called at ./imapsync line 2091
main::login_imap('localhost', 143, 'titi', undef, 'HUwtEd', undef, 120, 0, undef, ...) called at ./imapsync line 1014

On 24/01/2014 23:19, Dennis Schridde wrote:

What I don't have is tls servers with obsolete/broken certificates

It's easier than that. If you have a server with a valid certificate that was signed by a local CA, and you do not put the CA cert in /usr/local/share/ca-certificates, you'll be able to reproduce. CAcert will probably work as well, if it is not automatically trusted by your distribution.


Reply to this email directly or view it on GitHub #15 (comment).

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

Contributor

gilleslamiral commented Jan 28, 2014

Hallo Dennis,

In fact I had nothing tp prepare,
Patch sounds working since some previous fake tls tests started to break.

But it more complicated, there's still something weird, since I found that:

--ssl1 --tls2 fails on host2 login with "Unable to start TLS: Cannot determine peer hostname for verification"
--tls1 --tls2 succeeds
--tls2 succeeds
--tls1 --ssl2 succeeds

I search.

2 ll_dev_reconnect_ssl_tls

Transfer started at Tue Jan 28 15:35:55 2014
PID is 13974
$RCSfile: imapsync,v $ $Revision: 1.582 $ $Date: 2014/01/24 01:43:19 $
Here is a [linux] system (Linux mail.cardio-sfc.org 2.6.32-54-generic #116-Ubuntu SMP Tue Nov 12 19:27:09 UTC 2013 i686)
With perl 5.10.1 Mail::IMAPClient 3.35
Command line used:
./imapsync --host1 localhost --ssl1 --user1 tata --passfile1 ../../var/pass/secret.tata --host2 localhost --tls2 --user2 titi --passfile2 ../../var/pass/secret.titi --folder INBOX --useuid --delete2
Temp directory is /tmp
PID file is /tmp/imapsync.pid
Modules version list:
Mail::IMAPClient 3.35
IO::Socket 1.31
IO::Socket::IP ?
IO::Socket::INET 1.31
IO::Socket::SSL 1.31
Net::SSLeay 1.35
Digest::MD5 2.39
Digest::HMAC_MD5 1.01
Digest::HMAC_SHA1 1.01
Term::ReadKey 2.30
Authen::NTLM 1.09
File::Spec 3.31
Time::HiRes 1.9719
URI::Escape 3.29
Data::Uniqid 0.12

Info: will act as --uidexpunge2
Info: turned ON syncinternaldates, will set the internal dates (arrival dates) on host2 same as host1.
Info: will try to use LOGIN authentication on host1
Info: will try to use LOGIN authentication on host2
Info: imap connexions timeout is 120 seconds
Host1: IMAP server [localhost] port [993] user [tata]
Host2: IMAP server [localhost] port [143] user [titi]
Host1: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE AUTH=PLAIN ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information.
Host1: localhost says it has NO CAPABILITY for AUTHENTICATE LOGIN
Host1: success login on [localhost] with user [tata] auth [LOGIN]
Host2: * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2008 Double Precision, Inc. See COPYING for distribution information.
Can not go to tls encryption on [localhost]:Unable to start TLS: Cannot determine peer hostname for verificationerror:00000000:lib(0):func(0):reason(0)
at ./imapsync line 2370
main::die_clean('Can not go to tls encryption on [localhost]:', 'Unable to start TLS: Cannot determine peer hostname for verif...', '\x{a}') called at ./imapsync line 2091
main::login_imap('localhost', 143, 'titi', undef, 'HUwtEd', undef, 120, 0, undef, ...) called at ./imapsync line 1014

On 24/01/2014 23:19, Dennis Schridde wrote:

What I don't have is tls servers with obsolete/broken certificates

It's easier than that. If you have a server with a valid certificate that was signed by a local CA, and you do not put the CA cert in /usr/local/share/ca-certificates, you'll be able to reproduce. CAcert will probably work as well, if it is not automatically trusted by your distribution.


Reply to this email directly or view it on GitHub #15 (comment).

Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06

@nbebout nbebout closed this Aug 30, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment