"Duplicate folder name" during restore

[MarkSte]

New user of v1.0-beta. Happy user. I downloaded all 20,000 emails from one Gmail account and restored them to another. Everything worked, even more easily than I hoped.

So, then I tried the same thing with another Gmail account. Downloaded all 2,000 emails as problem-free as before. But when I went to restore them to another Gmail account, only two of the emails were restored before I hit this error:

Error create failed: '[ALREADYEXISTS] Duplicate folder name (Failure)'. For more information see log file.

I can't find the log file. It's not in the .gmvault directory. It's not in the gm-vault.db directory. It's not in the directory where gmvault itself is installed. Rerunning with --debug doesn't seem to create a log file, either. In both cases, a log is dumped to the screen (which is where I pulled the error message above), but I can't find any more information about what this error means or how to get around it. It's repeatable.

Help, please.

[MarkSte]

OK, I found the log file. It's in my home directory. This is the relevant log:

Traceback (most recent call last):
File "gmv_cmd.py", line 475, in run
File "gmv_cmd.py", line 383, in _restore
File "gmvault.pyc", line 858, in restore
File "imap_utils.pyc", line 381, in create_gmail_labels
File "imapclient\imapclient.pyc", line 456, in create_folder
File "imapclient\imapclient.pyc", line 790, in _checkok
File "imapclient\imapclient.pyc", line 775, in _check_resp
error: create failed: '[ALREADYEXISTS] Duplicate folder name (Failure)'

[MarkSte]

OK, the debug log suggested which email it was having trouble restoring. I looked at it and it was strange. It was a test email I sent when I first set up the Gmail account. It was "sent" to my email address about a half dozen times. Meaning the "to" list had my email address in it a half dozen times.

Anyway, I just deleted that email from my gmvault db and kicked off another restore. This one is chugging away nicely.

[gaubert]

@MarkSte Thanks for reporting the problem. This look strange to me because normally I do not recreate the labels if they already exists. I would be happy to have the logs regarding the 2 emails before.I want to see the labels for these emails because it is a label problem here.
I am not sure having 12 times yourself in the to is a problem.
As said before I would be happy to have the logs before the issue to see what were the labels of the 2 first emails

Thanks

[MarkSte]

Thanks for the quick response. Restore has failed several more times in this exercise. I've determined what's in common with all the failures. It's the presence of the label "[Imap]/Trash." If I edit the .meta file and remove that label, then rerun gmvault, the email is successfully restored.

[MarkSte]

I just found another problem label. It's "[Imap]/Sent." Apparently, gmvault is having trouble with emails with Imap labels.

[gaubert]

hmmm I have never tested emails in "[Imap]"/Trash. To be clear, is it the Trash dir of you local email client not the Trash dir of Gmail ?
If this is the case I will have a look at this use-case and let you know

[MarkSte]

Here's the usage (I think). On my Android phone, I access my Gmail account through Imap. Sometimes, I delete the email on the phone. That puts it into an [Imap]/Trash folder. That folder shows up in my Gmail account itself, with the deleted emails in it. Same for [Imap]/Sent, for emails sent from my Android phone. The Gmail account I'm trying to restore these emails to already as its own [Imap]/Trash and [Imap]/Sent folders, but gmvault is unable to restore emails to those folders. Instead it looks like it's trying to create them afresh and fails. Anyway, that's my guess as to what's going on.

[MarkSte]

One easy workaround, if I had known of this situation, would have been to clean up my [Imap]/Trash folder before trying to sync/restore this Gmail account. If the emails really are trash, I could just delete them and not have to bother gmvault with handling them at all. And if, they aren't intended to be trashed, I could move them to other folders before running gmvault sync. So, I don't consider this a showstopper, but if it can't be "fixed" it should be noted in the getting-started documentation.

[gaubert]

@MarkSte it should work and I need to understand why Gmvault thinks that the dir doesn't exist.
I would really appreciate to have the logs in debug in order to see the labels and not work on what I got from you.
Could you run it in debug and send me the logs (also possible via email guillaume((dot))aubert((at))gmail((dot))com

Thanks

[MarkSte]

I finally edited all the offending emails in the db and managed to get them all restored. I'll see if I can recreate something to show you. Give me a day or two.

[MarkSte]

In a separate email, I've sent you an offending email and the debug log file.

[gaubert]

@MarkSte Thanks, I got it and will investigate tonight

[diedummydie]

I have the same problem, and I have also sent/deleted mail from a mobile IMAP device. In addition, I used my account to access external email accounts, and it may be the first one of those encountered that stops the works. (I may be wrong about which email caused the problem; is it the last "Restore email with id ..." right before the error, or is it the ID after that one?)

Meta

{"msg_id": "EMEW-k3M7f7c438e17aecb49e0e5e0ce78e13157a47-cf4.28be17df.35407a4b@aol.com", "gm_id": 1267676855994393694, "labels": ["\Inbox", "K12.wv.us"], "thread_ids": 1267676855994393694, "flags": ["\Seen"], "internal_date": 1208936459, "subject": "This is the subject.\r"}

Terminal Output

Error: create failed: '[ALREADYEXISTS] Folder name conflicts with existing folde
r name. (Failure)'.

=== Exception traceback ===
Traceback (most recent call last):
File "gmv_cmd.py", line 515, in run
File "gmv_cmd.py", line 423, in _restore
File "gmvault.pyc", line 976, in restore
File "imap_utils.pyc", line 395, in create_gmail_labels
File "imapclient\imapclient.pyc", line 456, in create_folder
File "imapclient\imapclient.pyc", line 790, in _checkok
File "imapclient\imapclient.pyc", line 775, in _check_resp
error: create failed: '[ALREADYEXISTS] Folder name conflicts with existing folde
r name. (Failure)'

=== End of Exception traceback ===

[diedummydie]

SOLVED - For me, anyway.

The "K12.wv.us" label on mail that Gmail had been fetching from another account has a capital "K" in the meta files, while the account into which I was importing had the label with a lower-case "k". Renaming the label in the recipient account to match the meta files solved the problem.

It may be interesting to find out if @MarkSte's IMAP folders have the same letter case in the meta files vs the receiving account.

[codebox]

I've also had this problem, stack trace below, let me know if you need anymore info

Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gmv/gmv_cmd.py", line 515, in run
self._restore(args, credential)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gmv/gmv_cmd.py", line 423, in _restore
restorer.restore(extra_labels = labels, restart = args['restart'])
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gmv/gmvault.py", line 976, in restore
self.src.create_gmail_labels(labels_to_create)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gmv/imap_utils.py", line 395, in create_gmail_labels
if self.server.create_folder(directory) != 'Success':
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/imapclient/imapclient.py", line 456, in create_folder
self._checkok('create', typ, data)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/imapclient/imapclient.py", line 790, in _checkok
self._check_resp('OK', command, typ, data)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/imapclient/imapclient.py", line 775, in _check_resp
raise self.Error('%s failed: %r' % (command, data[0]))
error: create failed: '[ALREADYEXISTS] Folder name conflicts with existing folder name. (Failure)'

[MarkSte]

vrenna, I believe that my problem with "[Imap]/Trash" is not related to case insensitivity. At least, the problem case I sent to gaubert doesn't have this problem.

[gaubert]

@MarkSte @codebox @VRenna I investigated more and it has to do with the fact that GMail labels are case insensitive but returned with upper and lower cases by IMAP. So in some cases, they are sometimes in lower cases and in mixed cases.
I will test my change and will make a new version for testing purposes.
Will you be ready to test and for what platform ?
Let me know.

[codebox]

Cool - happy to test new version, I'm on OSX

[gaubert]

@MarkSte @codebox It is more complicated than expected but I am progressing.
Could you please show me a .meta file (in gmvault-db) for an email that contains a label:"[Imap]/Trash" (or another [Imap]/something).
I would need to see exactly what is saved.
Go to the gmvault-db and grep for "[Imap]/Trash"

Again thanks a lot

[gaubert]

@MarkSte @codebox I would need to know something else. On Gmail the labels '[IMAP]/Trash', [IMAP]/Sent are nested or flat ?
I suspect they are flat and seen as different labels and not all nested under [IMAP].

Thanks

[MarkSte]

That was the situation in the attachment I sent May 14. I am including it again.

Mark Steger

On Wed, May 23, 2012 at 2:49 PM, Guillaume Aubert
reply@reply.github.com
wrote:

@MarkSte @codebox It is more complicated than expected but I am progressing.
Could you please show me a .meta file (in gmvault-db) for an email that contains a label:"[Imap]/Trash" (or another [Imap]/something).
I would need to see exactly what is saved.
Go to the gmvault-db and grep for "[Imap]/Trash"

Again thanks a lot

---

Reply to this email directly or view it on GitHub:
#31 (comment)

[MarkSte]

The "[Imap]/Trash" is nested in the receiving Gmail account of the
restore. The case of Imap is as written.

Mark Steger

On Wed, May 23, 2012 at 3:07 PM, Guillaume Aubert
reply@reply.github.com
wrote:

@MarkSte @codebox I would need to know something else. On Gmail the labels '[IMAP]/Trash',  [IMAP]/Sent are nested or flat ?
I suspect they are flat and seen as different labels and not all nested under [IMAP].

Thanks

---

Reply to this email directly or view it on GitHub:
#31 (comment)

[codebox]

Hi,

Attached is a screenshot of how the imap/trash folder appears in the
account where the mails came from. I can't find any .meta files containing
imap labels in the db folder - these are the commands I ran, let me know if
I need to do something differently:

robs-macbook-pro-2:db rob$ pwd
/Users/rob/gmvault-db/db
robs-macbook-pro-2:db rob$ ls
2002-09 2004-08 2004-12 2005-04 2005-08 2005-12 2006-04 2006-08 2006-12
2007-04 2007-08 2007-12 2008-04 2008-08 2008-12 2009-04 2009-08 2009-12
2010-04 2010-08 2010-12 2011-04 2011-08 2011-12 2012-04
2002-10 2004-09 2005-01 2005-05 2005-09 2006-01 2006-05 2006-09 2007-01
2007-05 2007-09 2008-01 2008-05 2008-09 2009-01 2009-05 2009-09 2010-01
2010-05 2010-09 2011-01 2011-05 2011-09 2012-01 2012-05
2002-11 2004-10 2005-02 2005-06 2005-10 2006-02 2006-06 2006-10 2007-02
2007-06 2007-10 2008-02 2008-06 2008-10 2009-02 2009-06 2009-10 2010-02
2010-06 2010-10 2011-02 2011-06 2011-10 2012-02
2004-07 2004-11 2005-03 2005-07 2005-11 2006-03 2006-07 2006-11 2007-03
2007-07 2007-11 2008-03 2008-07 2008-11 2009-03 2009-07 2009-11 2010-03
2010-07 2010-11 2011-03 2011-07 2011-11 2012-03
robs-macbook-pro-2:db rob$ grep -ri --include="*.meta" imap .
robs-macbook-pro-2:db rob$

Rob

On Wed, May 23, 2012 at 9:07 PM, Guillaume Aubert <
reply@reply.github.com

wrote:

@MarkSte @codebox I would need to know something else. On Gmail the labels
'[IMAP]/Trash', [IMAP]/Sent are nested or flat ?
I suspect they are flat and seen as different labels and not all nested
under [IMAP].

Thanks

---

Reply to this email directly or view it on GitHub:
#31 (comment)

[gaubert]

@MarkSte @codebox @VRenna I have created a fix for the [IMAP]/.. issue. Man labels are a mess in Gmail. It is not handled clearly at the IMAP level. Anyway I am testing my fix now and will build a test version for you tomorrow or on saturday. Hope you will be ready to test it.
Again thanks a lot for your help because this is important to have Gmvault working with Android

[diedummydie]

Great! Thanks, @gaubert !

[gaubert]

@MarkSte @codebox @VRenna I have created v1.6-develop that should fix the issue with [IMAP](hopefully ;-) ).
For @VRenna it should also take care of the uppercase issue in the labels.
It is available from here : http://bit.ly/JtfbvD
Pick the right version for your plateform and please test it with --debug
$>gmvault restore myuser@gmail.com --debug

In case of problems I would appreciate if you could send me the full log file available in ~/gmvault.log for mac and linux and C:\Users\gmvault.log for windows.
You can send it to guillaume((dot))aubert((at))gmail((dot))com.

Hope this will fix the issue. Let me know.

Many thanks for your efforts.

[MarkSte]

I used v1.6-develop successfully. My test case had some [Imap]/Trash
and [Imap]/Sent emails. All were restored successfully. I can't say I
had the same exact combination before and after as in my earlier
failures. It seems like there are all sorts of ways those Imap Trash
and Sent folders can get created (unnested, nested, sometimes even
both in the same account). But I can say that everything worked as
expected this time. I think it's likely you've fixed the problem.
Thanks for the quick attention to this.

MarkSte

On Sat, May 26, 2012 at 8:15 AM, Guillaume Aubert
reply@reply.github.com
wrote:

@MarkSte @codebox @VRenna I have created v1.6-develop that should fix the issue with [IMAP](hopefully ;-) ).
For @VRenna it should also take care of the uppercase issue in the labels.
It is available from here : http://bit.ly/JtfbvD
Pick the right version for your plateform and please test it with --debug
$>gmvault restore myuser@gmail.com --debug

In case of problems I would appreciate if you could send me the full log file available in ~/gmvault.log for mac and linux and C:\Users\gmvault.log for windows.
You can send it to guillaume((dot))aubert((at))gmail((dot))com.

Hope this will fix the issue. Let me know.

Many thanks for your efforts.

---

Reply to this email directly or view it on GitHub:
#31 (comment)

[diedummydie]

Camping today. I'll share results tomorrow evening.

[gaubert]

@MarkSte very good. I had to take some decision on how to create the nested labels but it seems ok now.
Gmail labels are a strange beast and nested labels are even stranger.
Very happy it worked for you.
I will fix few other things before to release 1.6 so look for it.
Thanks for your testing

[gaubert]

@VRenna Ok enjoy the camping. Waiting for the results.

[diedummydie]

@gaubert Everything ran perfectly. Before testing I renamed the relevant tag to make sure there was a case mismatch. Running the "restore" correctly placed all of the UC-tagged archive emails into the LC-tag on the account. Thank you again for all of your hard work.

[gaubert]

@VRenna great to learn that the fix seems ok