Wrong behavior of `folder_exists` method

[mjs]

Originally reported by: Anton Koval' (Bitbucket: psihonavt)

---

Hello,

code below demonstrates bug quite clearly:

#!python

In [4]: imap.list_folders()
Out[4]:
[(('\\NoInferiors',), '/', u'INBOX'),
 (('\\NoInferiors', '\\Junk'), '/', u'Bulk Mail'),
 (('\\HasNoChildren',), '/', u'Deleted Messages'),
 (('\\NoInferiors',), '/', u'Drafts'),
 (('\\HasNoChildren',), '/', u'Email_Templates'),
 (('\\HasNoChildren',), '/', u'Notes'),
 (('\\NoInferiors',), '/', u'Send_Later'),
 (('\\HasNoChildren',), '/', u'Sent Messages'),
 (('\\NoInferiors', '\\Sent'), '/', u'Sent Items'),
 (('\\NoInferiors', '\\Trash'), '/', u'Trash')]

In [5]: imap.fol()
imap.folder_encode  imap.folder_exists  imap.folder_status

In [5]: imap.folder_exists(u'Sent Items')
Out[5]: False

In [6]: imap.select(u'Sent Items')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-4be9fc151e24> in <module>()
----> 1 imap.select(u'Sent Items')

AttributeError: 'IMAPClient' object has no attribute 'select'

In [7]: imap.select_folder(u'Sent Items')
Out[7]:
{'EXISTS': 2311,
 'FLAGS': ('\\Draft',
  '\\Answered',
  '\\Flagged',
  '\\Deleted',
  '\\Seen',
  '\\Recent'),
 'PERMANENTFLAGS': ('\\Draft',
  '\\Answered',
  '\\Flagged',
  '\\Deleted',
  '\\Seen'),
 'READ-WRITE': True,
 'RECENT': 0,
 'UIDNEXT': 2312,
 'UIDVALIDITY': 1}

imapclient version is 1.0.0

---

• Bitbucket: https://bitbucket.org/mjs0/imapclient/issue/194

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

I can't replicate this problem. There was a bug in folder_exists which was introduced after 0.13 but was fixed before 1.0.0 was released. Is it possible that you're using a pre-release version of 1.0.0 somehow?

Please run the following to confirm:

python -c "import imapclient; print imapclient.__version__"

The version string will have an "-alpha" suffix if you're using a pre-release.

If that's not the issue then there must be something up with the way the IMAP server handles LIST patterns. It might be that I have to revert the folder_exists implementation to something less efficient but perhaps more reliable.

To help confirm can you please repeat your example but do imap.debug = True before the list_folders and folder_exists calls are made and report the output here.

Also, which IMAP server or mail provider are you seeing this with?

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

I've been digging into this some more and I still can't replicate this. I've gone over the specs and other documents again and I really do think IMAPClient is using LIST correctly. Without seeing the debug output for a session like above I can't diagnose this further.

I'm very reluctant to request all the folders from the server just if one exists as this can be very inefficient if a user a large number of mailboxes.

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

This has been bumped to a later release as there's obviously something going here but I don't know how to fix it given the information I have right now.

[mjs]

Original comment by Anton Koval' (Bitbucket: psihonavt):

---

What kind of debug output do you need? I have access to this account (at least for now) and can provide anything you require. My guess about required output:

#!python

In [12]: imap.debug = 4
In [14]: imap.list_folders()
11:54.196977 > EGNC2 LIST "" "*"
11:54.524498 < * LIST (\\NoInferiors) "/" "INBOX"
11:54.524690 < * LIST (\\NoInferiors \\Junk) "/" "Bulk Mail"
11:54.524816 < * LIST (\\HasNoChildren) "/" "Deleted Messages"
11:54.524945 < * LIST (\\NoInferiors) "/" "Drafts"
11:54.525035 < * LIST (\\HasNoChildren) "/" "Email_Templates"
11:54.525116 < * LIST (\\HasNoChildren) "/" "Notes"
11:54.525206 < * LIST (\\NoInferiors) "/" "Send_Later"
11:54.525281 < * LIST (\\HasNoChildren) "/" "Sent Messages"
11:54.525356 < * LIST (\\NoInferiors \\Sent) "/" "Sent Items"
11:54.525430 < * LIST (\\NoInferiors \\Trash) "/" "Trash"
11:54.525514 < EGNC2 OK  Success
Out[14]:
[(('\\NoInferiors',), '/', u'INBOX'),
 (('\\NoInferiors', '\\Junk'), '/', u'Bulk Mail'),
 (('\\HasNoChildren',), '/', u'Deleted Messages'),
 (('\\NoInferiors',), '/', u'Drafts'),
 (('\\HasNoChildren',), '/', u'Email_Templates'),
 (('\\HasNoChildren',), '/', u'Notes'),
 (('\\NoInferiors',), '/', u'Send_Later'),
 (('\\HasNoChildren',), '/', u'Sent Messages'),
 (('\\NoInferiors', '\\Sent'), '/', u'Sent Items'),
 (('\\NoInferiors', '\\Trash'), '/', u'Trash')]

In [15]: imap.fol
imap.folder_encode  imap.folder_exists  imap.folder_status

In [15]: imap.folder_exists("Sent Items")
12:06.269225 > EGNC3 LIST "" "Sent Items"
12:06.612250 < EGNC3 OK  Success
Out[15]: False

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

As per my earlier reply, I wanted the output with debug set to True. Setting it to 4 is equivalent so what you've done is perfect. The server response is a little unusual and I think this is where the problem is.
I need to recheck the RFCs but I think IMAPClient can be changed to handle this.

Probably not to after Xmas though.

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

Sorry, I had misread your reply (was in a hurry and reading on my phone). This really looks like the server is broken. IMAPClient is asking for folders matching the pattern "Sent Items" and the server isn't returning anything.

Do you happen to know what software the server is running?

Can you please try the following for me?

imap.debug = 4
imap.capabilities()
imap.list_folders("Sent Items%")

Thanks.

[mjs]

Original comment by Anton Koval' (Bitbucket: psihonavt):

---

#!python

In [4]: imap.debug = 4

In [5]: imap.capabilities()
08:15.007745 > JBDJ2 CAPABILITY
08:15.383914 < * CAPABILITY IMAP4rev1 UNSELECT STARTTLS ID CHILDREN NAMESPACE IDLE
08:15.384032 < JBDJ2 OK CAPABILITY completed
Out[5]: ('IMAP4REV1', 'UNSELECT', 'STARTTLS', 'ID', 'CHILDREN', 'NAMESPACE', 'IDLE')

In [6]: imap.list_folders("Sent Items%")
08:29.573325 > JBDJ3 LIST "Sent Items%" "*"
08:29.764029 < JBDJ3 OK  Success
Out[6]: []

Do you happen to know what software the server is running?
-- unfortunately no :(

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

Thanks for that.

Sorry, I gave you the wrong list_folders call. It should have been:

imap.list_folders("", "Sent Items%")

Can you tell me what that returns please?

[mjs]

Original comment by Anton Koval' (Bitbucket: psihonavt):

---

#!python

In [11]: imap.list_folders("", "Sent Items%")
13:13.006478 > LNAB8 LIST "" "Sent Items%"
13:13.196041 < LNAB8 OK  Success
Out[11]: []

Btw:

#!bash

Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 UNSELECT STARTTLS ID CHILDREN NAMESPACE IDLE] Courier-IMAP ready. Copyright 1998-2004 Double Precision, Inc.  See COPYING for distribution information.
. LOGIN ** ***
. OK LOGIN Full IMAP support is enabled
. LIST "" "Sent Items"
. OK  Success
. LIST "" "*"
* LIST (\NoInferiors) "/" "INBOX"
* LIST (\NoInferiors \Junk) "/" "Bulk Mail"
* LIST (\HasNoChildren) "/" "Deleted Messages"
* LIST (\NoInferiors) "/" "Drafts"
* LIST (\HasNoChildren) "/" "Email_Templates"
* LIST (\HasNoChildren) "/" "Notes"
* LIST (\NoInferiors) "/" "Send_Later"
* LIST (\HasNoChildren) "/" "Sent Messages"
* LIST (\NoInferiors \Sent) "/" "Sent Items"
* LIST (\NoInferiors \Trash) "/" "Trash"
. OK  Success
. LIST "" "Sent Messages"
* LIST (\HasNoChildren) "/" "Sent Messages"
. OK  Success
. LIST "" "Bulk Mail"
* LIST (\NoInferiors \Junk) "/" "Bulk Mail"
. OK  Success

-- looks like really some bug in that IMAP server (Courier-IMAP).

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

What happens if you do imap.list_folders("", "<folder>") for one of the other mailboxes with the "\NoInferiors" attribute?

Good to know that it's Courier. I'll try and spin up a Courier instance locally to play with. I've been meaning to do that for a while anyway.

You should probably change this account's password since you've just included it here :)

[mjs]

Original comment by Anton Koval' (Bitbucket: psihonavt):

---

Yeah, my bad with password :)
Listing works ok:

#!python

In [18]: imap.list_folders("", "Trash")
33:16.557277 > LNAB15 LIST "" "Trash"
33:16.747643 < * LIST (\NoInferiors \Trash) "/" "Trash"
33:16.747792 < LNAB15 OK  Success
Out[18]: [(('\\NoInferiors', '\\Trash'), '/', u'Trash')]

In [19]: imap.list_folders("", "Bulk Mail")
33:21.021108 > LNAB16 LIST "" "Bulk Mail"
33:21.213970 < * LIST (\NoInferiors \Junk) "/" "Bulk Mail"
33:21.214118 < LNAB16 OK  Success
Out[19]: [(('\\NoInferiors', '\\Junk'), '/', u'Bulk Mail')]

In [21]: imap.list_folders("", "Send_Later")
33:55.949148 > LNAB18 LIST "" "Send_Later"
33:56.145417 < * LIST (\NoInferiors) "/" "Send_Later"
33:56.145566 < LNAB18 OK  Success
Out[21]: [(('\\NoInferiors',), '/', u'Send_Later')]

, except for "Sent Items" ...

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

This looks like a server bug or that something is broken with that folder on the server. Can you successfully use search, fetch and append on the Sent Items folder?

[mjs]

Original comment by Menno Smits (Bitbucket: mjs0):

---

Closing due to inactivity and lack of information. Feel free to reopen if this is still an issue.