mailbox.Maildir doesn't create subdir structure when create=True and base dir exists

[webknjaz]

BPO	30088
Nosy	@warsaw, @bitdancer, @serhiy-storchaka, @webknjaz, @mlouielu, @miss-islington
PRs	bpo-30088: Document that existing dir structure isn't verified by mailbox.Maildir #1163 [3.8] bpo-30088: Document that existing dir structure isn't verified by mailbox.Maildir (GH-1163) #14749 [3.7] bpo-30088: Document that existing dir structure isn't verified by mailbox.Maildir (GH-1163) #14750
Files	poc.py: PoC of the current Maildir hehavior

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-07-13.14:49:08.821>
created_at = <Date 2017-04-17.07:47:54.651>
labels = ['type-bug', '3.8', 'expert-email', '3.7', 'library', '3.9']
title = "mailbox.Maildir doesn't create subdir structure when create=True and base dir exists"
updated_at = <Date 2019-07-13.14:59:52.052>
user = 'https://github.com/webknjaz'

bugs.python.org fields:

activity = <Date 2019-07-13.14:59:52.052>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-07-13.14:49:08.821>
closer = 'asvetlov'
components = ['Library (Lib)', 'email']
creation = <Date 2017-04-17.07:47:54.651>
creator = 'webknjaz'
dependencies = []
files = ['46808']
hgrepos = []
issue_num = 30088
keywords = ['patch']
message_count = 10.0
messages = ['291789', '291793', '291794', '291795', '291810', '291811', '347808', '347834', '347837', '347838']
nosy_count = 6.0
nosy_names = ['barry', 'r.david.murray', 'serhiy.storchaka', 'webknjaz', 'louielu', 'miss-islington']
pr_nums = ['1163', '14749', '14750']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30088'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[webknjaz]

Hi,

I've faced an issue w/ mailbox.Maildir(). The case is following:

1. I create a folder with tempfile.TemporaryDirectory(), so it's empty
2. I pass that folder path as an argument when instantiating mailbox.Maildir()
3. Then I receive an exception happening because "there's no such file or directory" (namely cur, tmp or new) during interaction with Maildir

Expected result: subdirs are created during Maildir() instance creation.

**Actual result:** subdirs are assumed as existing which leads to exceptions during use.

Workaround: remove the actual dir before passing the path to Maildir(). It will be created automatically with all subdirs needed.

**Fix:** PR linked. Basically it adds creation of subdirs regardless of whether the base dir existed before.

[mlouielu]

I think this patch make the behavior changed.

Documentation wrote that: "If create is True, the mailbox is created if it does not exist.", the current version did that exactly, it won't create subdir (tmp, new, and cur) when dir exists.

The situation face here is that TemporaryDirectory() create a valid dir without Maildir structure subdir (tmp, new, and cur), and the create parameter won't create subdir (because the dir is exist!).

Not sure if this behavior change is worth or not. I'll also upload an PoC for this.

[bitdancer]

Just create a subdirectory inside the tempdir to hold your Maildir folder. I think it is not worth complicating the API for this use case, since it does have a simple solution.

[serhiy-storchaka]

I concur with David. Rather than using a temporary directory as a Maildir folder:

    with tempfile.TemporaryDirectory() as tmpdir:
        with mailbox.Maildir(tmpdir, create=True) as box:
            ...

create a Maildir folder in a temporary directory:

    with tempfile.TemporaryDirectory() as tmpdir:
        with mailbox.Maildir(os.path.join(tmpdir, 'mail'), create=True) as box:
            ...

[webknjaz]

create a Maildir folder in a temporary directory:

with tempfile.TemporaryDirectory() as tmpdir:
    with mailbox.Maildir(os.path.join(tmpdir, 'mail'), create=True) as box:
        ...

Yeah, I came up with the same solution. It just seems weird to me.

The doc says "If create is True, the mailbox is created if it does not exist.". So it operates the term mailbox, not a folder.

I would assume that a valid mailbox [1] is the whole **valid** directory structure with subfolders. If the corresponding subdirs are missing the folder should not be considered as a valid maildir, right?

Given above, I think the documentation doesn't clearly promise the same behavior that the code does. Shouldn't this be fixed? (Either docs, or code)

[1] http://www.qmail.org/man/man5/maildir.html

[bitdancer]

Right. Which is why you get an error if you try to use an empty directory as if it was a maildir :)

create=True creates the *directory* and initializes it.  That seems clear to me, and exactly what I would expect.  "The mailbox" is the directory.  If it already exists, it should be a Maildir folder.  I wouldn't want a program messing with an existing directory if it was invalid...it might trash my data :)

I don't think even a doc change is needed here, but if you want to propose something we can evaluate it.

[webknjaz]

Hi, I've tried clarifying this in docs as suggested @ #1163

[miss-islington]

New changeset e441847 by Miss Islington (bot) (Sviatoslav Sydorenko) in branch 'master':
bpo-30088: Document that existing dir structure isn't verified by mailbox.Maildir (GH-1163)
e441847

[miss-islington]

New changeset d152414 by Miss Islington (bot) in branch '3.7':
bpo-30088: Document that existing dir structure isn't verified by mailbox.Maildir (GH-1163)
d152414

[miss-islington]

New changeset b815669 by Miss Islington (bot) in branch '3.8':
bpo-30088: Document that existing dir structure isn't verified by mailbox.Maildir (GH-1163)
b815669