[WIP, HELP NEEDED] Fix many issues

[kdm9]

Hi,

This started as a fix for #19 , but has grown somewhat.

So far I have:

• Fixed memory consumption issues (MemoryError #8, Large memory usage #19, and probably Crashes on some Maildirs #3) by storing only mail file names (not whole Message objects) while detecting possible duplicate mails. The Messsage objects are loaded from disk while processing clusters of mail files that have the same hash.
• Fixed the cmp issue described in Python 3 - TypeError: 'cmp' is an invalid keyword argument for this function #34
• Attempted to fix several issues with encodings UnicodeDecodeError #32 . I have rudimentary fixes forcing decoding of the message lines, but there are still issues. Any help is welcome.
• Added a progress bar in place of printing a '.' every 100 messages

Cheers,
Kevin

[codecov-io]

Current coverage is 61.02% (diff: 66.66%)

Merging #35 into develop will increase coverage by 28.78%

@@            develop        #35   diff @@
==========================================
  Files             3          3          
  Lines           304        313     +9   
  Methods           0          0          
  Messages          0          0          
  Branches         67         70     +3   
==========================================
+ Hits             98        191    +93   
+ Misses          194         91   -103   
- Partials         12         31    +19   

Powered by Codecov. Last update 6773cad...3915d00

[kdm9]

For the encoding challenge, I suggest we try decoding with ascii, then utf-8 then run chardet. Failing all that I'm not sure what the best method would be.

[kdm9]

Also, it would appear that unlike the rest of python3, the iteritems() method has not been removed from Maildir objects, and the items() method returns a list. This will cause additional excess memory usage. See the source for more detail.

[kdeldycke]

Thanks for the PR! I'm going to review it soon. Do you plan to commit some more stuff or is it ready to merge?

Do you have the courage of adding a couple of unit-tests? Even one or two would be great.

[kdm9]

Ah, it appears to be working for me, but some review and testing especially of the encoding handling would be good. I'll take a look at tests.

[kdm9]

There is another issue. I have vim automatically running pylint, and it detects an undefined variable. Clearly I haven't reached this code path, and I can't tell what the variable should be. Could you take a look at this line please?

[kdeldycke]

Tracked the date_only variable back to c87818c , but it looks like I introduced the issue right after in e847797 .

This really makes me sad. There nothing to prevent anybody to push mistakes. Maildir-deduplicate is really helpful for a lot of people, but the code quality is not very good. Essentially because of its patchy history. If you plan to rely on it for important data, I'll be really happy to merge some unit-tests. Even a 10% in increased coverage would be a great leap forward.

[kdm9]

Working on a super basic test. Don't have heaps of time, but should at least test maildir handling.

[kdeldycke]

@kdmurray91 That's the spirit! If you can only bootstrap the unit-tests of the project, it would be easier, little by little, to improve the whole thing.

As for the date_only return, I cleaned it up in 7c191f5 .

[kdeldycke]

This PR is good to merged IMHO. Just waiting for the last unittest commit.

I'll probably do a proper release on PyPi after this one is merged.

[kdm9]

The test I have written is really basic, and could do with extension if you have a little time. (it's bedtime in Australia).

Things to do:

• test other strategies
• test use of message ids
• test on maildir without any dups
• make the testing function check the presence of a list of files, and assert that no extras are present.
• make the test class copy maildirs to a temp directory, then test without dry-runs

I'm happy for you to merge & release this as you see fit. There can always be more PRs.
Cheers,
K

[kdeldycke]

Thanks @kdmurray91 ! Just waiting for the tests to pass. Then I'll merge it right away.

[kdeldycke]

Looks like maildirs are expected to have their new subfolder.

[kdm9]

Ah, of course. git won't keep them as they are empty. Doh.

[kdm9]

Fix incoming

[kdm9]

Looks like I've screwed up the silencing of the logs. If you can easily work this out, go ahead and merge and fix this then merge (I've added you as a collaborator on my repo). Otherwise I'll do it tomorow.

[kdeldycke]

I'll merge the PR and try to fix the issue.

[kdm9]

Thanks mate!

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.