Encoding issue

[kdm9]

Hi @kdeldycke

Since the changes we made yesterday I have been able to analyse a bigger set of my maildir. It appears that we missed a few encoding issues. The attached seems to fix them, but checking this on another test-set (specifically a non-English maildir) would be wise.

Cheers,
K

[codecov-io]

Current coverage is 61.63%

Merging #36 into develop will not change coverage

@@            develop        #36   diff @@
==========================================
  Files             3          3          
  Lines           318        318          
  Methods           0          0          
  Messages          0          0          
  Branches         71         71          
==========================================
  Hits            196        196          
  Misses           91         91          
  Partials         31         31          

Powered by Codecov. Last update c96c027...c96c027

[kdm9]

Looks like the approach I take is python-2 incompatible. Any ideas on how to port this @kdeldycke?

[kdeldycke]

Isn't it because message_from_binary_file was introduced in Python 3.2?

In which case you should try to hard. Let's revert to the old behaviour for Python 2. What do you think?

[kdeldycke]

FYI, I've reverted to your previous behaviour in c96c027 for tests to pass in Python 2.

Can you try to rebase this PR on top of the develop branch?

[kdm9]

@kdeldycke Thanks heaps. I've rebased and pushed.

[kdeldycke]

Thanks. Tests still not passing though... :(

Is there a reason to not keep using the get_payload() method in 2e43a69 ?

[kdm9]

Ah, not really, other than that it kept raising UnicodeError and going the route I did always worked (on python 3).

[kdeldycke]

Sorry to be skeptical, it's just that it feels like the issue is a .decode('utf-8') away of being solved. That's why I'm not certain we need heavy-duty encoding guessing via chardet.

So here is what I propose: if you can isolate one message triggering a UnicodeDecodeError, I'll try to solve it through a much more restrained decoding path. In the mean time, I'll keep your current PR open and I'll merge it as a last resort. Unless you find a way to make the unittests pass in Python 2. What do you think?

[kdeldycke]

Handling of unparseable email has been added in #47, which supersedes and invalidates half of this PR. As a result, and because it's currently un mergeable as-is, I'll close this PR for now. Sorry @kdmurray91. :(

That being said, your effort was not vain as I pointed to your code in #46 as a possible way to enhance parsing of badly encoded mails.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.