Add upstream changes and clean up where possible

[dan-blanchard]

This is very much a work in progress at the moment, and I'm just creating the PR to make it easier for me to keep track of Travis results.

I have a few goals for this branch:

1. Pull in changes from Mozilla's upstream code. There aren't as many as I had initially expected but there are some.
2. Improve PEP8 compliance all over the place. The previous maintainers tried to keep variable names identical to the C code, presumably to ease the comparison with the Mozilla code, but we're going to be diverging from upstream after pulling in the changes mentioned in 1. Basically, Mozilla seems very likely to abandon their character encoding detector in the near future and switch to using ICU, but ICU doesn't support all of the codecs we currently do, because it is more web-focused. If our goal here is to be a truly universal character encoding detector, we'll need to go our own way in the future in that respect.
3. Make the unit tests pass, or at the very least make it obvious that the tests are actually failing (instead of ignoring the failures like our current Travis build does).

So far, I've done a little bit of point 1 and updated the Travis testing setup to use nose and report test coverage via Coveralls.

[dan-blanchard]

@sigmavirus24 I know this is a total bear of a PR at this point, but most of the changes are purely cosmetic. The main substantive changes are:

1. PEP8 compliance as much as possible (especially with variable names).
2. Implemented filter_english_letters (filter_with_english_letters #46)
3. Switched to using enums (via enum34 and Python 3.4 standard library) instead of constants in a few places.
4. Added a bunch of comments to UniversalDetector (I'll try to add more to others later).
5. Added the same language filtering support present in the upstream Mozilla code (which appears to be the only substantive change they made other than commenting out the Hungarian detectors).

I'd like to propose a more substantial refactoring of things to make it simpler to retrain some of the models, which were never updated after they were established ~15 years ago, but I'll file a separate issue about that.

If you get a chance to look this over, I'd appreciate your thoughts.

[dan-blanchard]

Further, if we're going to care about style, we should add it as a Travis CI job to at least catch regressions. Or we could add landscape.io. If we want to do the former, I'll send a PR to this branch with a set-up I use on pretty much every other project that makes it significantly easier.

Wow, I hadn't seen landscape.io before. That seems pretty nice. I'm fine with using that if you are.

[dan-blanchard]

So I have a few thoughts that I've left as comments. I noticed you selectively made one attribute of the probers public. Given the defined API for the library, why not make everything public? Or leave everything private as it was? I'm not sure I understand the point of it. Seems like unnecessary churn.

That was a mishap of all the massive find-and-replacing I was doing. That said, I'd honestly prefer to make everything public that doesn't currently need/use a property.

[dan-blanchard]

@sigmavirus24 I believe I've addressed all of your comments at this point.

[dan-blanchard]

I've gone ahead and disabled our Hungarian probers (like the Mozilla authors did). This fixes all but one of our unit test failures. The remaining failure is for a Greek XML file where we misidentify ISO-8859-7 as Windows-1253, which leads to a one character difference in the decoded output. Without retraining our models, I'm not sure how we could actually fix that one.

The difference is:

-  δραστήριου τετραπληγικού. Η εργασία είναι στην Αθήνα, στον Άγιο Νικόλαο (Αχαρνών). Τα καθήκοντά αφορούν στην υποστήριξη των υλικών καθημερινών αναγκών, την 
?                                                             ^

+  δραστήριου τετραπληγικού. Η εργασία είναι στην Αθήνα, στον ¶γιο Νικόλαο (Αχαρνών). Τα καθήκοντά αφορούν στην υποστήριξη των υλικών καθημερινών αναγκών, την 
?                                                             ^

[dan-blanchard]

I'm going to merge this because I believe I've addressed all of the objections @sigmavirus24 and we have much bigger future direction things to think about at the moment.

[sigmavirus24]

So I just realized, @dan-blanchard that you added a reliance on enum34 which is going to pose problems for requests. Any chance we can revert that work since it isn't necessary or a public API change?

[dan-blanchard]

@sigmavirus24 why is the enum34 reliance an issue for requests? Since requests is probably the main way that chardet gets used, we can change things for it if necessary, but IMHO the code is a lot cleaner using enums.

[sigmavirus24]

Because Kenneth insists on vendoring requests' dependencies. Which means dependencies with dependencies need to either vendor them or they're axed. Maybe once we can convince Kenneth to stop vendoring it won't be a problem, but this will prevent requests users from getting any of the bug fixes introduced in this mega pull request.

Disclaimer: I think Enums are superior too.

[dan-blanchard]

@sigmavirus24 I just made our enums inherit from object instead of IntEnum because we weren't actually using any of the fancier features. That should eliminate the problem. 02edb35