Unicode

[casperdcl]

Fallback ascii if not specified and not explicitly supported by stream

[casperdcl]

fixes #15

[hadim]

Could you add a proper doctstring for parameters please ?

[hadim]

Thanks for working on this !

[casperdcl]

lol tbh I'm just using this as procrastination. It might look like I've been working on this all day for the last few days but really I haven't. I work better when constantly switching projects :P

[casperdcl]

sheesh. @kmike late to the party, yes, maybe, but definitely partyin'

Total number of files: 14
Total number of lines: 586
Total number of commits: 60
+----------------------+-----+---------+-------+--------------------+
| name                 | loc | commits | files | distribution       |
+----------------------+-----+---------+-------+--------------------+
| Casper da Costa-Luis | 219 | 15      | 4     | 37.4 / 25.0 / 28.6 |
| Hadrien Mary         | 97  | 8       | 6     | 16.6 / 13.3 / 42.9 |
| Mikhail Korobov      | 75  | 10      | 9     | 12.8 / 16.7 / 64.3 |
| Ivan Ivanov          | 68  | 11      | 4     | 11.6 / 18.3 / 28.6 |
| noamraph             | 65  | 3       | 3     | 11.1 / 5.0 / 21.4  |
| Noam Yorav-Raphael   | 55  | 8       | 2     | 9.4 / 13.3 / 14.3  |
| arkottke             | 5   | 1       | 1     | 0.9 / 1.7 / 7.1    |
| stonebig             | 2   | 1       | 1     | 0.3 / 1.7 / 7.1    |
| Ford Hurley          | 0   | 1       | 0     | 0.0 / 1.7 / 0.0    |
| Jack McCracken       | 0   | 2       | 0     | 0.0 / 3.3 / 0.0    |
+----------------------+-----+---------+-------+--------------------+

[kmike]

What about extracting this to a helper function? Something like file_suports_unicode:

def _is_utf(encoding):
    return ('U8' == encoding) or ('utf' in encoding) or ('UTF' in encoding))

def file_supports_unicode(fp):
    if not getattr(fp, 'encoding', None):
        return False    
    return _is_utf(file.encoding)

[hadim]

+1

[casperdcl]

will leave it to you ;) i'm not pedantic about loop startup overhead so don't mind abstractions at this level.

[casperdcl]

haha @kmike rebase, sure. I've done it before -- #11 . But you still have to do your file_suports_unicode

[casperdcl]

nvm. was easier to correct than type chat message about it lol

[casperdcl]

going to sleep, then spend weekend at a data science competition hackathon (used nltk a LOT in the buildup stages, you might be interested to know, @kmike ). May rebase in a few days...