Add possibility to check unicode symbols

[Derfirm]

Added a label with unicode letters and a test for it.
Unicode is going like this:

import sys
import unicodedata
from collections import defaultdict

unicode_category = defaultdict(list)
for c in map(chr, range(sys.maxunicode + 1)):
    unicode_category[unicodedata.category(c)].append(c)

And from there on out use that map to translate back to a series of characters for a given category:

alphabetic = unicode_category['Ll']

[snguyenthanh]

1. Can you please explain why only category Ll is used to map the characters ?

2. Moreover, I have the following sentence: Эффекти́вного противоя́дия от я́да фу́гу не существу́ет до сих пор.
   Added a word противоя́дия to the wordlist, the word fails to be censored. Is this an expected behavior?

[Derfirm]

Ll is Letter Lowercase, so yes, I deliberately cut off special characters
Do they really need?

[Derfirm]

In your case (U+0430) + (U+0301) (as example) special character is used, called "Combining Diacritical Marks", which is always called before the letter. Is it worth expanding the list of characters for this case?

[snguyenthanh]

May be we should include category Lu as well.
I think the problem is not the list of characters, but the way the words merge together.

For example, blue tree is a word with an empty space as the separator. It should treat противоя́дия as 1 word with ́ as the separator as well.

[Derfirm]

` (U+0027) «APOSTROPHE» not include as Mn | Mark, Non-Spacing or Mc | Mark, Spacing Combining

[snguyenthanh]

I just pushed a commit to a new branch development, changing the way the words are merged together.

Currently in master branch, the word hand_job is treated as 1 word hard_job. The newly pushed commit, in development, treats it as 2 words hand and job. This allows a better and more stable comparison between the words.

I tested and it works well for Unicode characters. I will merge this into branch development and add a few more test cases before merging into master.

[snguyenthanh]

May I ask how you generated the alphabetic_unicode.json file ?
As I see, the characters are from categories Mn, Lu, Po, Ll, Lo, Cn, Mc.

[Derfirm]

import sys
import unicodedata
import json

unicode_category = defaultdict(list) 

for c in map(chr, range(sys.maxunicode + 1)): 
    unicode_category[unicodedata.category(c)].append(c) 

with open("alphabetic_unicode.json", "w") as js: 
    js.write(json.dumps(unicode_category['Ll'] + unicode_category['Lu'] + unicode_category['Mc'] + unicode_category['Mn'])) 

[snguyenthanh]

Thanks.
I created a beta release for Unicode support, which can be installed by:

$ pip install better-profanity==0.3b0

Please let me know if it has any issues.