Random UTF-8 string generation must generate valid string

[elyezer]

FauxFactory have generated the following string:

u'\u50dd\uf3b6\u1365\u4e5c\u2441\ue94f\u6b18\u4921\u6fc3\u052c'

But \ue94f [1] and \uf3b6 [2] are not valid unicode characters. FauxFactory need a better way to generate random unicode strings.

Thanks @Ichimonji10 for helping in the investigation process.

[1] http://www.fileformat.info/info/unicode/char/e94f/index.htm
[2] http://www.fileformat.info/info/unicode/char/f3b6/index.htm

[Ichimonji10]

👍

As it turns out, there are several versions of the unicode standard. This has practical implications. For example, Python's unicodedata module is pegged to unicode version 5.2.0 in Python 2.7 and unicode version 6.1.0 in Python 3.3.

[JacobCallahan]

I wonder if the differences between the versions are significant enough to warrant a change to the way we generate unicode. We could possibly accept custom unicode ranges and have a default range that is safe with both versions.

[elyezer]

Section 5.5.1 of [1] shows a table with character categories. We can use that to guide the generation, the character category can be easily get:

>>> for c in u'\u50dd\uf3b6\u1365\u4e5c\u2441\ue94f\u6b18\u4921\u6fc3\u052c':
...   print unicodedata.category(c)
...
Lo
Co
Po
Lo
So
Co
Lo
Lo
Lo
Cn

As you can see, some character of the "invalid" unicode string have Private_Use, Unassigned and Other_Punctuation characters.

I have excluded those categories from the string:

>>> print ''.join([c for c in u'\u50dd\uf3b6\u1365\u4e5c\u2441\ue94f\u6b18\u4921\u6fc3\u052c' if unicodedata.category(c) not in ('Co', 'Cn', 'So', 'Po')])
僝乜欘䤡濃

And the system accepted the new string as a valid data.

[1] http://www.unicode.org/reports/tr44/tr44-4.html

[Ichimonji10]

Good thought, elyezer.

Searching pypi for "unicode" turns up quite a number of results. There may be something there we can (fork and) use.