Unicode error in python2.7

[bajibabu]

Hi,
I got the following error while running char_rnn_classification_tutorial.py

print(unicodeToAscii('Ślusàrski'))
 File "src/char_rnn.py", line 17, in unicodeToAscii
   c for c in unicodedata.normalize('NFD', s)
TypeError: normalize() argument 2 must be unicode, not str

To circumvent the error I changed the code:

def unicodeToAscii(s):
    try:
        s = unicode(s, 'utf-8')
    except NameError:
        pass
    s = unicodedata.normalize('NFD', s)
    s = s.encode('ascii', 'ignore')
    s = s.decode('utf-8')
    return str(s)

[chsasank]

I think your implementation doesn't work on greek characters if I understand correctly. Also the n_letters is used across the code.

This might be a better choice:

# Turn a Unicode string to plain ASCII, thanks to http://stackoverflow.com/a/518232/2809427
def unicodeToAscii(s):
    try:
        s = unicode(s, 'utf-8')
    except NameError:
        pass

    return u''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
        and c in all_letters
    )

It'd be great if you can raise a PR with this.

[spro]

Another option is to add

from __future__ import unicode_literals

as the first import (Python 2.6 +)

[chsasank]

But will that work? lines = open(filename).read().strip().split('\n') will still be a list of str not uncode objects

[bajibabu]

I raised PR with the proposed modifications

[spro]

@chsasank You're right, the files would also need to be decoded:

    lines = open(filename).read().decode('utf8').strip().split('\n')

Either way works, I just like the one with fewer extra lines :)

In the future I will probably use unidecode