ENH: Allow dtype field names to be ascii encoded unicode in Python2

[chrisjbillington]

(#2407)

Allow unicode names in record arrays when datatype specified as tuples in Python 2.
Name is encoded as ascii if possible, raising an exception if not.

The result passes round-tripping with pickle

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals, print_function
import numpy as np
import cPickle as pickle

x = np.zeros(1, dtype=[('test', int)])
print('created array:', repr(x))

y = pickle.loads(pickle.dumps(x))
print('round-tripped pickle:', repr(y))
assert x == y

print('should raise an error now...')
x = np.zeros(1, dtype=[('tëßt', int)])

Result:

created array: array([(0,)], dtype=[('test', '<i8')])
round-tripped pickle: array([(0,)], dtype=[('test', '<i8')])
should raise an error now...
Traceback (most recent call last):
  File "12.py", line 18, in <module>
    x = np.zeros(1, dtype=[('tëßt', int)])
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-2: ordinal not in range(128)

[eric-wieser]

This is PyBaseString_Check(name) (from npy_3kcompat.h)

[eric-wieser]

Syntax error

[eric-wieser]

Not safe - you need to convert before you decref

[eric-wieser]

The approach looks good, just some implementation details to fix.

[chrisjbillington]

Thanks for the comments! I've made the changes.

[mhvk]

(No review, just changed the title to be meaningful.)

[eric-wieser]

Nit: could combine these lines

[eric-wieser]

Why does this fail on py3 too?

[chrisjbillington]

I'm not completely sure what you're asking, but this part of test_multiarray.py is within an if statement such that it only is defined on Python 2, if that answers your question.

[eric-wieser]

Got it. I would have expected a @skipIf decorator for that, but that's cleanup for another time.

[eric-wieser]

Implementation and test looks correct to me.

I've leave it to someone else to decide whether we want to fix #2407 at all - I think I'm +0.75 on it.

This maybe needs a release note, but I'd be ok without one. Again, will let someone else decide.

[mhvk]

I am in favour of the fix: this particular bug meant that people where discouraged from using unicode_literals in their code, and thus made making code for both python2 and 3 just that little bit more annoying. It is good to remove that annoyance even if it is nearly not needed any more.

Given the above, I think a release note is a good idea.

[chrisjbillington]

Thanks very much! I look forward to removing my workarounds for this once 1.15 is out.

[pv]

As in the earlier discussion, I'm also in favor.

[charris]

@chrisjbillington Needs a release note in `doc/release/1.15.0-notes.rst' under Enhancements.

[chrisjbillington]

Release note added!

[charris]

Thanks @chrisjbillington .