Fix numpy pickle for versions less than 1.9

[lesteve]

Fix #236. There were two bugs conflated in one:

Python 3 writer, Python 2 reader

That's the direction reported in #236. Standard pickle of dtype created with python 3 and numpy 1.9 can not be read with python 2.

Writer script (python 3, numpy 1.9)

import pickle
import numpy as np
dt = np.dtype('>f8')
with open('/tmp/test.pkl', 'wb') as f:
    pickle.Pickler(f, protocol=2).dump(dt)

Reader script (python 2, numpy 1.8)

import pickle
pickle.load(open('/tmp/test.pkl', 'rb'))

Traceback:

Traceback (most recent call last):
  File "/home/lesteve/miniconda3/envs/scratch_py27/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/home/lesteve/miniconda3/envs/scratch_py27/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/lesteve/dev/joblib/joblib/test/test_numpy_pickle.py", line 324, in test_compressed_pickle_python_2_3_compatibility
    result_list = numpy_pickle.load(fname)
  File "/home/lesteve/dev/joblib/joblib/numpy_pickle.py", line 524, in load
    obj = unpickler.load()
  File "/home/lesteve/miniconda3/envs/scratch_py27/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/home/lesteve/dev/joblib/joblib/numpy_pickle.py", line 384, in load_build
    Unpickler.load_build(self)
  File "/home/lesteve/miniconda3/envs/scratch_py27/lib/python2.7/pickle.py", line 1217, in load_build
    setstate(state)
TypeError: must be char, not unicode

Python 2 writer, Python 3 reader

Related to NumpyPickler/NumpyUnPickler cycle. The likely culprit is that we use encoding=bytes for performance reasons, but that also means that all python2 strings are loaded as bytes in python3. This seems a bit iffy to say the least ... @GaelVaroquaux @ogrisel any opinions ?

Writer script (python 2, numpy 1.9)

import joblib
pickler = joblib.numpy_pickle.NumpyPickler('/tmp/test.pkl')
import numpy as np
pickler = joblib.numpy_pickle.NumpyPickler('/tmp/test.pkl')
pickler.save(np.dtype('f8'))
# Don't forget to quit ipython here in order to flush to file ...

Reader script (python 3, numpy 1.8)

import joblib
unpickler = joblib.numpy_pickle.NumpyUnpickler('/tmp/test.pkl', open('/tmp/test.pkl', 'rb'))
unpickler.load()

Traceback

Traceback (most recent call last):
  File "/volatile/le243287/miniconda3/envs/scratch/lib/python3.4/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/volatile/le243287/miniconda3/envs/scratch/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "/volatile/le243287/miniconda3/envs/scratch/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/le243287/dev/joblib/joblib/test/test_numpy_pickle.py", line 325, in test_compressed_pickle_python_2_3_compatibility
    result_list = numpy_pickle.load(fname)
  File "/home/le243287/dev/joblib/joblib/numpy_pickle.py", line 524, in load
    obj = unpickler.load()
  File "/volatile/le243287/miniconda3/envs/scratch/lib/python3.4/pickle.py", line 1038, in load
    dispatch[key[0]](self)
  File "/home/le243287/dev/joblib/joblib/numpy_pickle.py", line 384, in load_build
    Unpickler.load_build(self)
  File "/volatile/le243287/miniconda3/envs/scratch/lib/python3.4/pickle.py", line 1505, in load_build
    setstate(state)
TypeError: must be a unicode character, not bytes

[lesteve]

The likely culprit is that we use encoding=bytes for performance reasons, but that also means that all python2 strings are loaded as bytes in python3. This seems a bit iffy to say the least ... @GaelVaroquaux @ogrisel any opinions ?

Full disclosure: at the moment the python 2/3 compatibility for non compressed pickles is actually broken because of this iffyness. An instance of NDArrayWrapper ends up with keys that are bytes rather than str.

[lesteve]

This has been superseded by #249.