Populate with Python 3.6: "TypeError: expected bytes, str found"

[vered1986]

Hi,

I'm trying to load parameters of a model using populate:

w.populate(load_from, '/w')

where w is a parameter and load_from is the file path string.

I get the following exception:

  File "_dynet.pyx", line 563, in _dynet.Parameters.populate
  File "_dynet.pyx", line 575, in _dynet.Parameters.populate_from_textfile
  File "stringsource", line 15, in string.from_py.__pyx_convert_string_from_py_std__in_string
TypeError: expected bytes, str found

I saved the parameters using the following code:

w.save(save_to, '/w')

where save_to is the same string as load_from.

I work with python 3.6 and I suspect that the exception stems from the difference between strings in python 2 and 3: http://docs.cython.org/en/latest/src/tutorial/strings.html.

Thanks!

[vered1986]

It seems that removing the string requirement for the arguments of populate_from_textfile solves the problem in python3.6 (i.e. I defined it as def populate_from_textfile(self, fname, key=""), similarly to the way it is defined for ParameterCollection and LookupParameters).

[neubig]

Ahh, yes, this is a common problem. Would you mind sending a pull request fixing the problem quickly?

@xunzhang I'm not sure why our unit tests didn't catch this though. Perhaps we need to add an IO unit test to Python?

[vered1986]

Sure, but I haven't tested it on python 2... I guess it should be fine assuming that ParameterCollection.populate_from_textfile and LookupParameters.populate_from_textfile work there.

[xunzhang]

I added some tests in the above pull request. But I don't have a python3 environment, @vered1986 could you help test it?

[vered1986]

Sure, what should I run? (I guess this? https://github.com/clab/dynet/pull/1189/files)

[xunzhang]

Yes, https://raw.githubusercontent.com/xunzhang/dynet/5b6cf0e0b5f76ab534943067676ab927c4ca6107/tests/python/test.py . You should get an error with current master and pass it with the modification of your pull request.

[vered1986]

Yes, works as expected.

Output for master:

[dynet] random seed: 267491601
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
....Reading clusters from cluster_file.txt ...
Read 10 words in 5 clusters (0 singleton clusters)
...........E........................
======================================================================
ERROR: test_save_load (__main__.TestIOPartial)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 311, in test_save_load
    a.populate(self.file, "/a")
  File "_dynet.pyx", line 563, in _dynet.Parameters.populate
  File "_dynet.pyx", line 575, in _dynet.Parameters.populate_from_textfile
  File "stringsource", line 15, in string.from_py.__pyx_convert_string_from_py_std__in_string
TypeError: expected bytes, str found

----------------------------------------------------------------------
Ran 40 tests in 0.056s

FAILED (errors=1)

Output for pull request:

[dynet] random seed: 3442494103
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
....Reading clusters from cluster_file.txt ...
Read 10 words in 5 clusters (0 singleton clusters)
....................................
----------------------------------------------------------------------
Ran 40 tests in 0.067s

OK

[xunzhang]

looks good

[xunzhang]

Fixed via #1188.