Dealing with typeError: must be str, not bytes

[FMCumhaill]

Dealing with typeError: must be str, not bytes which occurred for me when running this file. https://stackoverflow.com/questions/5512811/builtins-typeerror-must-be-str-not-bytes

This may be a local requirement and therefore not sufficient for a pull request?

[piskvorky]

All I/O writing or reading should definitely be happening in binary mode (rb, wb). With explicit encoding conversions for text where necessary. This is true for both Python 2 and Python 3.

@loretoparisi was there a particular reason for the w+? Can we replace it with wb?

One more request: @FMCumhaill can you please replace all calls to open() with smart_open()? (from smart_open import smart_open)

The reason is that in Gensim, we're trying to abstract away from the storage, so people can save files to S3, HDFS, compressed etc transparently, using the smart_open library.

[FMCumhaill]

@piskvorky done as requested! 👍

[piskvorky]

No w -- always use binary I/O (rb, wb) please. Encode text to utf8 where necessary.

[FMCumhaill]

Changing that w to wb gives me the following error:

Traceback (most recent call last):
File "word2vec2tensor.py", line 94, in
word2vec2tensor(args.input, args.output, args.binary)
File "word2vec2tensor.py", line 76, in word2vec2tensor
file_vector.write(vector_row + '\n')
TypeError: a bytes-like object is required, not 'str'

[piskvorky]

Yes -- please encode text to binary (utf8) where necessary.

[menshikh-iv]

please add a test for word2vec2tensor function (for avoid this issue in future)

[menshikh-iv]

must be wb

[menshikh-iv]

also, good idea to make a code more "flat", like this

with smart_open(...) as file_vector, smart_open(...) as file_metadata:
    ...

[FMCumhaill]

@menshikh-iv sorry this escaped me! ATM I don't have time to finish it sorry. I have no idea how to add a test and don't have time to research it. My apologies!

[menshikh-iv]

@FMCumhaill That's sad, but anyway, thanks for the answer, good luck!

[menshikh-iv]

@FMCumhaill thanks for PR, do you plan to finish it?

[menshikh-iv]

Close by #2125 (comment) reason

[piskvorky]

@menshikh-iv this is a bug, please don't close like this. We need to fix this either way (and the fix looks really trivial, including smart_open).

@loretoparisi was there a particular reason for the w+? Can we replace it with wb?

[menshikh-iv]

@piskvorky we need fix bugs, I agree, but this is PR, not a issue, if contributor don't have time to complete (and isn't close to finish) - I close it without "almost complete" badge.

Also, we already have opened issue for this bug #1958

About "amost_complete" - I don't agree, because:

• incorrect file modes
• missing tests
• 3 lines diff where correct only one (import), I don't think that we should mark it with badge

[piskvorky]

OK. Let me remove the label.