python gensim 不能加载词向量文件

[sudazzk]

D:\Program\Anaconda3\lib\site-packages\gensim\utils.py:860: UserWarning: detected Windows; aliasing chunkize to chunkize_serial
warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")
Traceback (most recent call last):
File ".\zzk_word2vec.py", line 101, in
test_word_embedding('D:\data\pretrain_word2vec\Chinese-Word-Vectors\sgns.zhihu.char\sgns.zhihu.char')
File ".\zzk_word2vec.py", line 76, in test_word_embedding
model = gensim.models.KeyedVectors.load_word2vec_format(vector_file, binary=False, encoding='utf8')
File "D:\Program\Anaconda3\lib\site-packages\gensim\models\keyedvectors.py", line 250, in load_word2vec_format
parts = utils.to_unicode(line.rstrip(), encoding=encoding, errors=unicode_errors).split(" ")
File "D:\Program\Anaconda3\lib\site-packages\gensim\utils.py", line 242, in any2unicode
return unicode(text, encoding, errors=errors)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 96-97: invalid continuation byte

[sudazzk]

用这个函数加载 知乎问答数据 sgns.zhihu.char
model = gensim.models.KeyedVectors.load_word2vec_format(vector_file, binary=False, encoding='utf8')
会报上面的错误

[pyanh]

@sudazzk 我也是用这个函数加载，可以正常运行。

from gensim.models.keyedvectors import KeyedVectors
w2v_model = KeyedVectors.load_word2vec_format("Chinese-Word-Vectors/sgns.zhihu.char",binary=False,unicode_errors='ignore')
你把最后一个参数设置成unicode_errors='ignore'

[shenshen-hungry]

我更新了词向量，应该不会有unicode编码问题了。
如果还有问题，可以在打开的时候加参数：

open(filename, errors='ignore')

[jly8866]

还不错哦

[jly8866]

from gensim.models.keyedvectors import KeyedVectors
w2v_model = KeyedVectors.load_word2vec_format("Chinese-Word-Vectors/sgns.zhihu.char",binary=False,unicode_errors='ignore')
你把最后一个参数设置成unicode_errors='ignore'

这一句靠谱