BleiCorpus after serialize cannot be loaded

[vincentmajor]

Python version 2.7.12
gensim version 0.13.2

I'm serializing my corpus into Blei LDA-C format using corpora.BleiCorpus.serialize(filename, corpus) which is then later used in a dynamic topic model, and not in python. (I know I can use the DTMModel wrapper, unrelated.)

If I need to come back and load the corpus back into Python I tried corpora.BleiCorpus.load(filename), I get an unpickling error:

 ---------------------------------------------------------------------------
 UnpicklingError                           Traceback (most recent call last)
<ipython-input-28-0dd15393a941> in <module>()
----> 1 test = corpora.BleiCorpus.load('corpus-mult.dat')

/Applications/anaconda/envs/py27/lib/python2.7/site-packages/gensim/utils.pyc in load(cls, fname, mmap)
    246         compress, subname = SaveLoad._adapt_by_suffix(fname)
    247 
--> 248         obj = unpickle(fname)
    249         obj._load_specials(fname, mmap, compress, subname)
    250         return obj

/Applications/anaconda/envs/py27/lib/python2.7/site-packages/gensim/utils.pyc in unpickle(fname)
    909     with smart_open(fname) as f:
    910         # Because of loading from S3 load can't be used (missing readline in smart_open)
--> 911         return _pickle.loads(f.read())
    912 
    913 

UnpicklingError: invalid load key, '7'.

The only other argument to load() is mmap but I don't believe the arrays were stored separately and using mmap='r' doesn't help anyway.

[tmylk]

@vincentmajor Thanks for reporting it. Are you trying to load the model with 0.13.3 or older version? There is a know issue that is currently being worked on #1082

[vincentmajor]

@tmylk Yes, gensim version 0.13.2 on Python 2.7.12.
But, that issue referenced LDA models whereas I am serializing and loading a corpus without a model. I don't know if the same absence of a random_state variable influences loading of a corpus which should be random state invariant.

[tmylk]

Agree, LdaModel is not related to BleiCorpus. There haven't been any changes. Are you serializing and loading on the same platform, say Linux?

[vincentmajor]

Yep, OS X. It doesn't even work loading straight after serializing.

[tmylk]

do you have a small text and code example to reproduce?

[vincentmajor]

Not really, I'll generate one now.

[vincentmajor]

Okay here is what I am doing. Apologies about the nltk import.

import numpy as np
import pandas as pd

df = pd.DataFrame(["The quick brown fox jumps over the lazy dog.", "This is a test."], columns=["text"])
#df.head()

from nltk.tokenize import RegexpTokenizer
tokenizer = RegexpTokenizer(r'\w+') 

token_list = [tokenizer.tokenize(document) for document in df["text"]]
#print token_list

from gensim import corpora
dictionary = corpora.Dictionary(token_list)

## using the dictionary to convert from document to bag of words (doc2bow) encoding
corpus = [dictionary.doc2bow(document) for document in token_list]
## store the corpus into lda-c format, same as used by DTM,
corpora.BleiCorpus.serialize('~/Desktop/test_corpus-mult.dat', corpus)

## fails here
test_corpus = corpora.BleiCorpus.load('~/Desktop/test_corpus-mult.dat')

[pranaydeeps]

Hi, I am looking to fix this issue. Could I get some directions as to where these particular functions are implemented in the code base?
Thank you.

[pranaydeeps]

So I looked around and it seems that the load() function loads a pickle whereas the serialize() function calls save_corpus() which does not store a pickle at all but stores in plain text with a .vocab and a .index file if I am not wrong. The fix would be to either change the load() function to read the plain text format or the serialize function calls save() instead of save_corpus() as save() stores in pickle format.
Tell me if I am going wrong anywhere @tmylk

[pranaydeeps]

Everything works fine if instead of using the load() function you use the init method from BleiCorpus ie.
a = corpora.BleiCorpus('/home/artemis/Desktop/test_corpus-mult.dat')
This uses the same format to load as the save_corpus() function. So to sum it up, the save_corpus() is analogous to the default constructor of the class and load() function stores in pickle and is analogous to the save() method.

[piskvorky]

Does this mean the issue is resolved? Can we close?

@pranaydeep-af what would have helped you avoid this mistake? I mean, how would you change the documentation or API so that the behaviour is clear, avoiding the issue? Or at least finding the solution more quickly.

[pranaydeeps]

@piskvorky Maybe we can change the save() and load() functions to save_as_pickle() and load_pickle() so the behaviour is clear.
Also, if both ways to store are not needed. Maybe we can implement one common way to save and load?
Awaiting confirmation of my findings from @tmylk

[piskvorky]

-1 on save_pickle and load_pickle. The pickle format is only incidental. And it's not even true it's always pickle -- we already store (sub)attributes in various ways, such as direct .npy files to allow memory-mapping.

There is only one way to load/save objects in gensim (called .load() and .save()). Serializing corpora (lossy text formats) is not the same as serializing objects (binary lossless formats).