Training on Windows returns error at DFfeatureselect.py step #54

jd-coderepos · 2016-05-16T15:36:22Z

I'm trying to train a new language identifier model on my own languages dataset. Unfortunately, it crashes at the DFfeatureselect.py script, returning "TypeError: marshal.load() arg must be file" error message. Below is the log until the crash point.

C:\langid.py-master\langid\train>C:\Python27\python.exe train.py corpus
corpus path: corpus
model path: ..model
langs(22): el(26) eo(42) en(1674) af(285) ca(287) am(2426) an(226) cy(79) ar(82) cs(432) et(449) az(534) es(457) be(292) bg(818) bn(65) de(2795) da(90) dz(220) br(532) bs(493) as(101)
domains(1): domain(12405)
identified 12405 documents
will tokenize 12405 documents
using byte NGram tokenizer, max_order: 4
chunk size: 50 (249 chunks)
job count: 8
whole-document tokenization
tokenized chunk [21140 keys]
Traceback (most recent call last):
File "train.py", line 196, in
doc_count = tally(b_dirs, args.jobs)
File "C:\langid.py-master\langid\train\DFfeatureselect.py", line 92, in tally
for i, keycount in enumerate(pass_sum_df_out):
File "C:\Python27\lib\multiprocessing\pool.py", line 620, in next
raise value
TypeError: marshal.load() arg must be file

zq2017 · 2017-12-04T01:39:56Z

Hello, I met the same problem as you. Have you solved this problem? Can you give me some solutions? Thank you so much!

zq2017 · 2017-12-05T05:40:20Z

@saffsd Dear Marco, can you give me some solutions? Thank you so much!

jd-coderepos · 2018-03-20T17:51:31Z

Hello,

Sharing my workaround to this problem here. The problem is related to this StackOverflow question: https://stackoverflow.com/questions/3249822/python-2-6-4-marshal-load-doesnt-accept-open-file-objects-made-with-subprocess

In the file common.py, I changed the 'unmarshal_iter' function as below:

def unmarshal_iter(path):
  tmpfolder = tempfile.mkdtemp()
  try:
      tmpfile = os.path.join(tmpfolder, "temp")
      with open(tmpfile, "wb") as binfile:
          binfile.write(gzip.open(path, 'rb').read())

      with open(tmpfile, "rb") as binfile:
          while True:
              try:
                  yield marshal.load(binfile)
              except EOFError:
                  break
  finally:
      if tmpfolder and os.path.isdir(tmpfolder):
          shutil.rmtree(tmpfolder)

Also, I'm using python version 2.7.3 to run the program.

Best.

antoniogois · 2018-05-16T23:14:45Z

@jenlindadsouza thanks for your tip! Don't forget to also add "import shutil" to the beginning of common.py

And one more thing, in case anyone else has an error in "NBtrain.py", line 256: What I did was change line 257
except NameError:
by
except:

I think some exception that was supposed to be ignored has a different name now, so with my change all exceptions are ignored here. I'm not sure about this fix, but it seems to be working. (at first this seemed like a windows path issue [using / instead of \] but I had this problem both on windows and linux)

EDIT: forget about this exception thing, all predictions are going crazy... There's probably a real problem behind the exception being thrown, I can't find it right now

jd-coderepos closed this as completed Mar 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Training on Windows returns error at DFfeatureselect.py step #54

Training on Windows returns error at DFfeatureselect.py step #54

jd-coderepos commented May 16, 2016

zq2017 commented Dec 4, 2017

zq2017 commented Dec 5, 2017

jd-coderepos commented Mar 20, 2018

antoniogois commented May 16, 2018 •

edited

Loading

Training on Windows returns error at DFfeatureselect.py step #54

Training on Windows returns error at DFfeatureselect.py step #54

Comments

jd-coderepos commented May 16, 2016

zq2017 commented Dec 4, 2017

zq2017 commented Dec 5, 2017

jd-coderepos commented Mar 20, 2018

antoniogois commented May 16, 2018 • edited Loading

antoniogois commented May 16, 2018 •

edited

Loading