Cannot load gzipped ARPA files

[GoogleCodeExporter]

MITLM tools cannot load gzipped ARPA LM files, even those produced by
estimate-ngram or interpolate-ngram.

This is what happens:

$ ~/lbin/mitlm-svn/evaluate-ngram --read-lm tmp.arpa.gz
--evaluate-perplexity dev.txt 
Loading LM tmp.arpa.gz...
terminate called after throwing an instance of 'std::invalid_argument'
  what():  Unexpected file format.


Backtrace from gdb:
(gdb) bt
#0  0x00000035c102ee25 in raise () from /lib64/libc.so.6
#1  0x00000035c1030770 in abort () from /lib64/libc.so.6
#2  0x00000035c27c0f74 in __gnu_cxx::__verbose_terminate_handler () from
/usr/lib64/libstdc++.so.6
#3  0x00000035c27bf0b6 in std::set_unexpected () from /usr/lib64/libstdc++.so.6
#4  0x00000035c27bf0e3 in std::terminate () from /usr/lib64/libstdc++.so.6
#5  0x00000035c27bf1ca in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00000000004181ac in NgramModel::LoadLM (this=0x5adff0,
probVectors=@0x7fff7925fc28, bowVectors=@0x7fff7925fc40,
lmFile=@0x7fff7925fe50)
    at src/NgramModel.cpp:289
#7  0x0000000000426d1a in ArpaNgramLM::LoadLM (this=0x7fff7925fc10,
lmFile=@0x7fff7925fe50) at src/NgramLM.cpp:141
#8  0x000000000046c38f in main (argc=5, argv=0x7fff79260118) at
src/evaluate-ngram.cpp:150


I'm using MITLM from SVN, Linux, amd64.
I attached the tmp.arpa.z file (produced with estimate-ngram)

Loading uncompressed ARPA files works fine.

Original issue reported on code.google.com by alu...@gmail.com on 9 Dec 2008 at 1:30

Attachments:

• tmp.arpa.gz

[GoogleCodeExporter]

- fseek() does not work on compressed files opened using processOpen().
- Implemented ZFile.ReOpen() to replace fseek().
- Modified signature of ZFile parameters from const ZFile & to ZFile & to allow 
ReOpen().
- Tweaked calls with ZFile arguments to explicitly create ZFile variable.

Original comment by bojune...@gmail.com on 9 Dec 2008 at 4:19

• Changed state: Fixed

[GoogleCodeExporter]

You are fast, thank you.

Original comment by alu...@gmail.com on 9 Dec 2008 at 4:53