Runtime exception: Hash map is full with 100 keys. Should never happen.

[GoogleCodeExporter]

What steps will reproduce the problem?
1. building a LM over some input files consistently generates this exception
2.
3.

What is the expected output? What do you see instead?

The expected output is a learned LM written to a file. Instead, I get the 
exception:
Runtime exception: Hash map is full with 100 keys. Should never happen.


What version of the product are you using? On what operating system?
berkeleylm 1.1.3 on Windows 7

Please provide any additional information below.

java.lang.RuntimeException: Hash map is full with 100 keys. Should never happen.
    at edu.berkeley.nlp.lm.map.ExplicitWordHashMap.put(ExplicitWordHashMap.java:56)
    at edu.berkeley.nlp.lm.map.HashNgramMap.putHelpWithSuffixIndex(HashNgramMap.java:283)
    at edu.berkeley.nlp.lm.map.HashNgramMap.putWithOffsetAndSuffix(HashNgramMap.java:247)
    at edu.berkeley.nlp.lm.io.KneserNeyLmReaderCallback.addNgram(KneserNeyLmReaderCallback.java:171)
    at edu.berkeley.nlp.lm.io.KneserNeyLmReaderCallback.call(KneserNeyLmReaderCallback.java:148)
    at edu.berkeley.nlp.lm.io.KneserNeyLmReaderCallback.call(KneserNeyLmReaderCallback.java:37)
    at edu.berkeley.nlp.lm.io.TextReader.countNgrams(TextReader.java:80)
    at edu.berkeley.nlp.lm.io.TextReader.readFromFiles(TextReader.java:53)
    at edu.berkeley.nlp.lm.io.TextReader.parse(TextReader.java:47)
    at edu.berkeley.nlp.lm.io.LmReaders.createKneserNeyLmFromTextFiles(LmReaders.java:301)
    at edu.berkeley.nlp.lm.io.MakeKneserNeyArpaFromText.main(MakeKneserNeyArpaFromText.java:57)
    at yr.haifa.NLP.lm.BerkleyLanguageModel.train(BerkleyLanguageModel.java:51)

Original issue reported on code.google.com by jethro...@gmail.com on 24 Apr 2013 at 1:14

[GoogleCodeExporter]

Can you try building the latest version from SVN? I believe I have fixed this 
bug, but I was waiting for others to verify before releasing it. 

Original comment by adpa...@gmail.com on 24 Apr 2013 at 2:16

[GoogleCodeExporter]

Problem solved!

Original comment by jethro...@gmail.com on 28 Apr 2013 at 6:01

[GoogleCodeExporter]

Version 1.1.4 has been released.

Original comment by adpa...@gmail.com on 12 May 2013 at 12:59

• Changed state: Fixed

[GoogleCodeExporter]

I had this problem, replaced the 1.1.3 with 1.1.5 and I still have it. Any 
thoughts?

Original comment by benajiba...@gmail.com on 5 Dec 2013 at 8:23

[GoogleCodeExporter]

I'm also getting this bug on 1.1.5. Is there a chance it's related to input 
data?

Original comment by bryan.t....@gmail.com on 6 Dec 2013 at 1:56

[GoogleCodeExporter]

Can you find a small input file + command line that I can run myself to 
reproduce the error? 

Original comment by adpa...@gmail.com on 6 Dec 2013 at 4:01

[GoogleCodeExporter]

I'm getting the error in this relatively small file of Japanese text in 1.1.5.

Original comment by acgris...@gmail.com on 7 Mar 2014 at 10:12

Attachments:

• 指摘している。.context.text

[GoogleCodeExporter]

Also getting this in 1.1.5

Original comment by AWNyst...@gmail.com on 3 Jul 2014 at 6:47

[GoogleCodeExporter]

I also get in in 1.1.5

Original comment by arthurxl...@gmail.com on 5 Sep 2014 at 4:10

[GoogleCodeExporter]

I have added a fix in the latest from SVN. Could you all please try updating 
and seeing if your issues are fixed. I will release a new version once I've 
verified the issue is fixed for all.

Original comment by adpa...@gmail.com on 7 Sep 2014 at 7:00

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

jethro...@gmail.com, How you get to solve the problem? Also getting this in 
1.1.5. Thanks. 

Original comment by alexport...@gmail.com on 2 Nov 2014 at 10:59

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Problem solved..

Original comment by alexport...@gmail.com on 6 Nov 2014 at 6:56

[GoogleCodeExporter]

Hi, just to confirm: is the problem solved at head in SVN?

Original comment by adpa...@gmail.com on 6 Dec 2014 at 11:23

[GoogleCodeExporter]

Yes. I downloaded the latest version of SVN and it worked. Thank you.

Original comment by alexport...@gmail.com on 7 Dec 2014 at 12:46