Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

[Mahhos]

Hi. Does (stanford-corenlp-full-2018-10-05) require the machine to have specific memory/computational capability?
I am using the Stanford sentiment toolkit (stanford-corenlp-full-2018-10-05) downloaded from Stanford CoreNLP website to do sentiment analysis on a text file. But, it does not work when I have for example 6000 lines (sentences) in the text file. However, It works fine with a 10 line text file. I am running a large scale study and need to get the sentiment of a very large amount of sentences.

[AngledLuffa]

If you do the sentences one at a time, it will save quite a bit of memory.

…

On Fri, Feb 7, 2020 at 5:00 PM Mahshid Hosseini ***@***.***> wrote: Hi. Does (stanford-corenlp-full-2018-10-05) require the machine to have specific memory/computational capability? I am using the Stanford sentiment toolkit (stanford-corenlp-full-2018-10-05) downloaded from Stanford CoreNLP website <https://stanfordnlp.github.io/CoreNLP/download.html> to do sentiment analysis on a text file. But, it does not work when I have for example 6000 lines (sentences) in the text file. However, It works fine with a 10 line text file. I am running a large scale study and need to get the sentiment of a very large amount of sentences. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#993?email_source=notifications&email_token=AA2AYWLHKTVTM4YGC5TL4XDRBX73DA5CNFSM4KRXEF52YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IL6SZ7A>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWNNF7CNYSV2EXHPMKTRBX73DANCNFSM4KRXEF5Q> .

[Mahhos]

Thanks for your reply. What is your suggestion? How should I get the sentiment of sentences on a large scale study? I followed the instructions in The Stanford CoreNLP Natural Language Processing Toolkit paper. They put each sentence in a line of a ".txt" file and have this file as input. I did the same. But it takes forever for a large number of sentences.
I saw other papers that used Stanford sentiment toolkit to run a large scale study. Is there anything that I should do?

[AngledLuffa]

The primary issue is that generally the Stanford tools treat a single text file as a single document. This is very useful for analyzing document-wide information, such as with coref, but makes it difficult to fit the whole thing in memory when looking at a large file you want to do data processing on. Instead of asking for a single file with the -file flag with .pipeline.StanfordCoreNLP, you can break your file into multiple pieces and use the -fileList flag. If you're using a command line which isn't StanfordCoreNLP to do the processing, please let us know so we can give the correct advice for that command line.

…

On Fri, Feb 7, 2020 at 5:29 PM Mahshid Hosseini ***@***.***> wrote: Thanks for your reply. What is your suggestion? How should I get the sentiment of sentences on a large scale study? I followed the instructions in The Stanford CoreNLP Natural Language Processing Toolkit paper. They put each sentence in a line of a ".txt" file and have this file as input. I did the same. But it takes forever for a large number of sentences. I saw other papers that used Stanford sentiment toolkit to run a large scale study. Is there anything that I should do? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#993?email_source=notifications&email_token=AA2AYWISHPXFYT7OJQJPILTRBYDHBA5CNFSM4KRXEF52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELFE74Q#issuecomment-583684082>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWICQXHKDALTDVW6XTTRBYDHBANCNFSM4KRXEF5Q> .

[Mahhos]

Thanks for your response. I am specifically running this command java -cp "*" -Xmx8g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,parse,sentiment -file myfile.txt (as suggested in the main paper) to get the sentiments of the sentences existing in myfile.txt. I am running this command on windows command line (which isn't StanfordCoreNLP).

By using -fileList flag, you mean I need to first break my file into multiple smaller files, then run the above command with the -fileList flag? The problem is that I have already broken my file into 20 smaller files (my original file includes 127K sentences where I broke it to 20 files with almost 6000 sentences each). However, it still cannot process the file and output the sentiments in the XML file.
It would be great if you could advise me on this.

[AngledLuffa]

Do you have more than 8g available on your system? In that case, perhaps more memory is the answer. Smaller file pieces will help, too. Long sentences are also an issue, especially since it seems you are using the PCFG parser with that command line. (Switching to the beam based SRParser will save some memory in the case of long sentences - look for -parse.model flag)

…

On Mon, Feb 10, 2020 at 11:36 AM Mahshid Hosseini ***@***.***> wrote: Thanks for your response. I am specifically running this command java -cp "*" -Xmx8g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,parse,sentiment -file myfile.txt (as suggested in the main paper) to get the sentiments of the sentences existing in myfile.txt. I am running this command on windows command line (which isn't StanfordCoreNLP). By using -fileList flag, you mean I need to first break my file into multiple smaller files, then run the above command with the -fileList flag? The problem is that I have already broken my file into 20 smaller files (my original file includes 127K sentences where I broke it to 20 files with almost 6000 sentences each). However, it still cannot process the file and output the sentiments in the XML file. It would be great if you could advise me on this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#993?email_source=notifications&email_token=AA2AYWIRCYXNXCF7Q3OTAIDRCGUETA5CNFSM4KRXEF52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELJ6JSY#issuecomment-584312011>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWOCY2ECTXKT2KZCXT3RCGUETANCNFSM4KRXEF5Q> .

[Mahhos]

I also tried to allocate 12g and still did not get the final XML file including the sentiments. Is there any documentation on the flags that I can look into and find some info on -parse.model? Or if you could let me know how to change my command to the beam based SRParser I would appreciate it.

[AngledLuffa]

You can download the latest English sr parser models here: http://nlp.stanford.edu/software/stanford-english-corenlp-2018-10-05-models.jar This should be roughly correct:

…

-parse.model edu.stanford.nlp.models.srparser.englishSR.beam.ser.gz but I still encourage you to break the file into smaller pieces if needed

On Mon, Feb 10, 2020 at 12:00 PM Mahshid Hosseini ***@***.***> wrote: I also tried to allocate 12g and still did not get the final XML file including the sentiments. Is there any documentation on the flags that I can look into and find some info on -parse.model? Or if you could let me know how to change my command to the beam based SRParser I would appreciate it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#993?email_source=notifications&email_token=AA2AYWPMZBPQJSI625TYU7DRCGW5HA5CNFSM4KRXEF52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELKA5OY#issuecomment-584322747>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWO3ZGU45E367DSJQK3RCGW5HANCNFSM4KRXEF5Q> .

[Mahhos]

So the only thing that I need to do is to put this jar file into my "stanford-corenlp-full-2018-10-05" folder and run the same command again. right?

[AngledLuffa]

Yes, although you need to specify -parse.model as described. It is also worth noting that this should fix the problem of long sentences running out of memory, but not long documents. In which case, splitting the file will still be necessary.

…

On Mon, Feb 10, 2020, 12:34 PM Mahshid Hosseini ***@***.***> wrote: So the only thing that I need to do is to put this jar file into my "stanford-corenlp-full-2018-10-05" folder and run the same command again. right? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#993?email_source=notifications&email_token=AA2AYWLWR3HBIQT2XVSXCDDRCG26FA5CNFSM4KRXEF52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELKFJ5A#issuecomment-584340724>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWK7LULQIQVVQBQ43OLRCG26FANCNFSM4KRXEF5Q> .

[Mahhos]

What should I put as the value of -parse.model flag?
I put -parse.model SRParser and it seems it cannot detect the parser: java.io.IOException: Unable to open "SRParser" as class path, filename or URL .

[AngledLuffa]

I think I already posted the expected flag, -parse.model, but I realize now I had put . instead of / for the path separators

…

-parse.model edu/stanford/nlp/models/srparser/englishSR.beam.ser.gz

On Mon, Feb 10, 2020 at 12:44 PM Mahshid Hosseini ***@***.***> wrote: What should I put as the value of -parse.model flag? I put -parse.model SRParser and it seems it cannot detect the parser: java.io.IOException: Unable to open "SRParser" as class path, filename or URL . — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#993?email_source=notifications&email_token=AA2AYWPVX35AEKI6OMM7MKLRCG4DZA5CNFSM4KRXEF52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELKGL5A#issuecomment-584345076>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWIUN64NAO3SUKYJMX3RCG4DZANCNFSM4KRXEF5Q> .

[Mahhos]

Now the command is running. Hope this helps to get the final output. Thank you so much for your help. I appreciate it.