Smaller memory footprint available?

[nlp12]

I am working with very large input files, (100kb-200kb) in Chinese (ZH) and am getting running out of memory exception. Is there any way to reduce the amount of memory required to run Corenlp in JAVA so that I can realistically run it without losing memory. Smaller files (20kb) work fine and only took 1 minute, but bulk of what I need run are very large files. Please help, thank you!

[AngledLuffa]

Can you reduce the size of the files? Which annotators are you using? The standard set? Do you use all of the annotations?

…

On Tue, Nov 23, 2021 at 12:31 PM nlp12 ***@***.***> wrote: I am working with very large input files, (100kb-200kb) in Chinese (ZH) and am getting running out of memory exception. Is there any way to reduce the amount of memory required to run Corenlp in JAVA so that I can realistically run it without losing memory. Smaller files (20kb) work fine and only took 1 minute, but bulk of what I need run are very large files. Please help, thank you! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1221>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWMCXJGODGF2UYVUMETUNP23BANCNFSM5IUNEDSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nlp12]

Yes, that's what we did for it to work, but was wondering if there was a different fix.

Just the standard set.

[AngledLuffa]

You could always figure out if there are any annotators you don't need and prune them from the list. That would be a good first step for saving memory

…

On Tue, Nov 23, 2021 at 5:39 PM nlp12 ***@***.***> wrote: Yes, that's what we did for it to work, but was wondering if there was a different fix. Just the standard set. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWLZKZU5HFBKXLXS7Z3UNQ65XANCNFSM5IUNEDSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nlp12]

Really I am just looking for a constituency parser separating phrases in my corpus, what would you recommend is the best command for this? Thank You!

[strongerfly]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

[AngledLuffa]

You can set annotators="tokenize,pos,parse" for that

…

On Tue, Dec 7, 2021, 8:51 AM nlp12 ***@***.***> wrote: Really I am just looking for a constituency parser separating phrases in my corpus, what would you recommend is the best command for this? Thank You! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWIUIZTVCWBLH5YV4KDUPY3QXANCNFSM5IUNEDSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nlp12]

Did that, however now we are dealing with some issues now. Hoping you can assist.

1. Is there any way to specify the level of the tree output as a tab sep. value? I want to get phrases at a specified level printed together in the same row, not separated.
2. The parse sometimes is not making sense, how to circumvent this issue? For example see attachment
   
3. Conll output is just the tokenization, how can I get specific parsed items thru command? For example, in my attachment, I just want the 2 NPs "五言" and "古诗" as such, but it prints it in conll as "五" "言" "古诗".
   Thank you!

[AngledLuffa]

"言 " is being used as a measure word here, right? I don't see anything wrong with the tokenization. The parse goes awry because there's no similar use of 言 as a measure word in the training data.

…

On Wed, Dec 8, 2021 at 1:47 PM nlp12 ***@***.***> wrote: Did that, however now we are dealing with some issues now. Hoping you can assist. 1. Is there any way to specify the level of the tree output as a tab sep. value? I want to get phrases at a specified level printed together in the same row, not separated. 2. The parse sometimes is not making sense, how to circumvent this issue? For example see attachment [image: Screen Shot 2021-12-08 at 3 42 24 PM] <https://user-images.githubusercontent.com/78334409/145289099-22478572-73b0-4613-85f1-9c41ddb56dfd.png> 3. Conll output is just the tokenization, how can I get specific parsed items thru command? For example, in my attachment, I just want the 2 NPs "五言" and "古诗" as such, but it prints it in conll as "五" "言" "古诗". Thank you! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWMLOBLCLICM5EG7OPLUP7G6FANCNFSM5IUNEDSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nlp12]

"五言" goes together, it refers to the syllable count per line of a style of poetry, nothing to do with measure word here.

The tokenization is fine, just looking to extract the specific levels of the constant tree thru command so I am grabbing phrases I want (e.g.,dependent gloss and governor gloss together on the same row)

[AngledLuffa]

Alright, my Chinese is not great so I'll take your word for it. Still, those two characters as a single word do not show up in the training data either. When you talk about dependent and governor glosses on the same line, are you actually more interested in dependency parsing or constituency parsing? There's also a Chinese dependency parser in CoreNLP (and a more accurate one in our python tool, for that matter).

[nlp12]

you are right, they are not a single word, so maybe this is a bad example. nevertheless, if "五言古诗" is split at constituency tree from top to bottom, its first "五言" “古诗” then "五" "言" "古诗", but not "五" "言古诗" like corenlp incorrectly split it.

I have had great success with python stanza and corenlp to get the dependency parser working splitting up words, but not larger units, that's why I turn to the constituency parser.

HOWEVER, I would like to just use the constituency parser so that I am only getting larger units (syntactic constituents), to be displayed as tab separated values.

MY question is, how can we get the constituents (e.g., subject NP and predicate VP) to be printed in the conll, rather than only being visible in the text output as a bracketed tree?
Thank you!

[AngledLuffa]

That makes sense. FWIW, I think this is just a bad example of something that doesn't exist in the training set. In terms of alternate output formats, it sounds like we don't have exactly what you're looking for, but you should be able to recreate it from the trees. Are you still working on the python side or are you using Java directly to get the constituency trees?

…

On Thu, Dec 9, 2021 at 8:23 AM nlp12 ***@***.***> wrote: you are right, they are not a single word, so maybe this is a bad example. nevertheless, if "五言古诗" is split at constituency tree from top to bottom, its first "五言" “古诗” then "五" "言" "古诗", but not "五" "言古诗" like corenlp incorrectly split it. I have had great success with python stanza and corenlp to get the dependency parser working splitting up words, but not larger units, that's why I turn to the constituency parser. HOWEVER, I would like to just use the constituency parser so that I am only getting larger units (syntactic constituents), to be displayed as tab separated values. MY question is, how can we get the constituents (e.g., subject NP and predicate VP) to be printed in the conll, rather than only being visible in the text output as a bracketed tree? Thank you! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWO3QTV4U5CPZQNZGOTUQDJZXANCNFSM5IUNEDSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nlp12]

We use both, but for the constituency trees, we are using JAVA.
What do you suggest we do so we can specify the levels in the constituency tree we want in the output as tab separated value?

[AngledLuffa]

The limitation here is that we don't know exactly what you're looking for. Are you only looking for the lowest NP, VP, etc nodes? In general, these nodes are nested (some brackets removed for readability): (NP more mail (PP than (NP (NP almost any other issue) (PP in memory)))) So, do you want all 3 NPs, or just the smallest one, or the biggest one, or ...? Another thought is, we can provide a more accurate Chinese model in Stanza (python) now, or we can provide an even more accurate one if you're okay with using Bert as part of a Stanza model. If that would be useful, let me know

…

On Fri, Dec 10, 2021 at 4:40 AM nlp12 ***@***.***> wrote: We use both, but for the constituency trees, we are using JAVA. What do you suggest we do so we can specify the levels in the constituency tree we want in the output as tab separated value? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1221 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWKC4UQB3EVCOPJ3OGDUQHYMLANCNFSM5IUNEDSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[nlp12]

I am wondering whether we can specify the level of the constituency parse we want so that the conll represents a specified level, for example in the example below, specifying the VP 无法集中, rather than what it is printed as (seperated out)

Yes, that would be useful to try out. Please let me know where to go and what to do to download this. Thank you!