How to start a CoreNLP Server to segment Chinese text?

[Gwill]

I started a CoreNLP Server with:

java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -props StanfordCoreNLP-chinese.properties -port 9000

and

wget --post-data '直观地表明这种数据结构如何简化应用程序的处理时间和复杂性。' 'localhost:9000/?properties={"annotators": "tokenize,ssplit,pos","outputFormat": "json"}' -O -

the response is:

{"sentences":[{"index":0,"parse":"SENTENCE_SKIPPED_OR_UNPARSABLE","basic-dependencies":[{"dep":"ROOT","governor":0,"governorGloss":"ROOT","dependent":2,"dependentGloss":"。"},{"dep":"compound","governor":2,"governorGloss":"。","dependent":1,"dependentGloss":"直观地表明这种数据结构如何简化应用程序的处理时间和复杂性"}],"collapsed-dependencies":[{"dep":"ROOT","governor":0,"governorGloss":"ROOT","dependent":2,"dependentGloss":"。"},{"dep":"compound","governor":2,"governorGloss":"。","dependent":1,"dependentGloss":"直观地表明这种数据结构如何简化应用程序的处理时间和复杂性"}],"collapsed-ccprocessed-dependencies":[{"dep":"ROOT","governor":0,"governorGloss":"ROOT","dependent":2,"dependentGloss":"。"},{"dep":"compound","governor":2,"governorGloss":"。","dependent":1,"dependentGloss":"直观地表明这种数据结构如何简化应用程序的处理时间和复杂性"}],"openie":[],"tokens":[{"index":1,"word":"直观地表明这种数据结构如何简化应用程序的处理时间和复杂性","originalText":"直观地表明这种数据结构如何简化应用程序的处理时间和复杂性","lemma":"直观地表明这种数据结构如何简化应用程序的处理时间和复杂性","characterOffsetBegin":0,"characterOffsetEnd":28,"pos":"NN","ner":"O","speaker":"PER0","before":"","after":""},{"index":2,"word":"。","originalText":"。","lemma":"。","characterOffsetBegin":28,"characterOffsetEnd":29,"pos":"SYM","ner":"O","speaker":"PER0","before":""}

it is not work.

[J38]

I think you need to send all of the properties in StanfordCoreNLP-chinese.properties in the json in your wget request. The issue is currently the server doesn't seem to work with custom annotators, and currently Chinese segmentation is handled by calling a custom annotator. Off the top of my head I don't know why we haven't added "segment" --> "ChineseSegmenterAnnotator" in our list of standard annotators. I will look into this more and I think we should be able to push a small fix for this so the server can be used on Chinese text. Thanks for raising the issue! I will post updates about this here!

[stellasia]

Hello!

I had the same issue (with the server not using the French models despite the -prop flag) and repeating the properties in wget perfectly worked (thanks @J38 for the trick!)

I was just wondering what the status was about the configuration of "not-english" models? Also is there a chance to have it (other models) working in the UI?

Thank you.

[jumboalex]

Quick question, if I add the "segment" into the source code default properties, and rebuild the jar file, would that work for Chinese? I'm getting the similar issue, and furthermore the Chinese characters cannot display in the output html. (Well I set everything the same as above)

[vache]

What is the status of this? I've tried this with the chinese properties passed in the URL with no luck. Is there any other workaround at the moment?

[rudaoshi]

You should write a server that accept a custom properties. Current server of corenlp 3.6 works in a stupid way, since one hardly changes their requirements on nlp components.

I wrote my own server: https://gist.github.com/rudaoshi/33f6efeee23a72fe565f8e33af0873a9

I'm glad if you find it helpful.

[aqppe]

guys, could you give a example using wget...
stellasia could you give the example you used.

[stellasia]

@aqppe, you can find the wget command I finally used in this SO answer

[juvaud]

I have troubles to get the server running for Chinese, as the server tells me that it doesn't find the segmenter, which is defined in the properties file. Somebody got it running for Chinese explicitly?

I keep getting java.lang.IllegalArgumentException: Unknown annotator: segment when trying to use the pipeline remotely. If I'm starting it not as server and client it works. How can I fix that? (keep in mind im using the config file, so customAnnotatorClass.segment = edu.stanford.nlp.pipeline.ChineseSegmenterAnnotator is defined.

[J38]

I'll repeat my answer from a different issue:

This issue has now been resolved in the current version of the code. If you enter a command like this:

wget --post-data '' 'localhost:9000/?properties={"pipelineLanguage":"zh", "outputFormat":"json"}' -O - the server will load the appropriate Chinese resources.

[unhandyandy]

I'm finding that when I run
wget --post-data '' 'localhost:9000/?properties={"pipelineLanguage":"zh", "outputFormat":"json"}' -O -
the server encounters an IndexOutOfBoundsException while loading the coref annotator.

Still seems to work though.

[hezhenghao]

When I use:
wget --post-data '' 'localhost:9000/?properties={"pipelineLanguage":"zh", "outputFormat":"json"}' -O -
I'm getting this error:
java.lang.NullPointerException at java.util.Properties$LineReader.readLine(Properties.java:434) at java.util.Properties.load0(Properties.java:353) at java.util.Properties.load(Properties.java:341) at edu.stanford.nlp.pipeline.StanfordCoreNLPServer$CoreNLPHandler.getProperties(StanfordCoreNLPServer.java:681) at edu.stanford.nlp.pipeline.StanfordCoreNLPServer$CoreNLPHandler.handle(StanfordCoreNLPServer.java:540) at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83) at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82) at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675) at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79) at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Have been struggling with this for quite a while now. Someone please help.

[fingertap]

It seems that the parsing result returned is not correct. I post a simple sentence and the server just recognized the punctuation.

wget --post-data '你可以解释这句话么？' 'localhost:9000/?properties={"pipelineLanguage":"zh", "annotators":"tokenize, ssplit, pos", "outputFormat":"json"}' -O - 2>/dev/null

The formatted result is:

{
  "sentences": [
    {
      "tokens": [
        {
          "after": "",
          "before": "",
          "pos": "CD",
          "characterOffsetEnd": 9,
          "characterOffsetBegin": 0,
          "originalText": "你可以解释这句话么",
          "word": "你可以解释这句话么",
          "index": 1
        },
        {
          "after": "^@^@^@^@^@^@^@^@",
          "before": "",
          "pos": "CD",
          "characterOffsetEnd": 10,
          "characterOffsetBegin": 9,
          "originalText": "？",
          "word": "？",
          "index": 2
        }
      ],
      "parse": "SENTENCE_SKIPPED_OR_UNPARSABLE",
      "index": 0
    }
  ]
}

Use edu.stanford.nlp.pipeline.StanfordCoreNLP directly from command line can give correct answers. Above result is obtained under Stanford-CoreNLP-3.6.0.

[henryyang42]

Since we need to pass all the config in StanfordCoreNLP-chinese.properties to properties, the following ugly python snippet may do the job.

Start server first:

java -Xmx8g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -props StanfordCoreNLP-chinese.properties -serverProperties StanfordCoreNLP-chinese.properties -port 9000 -timeout 15000

Then:

from pycorenlp import StanfordCoreNLP
filename = '/Users/henryyang/stanford-corenlp-full-2016-10-31/StanfordCoreNLP-chinese.properties'
properties = {}
with open(filename) as f:
    for line in f.readlines():
        tok = line.split('=')
        if len(tok) == 2:
            properties[tok[0]] = tok[1][:-1]
            
nlp = StanfordCoreNLP('http://localhost:9000')
res = nlp.annotate("过了下午四点，带母亲、肇嘉嫂、淑英等孩子们到天坛游览", properties=properties)
res = json.loads(res)
for sentence in res['sentences']:
    for tok in sentence['tokens']:
        print (tok['word'], tok['ner'])

Output:

过 O
了 O
下午 MISC
四点 MISC
， O
带 O
母亲 O
、 O
肇嘉嫂 O
、 O
淑英 O
等 O
孩子们 O
到 O
天坛 O
游览 O