How to correlate between parser input string and output tree/tokens

[ndvbd]

Sometimes an input string can have for example multiple spaces, or multiple sentences, or other characters that are being removed by the parser, and sometimes the parser returns a forest (list of trees, each representing a sentence).

I want to be able to correlate exactly from an input segment (start/end character), to the corresponding token in the output forest. How can I easily do it? I mean, the parser knows what characters were removed/trimmed, so it would be useful to add an API parameter (for the CoreNLP server), that instead of just returning the tokens, and their POS, also the start/end character in the corresponding input sentence.

[J38]

The json output includes a "sentences" key, each sentence includes a "tokens" key, and the hash for each token has both a "characterOffsetBegin" and "characterOffsetEnd" key to access the character offset information.

As you process the PTB representation of the parse tree, you simply need to map leaf number n to the nth entry in the tokens list for that sentence.

I do not know how you are processing the json after you get it back from the server. We do have some Python code for handling server requests.

Here is a link to the GitHub: https://github.com/stanfordnlp/python-stanford-corenlp

Here is some small sample code for iterating through the parse tree object and mapping the character offsets to the leaves:

import corenlp
  
client = corenlp.CoreNLPClient(start_server=False, annotators=["tokenize", "ssplit", "pos", "lemma", "ner", "parse"])

ann = client.annotate(u"...insert text here...")

def map_leaves_to_char_offsets(a_sentence):
    token_count = 0
    def go_through_tree(a_tree):
        nonlocal token_count
        for child in a_tree.child:
            if len(child.child) == 0:
                print(str(child.value)+"\t"+str(a_sentence.token[token_count].beginChar)+"\t"+str(a_sentence.token[token_count].endChar))
                token_count += 1
            go_through_tree(child)
    go_through_tree(a_sentence.parseTree)


map_leaves_to_char_offsets(ann.sentence[0])

[ndvbd]

@J38, I think there is still a tiny bug there.
When I parse 'The boy' - I get for "the" the positions (0,3) as it supposed to do.
When I parse ' The boy' - I get for "the" the positions (0,3), which is wrong (I added a leading space).
It feels like the whitespaces are trimmed at the beginning of the text? Either way, this is a situation where the tokens are not correlated to the plain text in full. Am I missing something?

[ndvbd]

Any ideas?