Adds a tokenization postprocessor for manual token cleanup

[Jemoka]

Description

• creates a tokenize_postprocessor config element to stanza.Pipeline which allows a callable to be passed which can override default tokenization
• creates stanza.models.tokenization.utils.reassemble_doc_from_tokens which allows for reassemblage of a document from re-adjusted tokenizations

For instance:

nlp = stanza.Pipeline(lang="en", processors="tokenize",
                                    tokenize_postprocessor=lambda draft: do_stuff_to_draft(draft))

whereby, the single argument passed to tokenize_postprocessor is a list of lists containing string sentence and word tokenizations

[['Joe', 'Smith', 'lives', 'in', 'California', '.'], 
 ['Joe', "'s", 'favorite', 'food', 'is', 'pizza', '.'], 
 ['He', 'enjoys', 'going', 'to', 'the', 'beach', '.']]

to mark a MWT, each element can optionally be a tuple with a Bool second element. If the word dai in the following sentence as an MWT, for instance, we will be passed

[['Diglielo', ('dai', True), 'venire', 'a', 'mangiare', 'un', "po'", 'di', 'margarina']]

The callable passed to tokenize_postprocessor must return a list of lists in the same format: adjusting word, sentence tokenizations and MWT designations as needed.

Unit test coverage

• integration test of postprocessor, and type check of the argument passed
• unit test of the reassemble_doc_from_tokens function against normal & OOV characters