Tokenization of English (especially contractions in informal genres)

[manning]

I was wondering whether we should try to do more to standardize/change the tokenization of English used in UD. (At the moment, decisions of tokenization are largely outside the scope of the UD guidelines apart from some general admonitions of UD being built on a lexicalist belief in words).

At present, UD_English-EWT almost entirely follows the tokenization done by newer LDC/Penn Treebanks like OntoNotes (or, here, it just follows the English Web Treebank). Their approach has always been a decompositional one so [don't] ➔ [do] [n't] and [wanna] ➔ [wan] [na]. This facilitated the provision of regular syntactic analyses, though this is maybe more necessary for Penn-style constituency analyses than UD analyses. (The two ways at present in which UD_English-EWT deviates from EWT are: (i) not escape parentheses as -LRB- and -RRB-, and (ii) not converting into ASCII characters in the source documents that were Unicode outside the ASCII range (curly quotes, dashes, etc.).)

In contrast, most (but not all) treebank annotators of tweets and other informal and user-generated content have felt that it is infeasible and maybe even wrong to adopt such a decompositional approach, and so have in general kept contracted tokens and abbreviations together. For example [wanna] stays as is, as does [smh] in [smh at your ignorance] "shaking my head at your ignorance". For example, for the small UD TweetAAE corpus, Blodgett, Wei and @brendano (ACL 2018) "did not attempt to split up these words into their component words (e.g. wanna → want to), since to do this well, it would require a more involved segmentation model over the dozens or even hundreds of alternate spellings each of the above can take". And a similar approach was used in the early Tweebank that @nschneid contributed to. There are strong practical arguments. As well as a multiplicity of forms, you just end up sometimes with a lack of letters (if you avoid null tokens). For example the spelling [ima] is sometimes used, instead of [imma] or [umma] for "I am going to". There just aren't enough letters in [ima] to assign them one each to the words in "I am going to". As well as the practical arguments, an earlier paper suggested a more principled argument: It is perhaps linguistically wrong to project formal English conventions onto the conversational chatter of young people. For instance, many young people now also speak (say the letters of) "wtf". (Nevertheless, in the other direction Liu, Zhu, Che, Qin, Schneider, and Smith (NAACL 2018) did try to split up all tokens in their Tweebank to try to provide a UDv2-consistent version – I think essentially trying to match UD_English-EWT.)

It would be nice to agree on a best, or least bad, convention here. A few thoughts on that:

• Separation is linguistically motivated for the possessive ['s] and verbal and modal auxiliaries like ['ll], which act as phrasal clitic not pieces of morphology of the word that hosts them, for example in [the Queen of England's hat] or [the team that's ahead'll try to slow down play].
• But this phrasal clitic analysis isn't motivated in the same way for contracted negation or to-auxiliary forms like [isn't] and [wanna].
• It seems slightly undesirable if [don't] and [dont] don't have the same analysis, but maybe this isn't that problematic in UD. It's not really more problematic than how UD copes with other null words such as dropped copulas in the choice of [you with me] vs. [you are with me].

So, what are the possible positions? There may be more, but some points on the spectrum are:

1. Stick with copying current LDC/Penn English tokenization. This offers compatibility with other LDC resources and it's decompositional nature gives more parallelism in syntax, at the cost of large difficulties in tokenization and kind of ridiculous allomorphs (like [a] for "to").
2. Only separate off the clearly phrasal clitics, as defined above. But if you do this consistently, since ['ll] for "will" is a phrasal clitic, you still have the tricky tokenization problem of distinguishing when [ill] is "ill" (sick) vs. should be [i ll] for "I will".
3. Decide to stick with dividing tokens for the traditional contractions of formal written English as well as for the phrasal clitics above. This would give substantial compatibility with LDC/Penn Treebanks, but we don't try to extend it to further more informal forms. That is, we'd get tokens like: [do n't], [John 's], and [she s] but [gonna], [imma], and [idc]. (So, the LDC's splitting of cases of [wanna] in EWT would be reversed in UD_English-EWT. Possibly we would also no longer split [cannot] into [can not] as the LDC does, though presumably we might still split [ca n't] for consistency with other "n't" forms?)
4. Decide that all this token splitting is a big mistake. We'd still tokenize punctuation marks, etc., from words, but all of these things would be a single word: [don't], [John's], [shes], [gonna], [imma], [idc]. Representing the "wrong" treatment of phrasal clitics would require loosening up the taxonomy of the morphological features, since possessive case can appear on verbs, prepositions, etc., but there is some precedent for that anyway in languages with case marking spreading to different word classes, such as in Australian languages.

Any thoughts? I'm wondering whether option 3 might be better than where we are now with option 1 (at least in the case of UD_English-EWT).

[brendano]

Thanks for the discussion. I agree with your characterization of our arguments.

[3] sounds pretty reasonable. Another benefit it has, is that it works with a rule-based tokenizer that hardcodes the usual formal/mainstream/contemporary-but-pre-internet English conventions for splitting. This problem is much easier and more clearly defined than full-out segmentation, and is solvable by radically simpler and more-deterministic software too, which will be a prerequisite step for any model hoping to use a UD-supervised model on actual text.

Having different tokenization-level treatment of [dont]=>[dont] versus [don't]=>[do n't] sounds OK to me. Presumably downstream NLP models should be smart enough to deal with both forms, especially if they already have to deal with, say, [dnt] [dn't] and [dunt] (as from this word cluster).

One question this raises is what should the annotation conventions be, for these one-token multiwords. Is there a precedent in UD in other languages for this? For dependency edges, in our ACL 2018 paper we proposed to think of the internal words as a hidden subgraph, and we tried to annotate all or most of the edges connecting words within the subgraph to words in the rest of the sentence. This often works well - especially when the subgraph is a connected tree, as we found as is the case for [wanna], [dont], and [ima].

However, it's not always possible to follow this convention; for example, if there are two words in the subgraph that have different governors outside the subgraph (I can't remember if we saw this particular instance in practice, but there were certainly some cases that didn't map cleanly to the subgraph principle).

In any case, sometimes we decided the compound/multiword was noncompositional enough such that it had a different syntactic function than it would have had if it was written as multiple explicit words; for example, it's pretty clear "lol" has ascended to a noncompositional status by now and should be annotated as an interjection in "i am here lol", where as a non-abbreviated "i am here. laughing out loud" might be more like a second sentence/utterance and not an interjection (?). This may be an area for annotator discretion?

[amir-zeldes]

I'm more or less on board with 3., and would like to emphasize that incompatibility with older LDC corpora is a substantial prices: The UD English corpora put together are still nowhere near the size of the full PTB, so I wouldn't want to give up on them.

As for specifics: 3. is pretty much what we do in UD-English_GUM, though we do segment alternative spellings of what would otherwise be multiple tokens, so "dont" and "don't" are both two tokens:

https://corpling.uis.georgetown.edu/annis/#_q=Im50Ig&_c=R1VN

I also don't feel that "gonna" needs more segmentation, since it can be labeled aux in a pretty natural way. I'm less sure about "Imma", since that contains an argument, which would otherwise be suppressed.

I do understand the view that downstream NLP will be smart enough to learn things, but that pre-supposes a specific use of the corpora. For those of us doing linguistic research, we may want to know the argument structure of verbs, and for some NLP applications as well, such as using entity spans, argument marking is important. In GUM, we would also break up "ill" meaning "I'll":

https://corpling.uis.georgetown.edu/annis/#_q=ImxsIg&_c=R1VN

If people accept this kind of guideline, then I'm not sure "Imma" is really different from "ill" - it's a contraction spelled without an apostrophe, but it's still a contraction. I would probably do two tokens, "I", "mma", PRON+AUX, though we haven't run into it. We currently leave "lol" & co. alone, though we don't really have a decision procedure to determine what is lexicalized.

[martinpopel]

What I am missing in this discussion is the UD distinction between words and tokens. I think the tokenization should be a relatively simple task, at least for languages using spaces: tokenize on spaces and punctuation (with language-specific rules about symbols like apostrophes and hyphens). However, segmentation of tokens into words may be potentially very difficult.

I have no opinion on whether "wanna", "dont", "omg", "etc" etc. are treated as a single word or multiple words. I just think these should be a single token. In general, I think that except for languages using scriptio continua, no token boundary should have alphabetical characters on both sides of the boundary unless the tokens are separated by a space.

This effectively means that if we want to keep "don't" as two words (which I would prefer), it has to be a single multiword token. English is one of a few outliers in UD, which systematically break the above-mentioned rule (which was not included in the UD guidelines yet, mostly because of legacy treebanks like PennTB, I guess).

I think this was already discussed here in several issues, but I cannot find them now (I guess @dan-zeman will be more successful).

[manning]

Hi @martinpopel, my discussion above was about what words to recognize, and so the discussion is precisely whether "imma" is a multiword token, and more precisely whether this token is regarded as containing 1, 2, 3, or 4 words.

I agree that it would be good to represent the tokens of multiword tokens in the UD_English-EWT – and probably other UD_English treebanks that lack them – but it hasn't happened yet, and I wouldn't hold my breath for v2.5.

[manning]

Along the lines @amir-zeldes suggested, I can see some appeal to the argument that you should separate off verbal arguments and so have "Imma" [ I <--nsubj-- mma], though we should of course remember that many, many languages allow subject pro drop, and even in English you can marginally have things like "Been meaning to talk to you about that." Even adopting that principle, we'd still have [dont] as one word vs. [do n't] as a two-word multi-word token.

p.s. @brendano: Thanks for comments. Your "lol" example is a much better example of something to definitely not expand than my example.

[amir-zeldes]

@martinpopel I think we could pretty easily include super-tokens in GUM, I think the main reason English TBs haven't been doing it is that it is usually not very complex/interesting in English, since cases like "Imma" are rather rare in the datasets we have available.

We do have an annotation in GUM that marks up things that are spelled together unusually, which we use to generate the SpaceAfter attribute:

https://corpling.uis.georgetown.edu/annis/#_q=dw&_c=R1VN

The main use case I've seen for this so far is to distinguish "cannot" from "can not", which I think are both attested in the corpus. If SpaceAfter is used reliably, I think it should be possible to write a simple script to create super tokens for known cases after filtering the output of unique types not involving punctuation.

[martinpopel]

I think it should be possible to write a simple script to create super tokens

I wrote such a script two years ago: https://github.com/udapi/udapi-python/blob/master/udapi/block/ud/joinasmwt.py
It follows the rule I mentioned in my previous post: it creates a multi-word token (MWT) if multiple neighboring words are not separated by a space and the boundaries between the word forms are alphabetical.
Do you see any cases when this script adds MWT where it should not?

[amir-zeldes]

@martinpopel I'm not sure... Sometimes there are tokens spelled without spaces such as typos or unusual spellings, which are not exactly multi-word tokens. Some examples:

• Fstatistic (tokenized as F <--compound-- statistic with SpaceAfter=No)
• meirl - typo(?) for me + IRL (in real life) - not sure this should get a MWT, we tokenized it as two tokens with SpaceAfter=No

Conversely:

• You'da never done it - tokenized as You + 'd + a, where the second item has an apostrophe (non-alphabetic), but should(?) produce a MWT

And then there are phrasal modifiers that act like a word with hyphens, like "steak-on-window impact", not sure whether we do or don't want MWTs there, but hyphens would rule them out, right?

[martinpopel]

I think Fstatistic and meirl should be a single token, but if the majority of UD core thinks otherwise, I will have to give up my rule.

Note that I don't claim my script joinasmwt.py finds all MWTs. I know that apostrophes and hyphens are treated differently in different languages and sometimes even within the same language. Thus these are difficult to be handled automatically.

[amir-zeldes]

I think almost any rule will have a few exceptions and disagreements, but that's not a reason not to add super tokens to English UD corpora :)

@logan-siyao-peng - would you like to take a stab at writing up some suggested guidelines for English and implementing for GUM? We could then see if other English UD corpora want to use it as well.