Nonstandard DEPRELs for nonstandard syntax

[harisont]

Context: together with @AleksandrsBerdicevskis, @Turtilla and @elenavolodina, I have spent a substantial amount of time thinking about how to make use of UD for annotating learner productions in a cross-lingually consistent fashion. We recently condensed these thoughts into a paper. While our suggestions did not meet any substantial objections when they were presented at SyntaxFest, I believe they would benefit from further discussion, not the least because they create some validation issues. So, I thought I'd distill them into a series of issues, of which this is the first one.

---

In learner corpus research, it is often recommended to follow so-called literal rather than distributional annotation criteria. The former implies that morphosyntactic analysis be guided solely by the observed word forms, whereas a distributional (or, as I would call it, interpretative) approach is one where context is used to figure out the learner's intended meaning and annotate accordingly.

We argue that literal criteria work well for lemmatization, morphological analysis and POS tagging, but less so for dependency annotation, which:

• is intrinsically, to some extent, "distributional", and as such it can be a good complement to literal token-level annotation, maximizing the "informativeness" of the treebank (see example 1 below)
• should, at least for ambiguous sentences, be grounded in what the annotator thinks is the intended meaning.

This is particularly relevant for learner treebanks, where nonstandard language use often causes UPOS-DEPREL mismatches and/or syntactic ambiguities.

In most cases, annotating according to this principle works just fine. Sometimes, it breaks (language-specific) guidelines that are not (yet) enforced by the validator. But in rare cases, the validator complains (examples from UD_Swedish-SweLL; feel free to add more for other languages):

1. barnen kan ärva egenskaper från båda mamma och pappa ('the.children can inherit characteristics from both mom and dad'): båda is an existing adjective form, but is used as the conjunction både. We would like to use ADJ + cc, but this combination is not currently allowed by the validator
2. du har lång erfarente på området som du här sökt jobb ('you have long expreince in the.area which you here looked.for job'): should be something like du har lång erfarenhet på området som du har sökt jobb inom ('you have long experience in the.area which you have looked.for job in'), which would make job into an obl of sökt. In the original sentence, we would like to use obl for both som and jobb, but this results in a "too many objects" error
3. Till skillnad mot Ur : Samuel August från Sevedstorp och Hanna i Hult av Astrid Lindgren handlar den om en kärleksfull familj ('to difference against Ur: Samuel August från Sevedstorp och Hanna i Hult by Astrid Lindgren talks it about a loving family'): difficult sentence, but we are quite sure handlar has two subjects (Samuel August från Sevedstorp och Hanna i Hult and den), which is also not allowed

Of course, I'm not saying these shouldn't be flagged by the validator!
On the other hand, I argue that there should be a mechanism for the annotator to mark deliberate violations of the validation rules corresponding to nonstandard syntax in the underlying text.

In the paper, we proposed a new subtype, :*. However, @jnivre pointed out that this could be a problem in cases where the DEPREL at hand already has a subtype, as UDv2 does not allow subtype stacking (such as nsubj:pass:*).
I think this could be solved by using a new MISC item instead, although I'm not sure what a good name for it would be. To partly align with the guidelines for typos and other errors in underlying text, we could perhaps introduce CorrectDEPREL=DEPREL-THAT-WOULD-BE-USED-IN-A-CORRECTED-VERSION-OF-THE-SENTENCE :)

[amir-zeldes]

Thanks for raising these important issues and for the suggestions - this makes sense to me. I think when annotating an interpretation of the data that does not match the words, it's important to include not only the categories being annotated, but also an explicit target hypothesis of what we think the learner was trying to say, not because this makes us right, but because it allows users to understand what the analysis is based on. Then the validator could take that into account to.

So for example for "båda" as a cc, I would also include CorrectForm=både and CorrectPos=CCONJ, and then the validator could know to accept the unusual combination. In 2., I think we have had cases like this in non-standard English texts and we have annotated the relative pronoun as obl:unmarked, basically treating it as an oblique with no preposition, rather than a second direct object. For the two subjects in 3., I wonder if an interpretation with dislocated might be possible, in which we treat one instance as the "real" subject and another as the copy? Or do you think the learner utterance is actually intentionally using two subjects in something like a weird non adjacent coordination?

I also like the idea of CorrectDeprel better than using a subtype, which I think overloads the deprel with too much information. The CorrectX idiom is already used for other things, so it could make sense to extend it IMO.

[harisont]

Thanks for raising these important issues and for the suggestions - this makes sense to me. I think when annotating an interpretation of the data that does not match the words, it's important to include not only the categories being annotated, but also an explicit target hypothesis of what we think the learner was trying to say, not because this makes us right, but because it allows users to understand what the analysis is based on. Then the validator could take that into account to.

So for example for "båda" as a cc, I would also include CorrectForm=både and CorrectPos=CCONJ

Indeed! Initially, I thought that CorrectXXX attributes could be a bit redundant for parallel learner treebank (which is what I'm working with), but I'm gradually realizing that they'd be really good for interoperability with non-parallel datasets.

In 2., I think we have had cases like this in non-standard English texts and we have annotated the relative pronoun as obl:unmarked, basically treating it as an oblique with no preposition, rather than a second direct object.

That makes sense, for languages that use that subtype. In Swedish, though, it introduction seems a bit unlikely, and I'm not convinced that cases like this are a good argument for adding it.

As for 3, I think the author of the essay the sentence forgot that there was already a subject. I can see that happening to me too, at least in speech, independent of the language.

I also like the idea of CorrectDeprel better than using a subtype, which I think overloads the deprel with too much information. The CorrectX idiom is already used for other things, so it could make sense to extend it IMO.

Good, then maybe we can consider that the candidate solution!

[amir-zeldes]

That makes sense, for languages that use that subtype. #1172, and I'm not convinced that cases like this are a good argument for adding it

Right, I'm not suggesting to introduce the subtype, but you could just use the major type obl for that and say it's not actually a case of obj.

[harisont]

Well, but then we accidentally go in the direction opposite of what I was thinking: the analysis sort of encodes the correction, rather than describing what the learner wrote.

In any case, I appreciate the suggestion, but the point is not necessarily how to annotate these few examples, but rather to decide whether there should be a mechanism for marking analyses that deliberately deviate from the validation rules.
If the answer is yes, as I think it should be, two related questions are:

1. what that mechanism should be (at the moment, extending CorrectXXX in MISC seems like a viable solution)
2. whether to only mark violations of existing validation rules or also violations of guidelines that the validator does not enforce (yet)

[ellepannitto]

I fully support this as I also have encountered similar cases with spoken languages, in particular when the speech is produced by non-native speakers.

I know it's just a silly nomenclature issue, but I don't particularly like the "CorrectXXX" label, maybe something like "Target" would sound more neutral?

If I had to choose, I'd probably prefer to have this kind of marking on the relation itself (if not as a subtype, it could be a prefix like *cc instead of cc, but I see how this could cause more mess as far as tools are concerned.

[dan-zeman]

whether there should be a mechanism for marking analyses that deliberately deviate from the validation rules

This is a can of worms that I do not want to open. We should release valid treebanks. Analyses should not be allowed to deviate, whether it is deliberate or not.

[ellepannitto]

I think the issue arises from the fact that some rules in the validator check formal validity (which we should defnitely support) while others check "grammatical" validity, and grammatical validity is not always a well defined thing...

[dan-zeman]

The aim of the validator is to check rules that are well defined in the UD guidelines and are testable automatically.

[harisont]

Well, then I can rephrase the problem in terms of guidelines: current UD guidelines fail to capture the idiosyncrasies of learner language and, in general, of nonstandard language. Should we change those guidelines and say that, for instance, a verb can indeed have two subjects/objects and update the validator accordingly by turning all of the grammatical validity checks into warnings? That seems too drastic to me, but it's a solution too.

[ellepannitto]

It wouldn't be even anything new from the linguistic theory perspective, duplicate subjects are well studied in language acquisition for instance... Now what we're saying is that is it not an acceptable analysis, but I don't really see this as a neutral choice from the linguistic perspective.