Awareness of inverse roles, e.g. inverse of 'domain' is 'mod'

[kovvalsky]

Currently Smatch is not aware that mod is the inverse role of domain. Also it has simplistic treatment of inverse roles. ROLE-of is always converted to ROLE.
This rule in some cases produces non-existing roles, e.g., these are primary roles consist-of, prep-on-behalf-of or prep-out-of, and their reduced versions are not AMR roles.
See the AMR issue

[snowblink14]

mod becomes the inverse of domain quite recently so smatch hasn't caught up yet. This probably just needs a rule change.
Combined with the fact that now we have "-of" in primary rules, it might be easier to extend as AMR evolves if we use a file to store role names and their inverse, and read/validate them in AMR (i.e. separating the validation and computing module).

[goodmami]

Personally I would rather not bake in special cases for the semantic model (AMR). Smatch has a nice, general-purpose DAG similarity metric and putting in special cases for AMR makes it less useful for non-AMR evaluation (e.g., I use it for comparing DMRSs encoded in PENMAN notation). But inverse roles do pose an issue (see below), so an optional feature enabled via a command-line switch would be nice.

To describe the problem further:

1. If AMR 1 has (a :domain-of b) (Smatch reads as (domain, b, a)) and AMR 2 has (a :mod b) (Smatch reads as (mod, a, b)), then there is a missed opportunity for a match if you consider :domain-of == :mod and :mod-of == :domain.
2. Similarly, if AMR 1 has (b :domain a) and AMR 2 has (a :mod b), there's a potential match that gets missed.
3. The others (:consist-of, :prep-on-behalf-of, :prep-out-of) don't have the same issue; their incorrect inverses (:consist, :prep-on-behalf, :prep-out) are not currently valid AMR roles, so there's no risk of overlap. However, if AMR 1 has (a :consist-of b) and AMR 2 has (b :consist-of-of a), Smatch will fail to match them.

Simply using an AMR-model-aware parser can solve (3) because all roles get de-inverted correctly. (1) and (2) are the more difficult cases. You can canonicalize roles as a preprocessing step to resolve (a :domain b) vs (a :mod-of b), but canonicalization would introduce differences in cases like (a :domain b) vs (b :domain-of a). You could instead invert all :domain and :domain-of roles to :mod-of and :mod, respectively (so all such relations use :mod as the primary role), but this risks making a constant into the source of a relation (see amrisi/amr-guidelines#235) which IIRC also causes problems with Smatch.

To really consider :mod and :domain as inverses, you'd probably need a special triple comparison function inside the hill-climbing loop that evaluates Smatch's (mod, a, b) and (domain, b, a) as a match. But, again, I think this behavior should be optional. Or alternatively allow Smatch to somehow deal with constants as the source of a relation.

These ideas are explored further in https://arxiv.org/abs/1909.01568 (disclaimer: I'm the author).

[oepen]

i am attaching to this thread some earlier communication that @khlmnn and i had with the AMR folks at ISI, notably @ULFULF, about incomplete normalization of inverse roles in AMR, including a proposed patch. at the time, however, i suspect we were not thinking about the interaction of inverse roles and constant values. more on that in a separate comment.

From: Marco Kuhlmann <marco.kuhlmann@liu.se>
Date: Fri, Sep 18, 2015 at 11:02 AM
Subject: Re: Normalised AMR
To: Ulf Hermjakob <ulf@isi.edu>
Cc: Kevin Knight <knight@isi.edu>, David Chiang <dchiang@nd.edu>, Stephan Oepen <oe@ifi.uio.no>

Hi Ulf,

here is a new version of our proposed patch against the amr.py in Smatch 2.0; we hope that we got everything right. Do not hesitate to contact me in case you have any questions related to the code.

Best,
Marco

> On 17 Sep 2015, at 02:16, Ulf Hermjakob <ulf@isi.edu> wrote:
>
> Thanks, Marco, for reporting this problem.
>
> Yes, you got the special inverse cases (almost) correct:
>  :domain (3468 instances) has the inverse :mod (36237)
>  :consist-of (552) has the inverse :consist-of-of (0)    (not :consists-of/:consists-of-of)
>  :prep-on-behalf-of (14) has the inverse :prep-on-behalf-of-of (0)
>  :prep-out-of (1) has the inverse :prep-out-of-of (0)
>  :prep-instead-of (32) had the inverse :prep-instead-of-of (0)
>
> The integers in parentheses are role instance counts in the current AMR corpus,
> except for :prep-instead-of where the count is for last general release.
> (:prep-instead-of has been superseded in latest AMR version by a new frame instead-of-91.)
>
> Note however, that :prep-with-of (1) is the inverse of :prep-with (193).
>
> The smatch tool does indeed not properly deal with some of these inverses,
> in particular for the common case of :domain/:mod.
> I agree that the defect is severe enough to fix it sooner than later.
>
> The person who wrote the smatch tool has left ISI, and I don't actively
> program much in Python, so I would be thankful if you could send me your
> proposed patch and test file, which I didn't get as I wasn't part of the
> thread yet when you sent it.
>
> Thanks,
>
> -- Ulf
>
> ----- Original Message -----
>> From: "David Chiang" <dchiang@nd.edu>
>> To: "Marco Kuhlmann" <marco.kuhlmann@liu.se>, "Ulf Hermjakob" <ulf@isi.edu>
>> Cc: "Kevin Knight" <knight@isi.edu>, "Stephan Oepen" <oe@ifi.uio.no>
>> Sent: Wednesday, September 16, 2015 9:03:50 AM
>> Subject: Re: Normalised AMR
>>
>> Hi Marco, I'm adding Ulf Hermjakob to the conversation, who is the authority
>> on these things :)
>>
>> David
>>
>>> On 2015/9/16, at 11:48, Marco Kuhlmann <marco.kuhlmann@liu.se> wrote:
>>>
>>> Dear Kevin and David,
>>>
>>> following up on some of the conversations that we had at Dagstuhl, Stephan
>>> (cc) and I are trying to get a better understanding of the structural
>>> differences between various semantic representations.
>>>
>>> Regarding AMR, we are looking into the differences between AMR as annotated
>>> and the normalised version that David showed at Dagstuhl. Our
>>> understanding is that to obtain this normalised version one needs to do
>>> the following:
>>>
>>> 1. reverse all arcs labeled with a relation type that ends in “-of” (and
>>> strip that suffix off the relation name), except “consists-of” and
>>> prepositional relations such as “prep-instead-of” and “prep-on-behalf-of”
>>>
>>> 2. reverse all arcs labeled “mod” and relabel them as “domain”
>>>
>>> Did we get that right?
>>>
>>> We noticed that the Smatch reader (amr.py) attempts to reverse *all*
>>> “-of”-arcs without exceptions, but misses some cases. For example, in the
>>> attached amr-test.txt the reader misses to reverse the first occurrence of
>>> the “part-of” relation while it reverses the second. Could this be an
>>> oversight?
>>>
>>> We have prepared a patch that we believe fixes the reader; please find it
>>> attached. The patch encapsulates the decision about which arcs to reverse
>>> (and relabel) into a static method inverted_label. Our current
>>> implementation of this method tries to follow the specification given
>>> above.
>>>
>>> We would love to hear back from you about the normalisation. Our aim is to
>>> compute statistics such as the percentage of cyclic graphs, which is
>>> non-neglible (3.7%) for the non-normalised version of the annotation but –
>>> as far as we understand it – should be 0% for the normalised version.
>>>
>>> Best,
>>> Marco

patch.txt

[oepen]

regarding :domain and :mod, i believe you have the directionality of inversion reversed, mike. i read the AMR guidelines as treating :mod as the inverse of :domain, i.e. an alias for :domain-of. assuming that inverse roles cannot be further inversed, :mod-of should not occur, and normalization should always target :domain as the base relation.

this is essentially, what mtool implements in its AMR codec:

https://github.com/cfmrp/mtool/blob/6de2366d3dff73e5a5b8a01a0f43727a8de0b2f3/codec/amr.py#L104

regarding the interactions with constants, i do not see why that (suppression of) normalization would need to be sensitive to node alignments? whether or not the source or target of a relation is constant is a static property of each individual AMR. so, assuming that one would not want constants as source nodes on (normalized) edges, why not just suppress normalization in such cases, i.e. leave something like A :mod "foo" unchanged, i.e. with an edge label of :mod?

[nschneid]

https://www.isi.edu/~ulf/amr/lib/amr-dict.html#:mod documents a situation where :mod CONSTANT is correct:

identifier (often numbers or letters)

• Chapter 1 (chapter :mod 1)
• Figure 6a (figure :mod "6a")

In hindsight it probably would have been better to use a separate role here so that :mod would always be equivalent to :domain-of. But I suppose smatch could leave these cases alone (as @oepen suggests) even while normalizing the rest of them.

[goodmami]

Thanks, @oepen, for the context. I think the information in that email thread was repeated in amrisi/amr-guidelines#194 (specifically this comment).

Regarding the directionality of the inverse of :domain vs :mod, I don't mind either way. The AMR guidelines mention it twice, as does the AMR Annotation Dictionary that @nschneid linked to (thanks for that, btw). They are both clear that :mod and not :domain-of are the inverse of :domain. It is not specified if :domain and not :mod-of is the inverse of :mod, so how you interpret that probably depends on whether you consider :mod a fully-fledged and independent role or not. My perspective on this may be different than most, as with Penman I am concerned not only with reading hand-annotated AMRs to graphs but also writing AMRs from graphs, possibly in different configurations (with different edge directions). Therefore, as I see it, both :domain and :mod are primary (non-inverse) roles. This is convenient for the constant-target issue, for one, but it is also consistent with Ulf's statement (first line of the comment linked above) that all roles listed at https://www.isi.edu/~ulf/amr/lib/roles.html are primary roles, and :mod is listed there (twice, even). So if we are to normalize them as true inverses, it does not matter to me if they always become :domain or :mod. I chose :mod above because (a) it's more common, (b) it has a reification (have-mod-91) whereas :domain does not, and (c) it's shorter to type.

i do not see why that (suppression of) normalization would need to be sensitive to node alignments?

If I understand you correctly, then no, that is not what I meant. The OP of this issue did not spell out why the lack of awareness of AMR's proper inverses is a problem for Smatch, so I was trying to fill in the blanks. E.g., if a gold corpus has a :mod b but a parser outputs a :domain-of b and you consider :mod == :domain-of, then Smatch currently gives you a lower score than it should. Normalizing inverses could gain back these points, but there are other issues, such as constants becoming the source of relations (which get ignored):

$ cat a
(c / chapter
   :mod 7)
$ cat b
(c / chapter
   :domain-of 7)
$ cat c
(c / chapter
   :domain 7)
$ cat d
(c / chapter)
$ python smatch.py -f a b
F-score: 0.80
$ python smatch.py -f a c
F-score: 0.67
$ python smatch.py -f a d
F-score: 0.80

The original point of my comment above, however, is that I'd prefer that these AMR-aware deinversions happen optionally or outside of Smatch proper, not only because they would otherwise diminish Smatch's (perhaps fortuitous) generality for non-AMR graphs, but because AMR continues to evolve and we probably do not want to pin a new Smatch release to be used for every AMR release.

[nschneid]

it is also consistent with Ulf's statement (first line of the comment linked above) that all roles listed at https://www.isi.edu/~ulf/amr/lib/roles.html are primary roles, and :mod is listed there (twice, even).

But his comment goes on to say: "Please also note that the inverse of :domain is :mod." I read that as an exception to the statement about primary roles. I.e. my understanding is that :mod is an inverse role, essentially shorthand for :domain-of. (Except where it involves a constant.) :mod is easier to read and thus stylistically preferable for human-facing AMRs.

@goodmami I think you're saying that two non-inverse roles, :domain and :mod, are nevertheless inverses of each other? As I understand the term "inverse role" that seems like a contradiction.

The original point of my comment above, however, is that I'd prefer that these AMR-aware deinversions happen optionally or outside of Smatch proper, not only because they would otherwise diminish Smatch's (perhaps fortuitous) generality for non-AMR graphs, but because AMR continues to evolve and we probably do not want to pin a new Smatch release to be used for every AMR release.

FWIW, I'm not too concerned about those issues because

a) Smatch was really designed for AMR; if people can adapt it for other formalisms, then great, but I don't think implementations can anticipate all possible needs of non-AMR formalisms. I think the defaults in the implementation should reflect what we think is best for AMR specifically.

b) There are no current plans for further development of (canonical, English) AMR. There are various proposed extensions but the funded project that created the Little Prince and LDC corpora has ended, and the final LDC data release from that project is scheduled for 2020. (A new and ambitious approach aiming to increase logical expressivity and incorporate typological considerations is being developed under the name UMR. The team working on this has made progress on theoretical issues, but I don't know if any of the details, including evaluation, have been worked out.)

What I would like to see is a canonical Smatch implementation that developers of AMR parsers can run out-of-the-box with minimal configuration. Otherwise replicability becomes difficult.

[goodmami]

@nschneid ok, it sounds like we were reading that differently. Also, my interpretation has shifted from basically your position to my current one in order to reconcile with the AMRs I encountered, as well as things like :mod's reification (if :mod is just shorthand, why isn't the reification of :domain instead?), etc. In any case, I don't wish to belabor these points if they're finding no traction.

(a) Smatch was really designed for AMR; [...]

Yes. It was perhaps a happy accident that Smatch's graph matching is useful for other tasks. If we view the AMR-parsing as a separate component (as tokenization can be for BLEU), then Smatch's utility can persist beyond AMR. But, again, I don't want to push an unpopular agenda on a repo that's not mine, so I can just work with a fork if necessary.

b) There are no current plans for further development of (canonical, English) AMR. [...]

Thanks for this context. If UMR is the next promising evolution of AMR then I'll keep an eye on that.

What I would like to see is a canonical Smatch implementation that developers of AMR parsers can run out-of-the-box with minimal configuration. Otherwise replicability becomes difficult.

I wholeheartedly agree. While I advocate for the decoupling (or avoidance of coupling) of the AMR model with the graph matching, I intended for the AMR model to be the default or to be very clearly and easily accessible (e.g., an --amr option, current behavior otherwise).

[oepen]

one more comment in this thread (as i just rediscovered it in an idle browser tab :-): i would argue that :mod (aka :domain-of) as a relation between two nodes formally is distinct from :mod when its value is a constant. the former corresponds to an actual edge, involving two graph nodes, whereas the latter rather is a property of a single node, i.e. the ‘target’ is not an actual node and, hence, strictly speaking there is no edge involved; :mod in this case is a node-local property rather than an edge label.

this distinction was present already in the Elementary Dependency Match (EDM) metric of Dridan & Oepen (2011), from which SMATCH derives. the difference is relevant for scoring, where a constant value is not considered as a node in its own right. in the context of the 2019 MRP Shared Task, we have formalized this distinction (and report separate statistics for each distinct type of tuple) and adopted a graphical rendering that avoids the false impression that constant values correspond to separate nodes (see also page #29 in the 2019 tutorial on graph-based meaning representations by Alexander Koller, myself, and Weiwei Sun).

being reminded of this formal distinction, i now believe that the right approach for SMATCH will be to normalize :mod to :domain for genuine relations and leave the node property (aka attribute) :mod for constant values.

[goodmami]

Regarding the edge vs property distinction, it is already the case that smatch makes the distinction between triples whose target is a variable vs a constant, but it calls the latter "attributes" rather than "properties". There is no difference in weights for these in scoring, but smatch has the --justinstance, --justattribute, and --justrelation options to restrict evaluation to only consider node labels (instances), node attributes, or node-to-node edges (relations).

The trouble is when we have inverted edges whose source (on deinversion) is a constant. This issue goes beyond the :mod / :domain issue we are discussing, so I've created #26.

Otherwise I think your proposal to normalize :mod relations (true edges) to :domain-of while leaving :mod attributes as-is sounds reasonable. If #26 is resolved, then we wouldn't need this conditional deinversion to apply to all other roles as well.

I still have some reservations with the wholesale normalization of :mod to :domain where it concerns the reified form have-mod-91, but this is a separate discussion.