Lots of errors when looking at model.errors(graph)

[bjascob]

I'm trying to use the library to weed out any bad graphs that get produced by my AMR parser but I'm finding that even on standard AMR training data (LDC2015E86) this is giving me a lot of errors. Here's what I'm doing...

    from penman.models.amr import model
    entries = load_raw_amr(infn) # load text and split into entries on 2 line-feeds
    for i, entry in enumerate(entries):
        graph = penman.decode(entry)
        error_dict = model.errors(graph)

If I accumulate some stats on LDC2015E86 here's the counts / errors...

20697 ('invalid role', ':wiki')
  466 ('invalid role', ':compared-to')
  375 ('invalid role', ':consist')
   30 ('invalid role', ':range')
   13 ('invalid role', ':prep-on-behalf')    
    3 ('invalid role', ':ARG6')
    2 ('invalid role', ':ARG7')
    1 ('invalid role', ':prep-versus')
    1 ('invalid role', ':prep-out')

About 2/3 of the graphs have some type of error. Am I using the library wrong?

Also, since decoding uses the model, is there a simple way to get the errors after calling decode (or possibly load) instead of passing the graph back into model.errors(graph)?

One more thing, when decoding I routinely see warnings such as... WARNING:penman.layout:cannot deinvert attribute: ('l', ':poss-of', '700'). Is this an issue? I would think that this would be somewhat typical for an attribute. I notice this only comes up with the output of my parser. I don't see it on the LDC data so maybe the x-of is not legal on an attrib?

[goodmami]

Thanks, @bjascob! I don't currently have access to the LDC data so I appreciate your bug reports for things I can't easily test. As to your problems, there are a few things going on, detailed below. Also, there are several documents I refer to:

• AMR role inventory
• AMR dictionary,
• Propbank frames
• AMR specification

The roles defined in penman.models.amr come from the role inventory.

• :wiki : this is not included in the role inventory. It is included in the AMR dictionary, and it is clearly well-used, so I should probably add it to the model.
• :compared-to : this doesn't exist in the role inventory and was marked "OBSOLETE" in the AMR dictionary. I could either have different AMR model versions for the AMR versions or LDC releases, or include it since it was valid at some point.
• :consist, :prep-on-behalf, :prep-out : these are incorrect inversions of :consist-of, :prep-on-behalf-of, and :prep-out-of, as described in the role inventory. Thus, they are bugs in the annotations. However, clarifying comments of the correct inverse forms came out after some data releases, so it's possible that they were not exactly incorrect at the time the data was produced. At the moment I'm leaning toward keeping these as errors, but if I have different AMR models for each release, then I could allow them for the earlier data.
• :range : this is not described in the role inventory in any obvious place. I now see it in the AMR dictionary under the heading "In", and it's in the pop-up examples of the :ord role in the role inventory. I guess this is a valid role, but it is not well-documented.
• :ARG6, :ARG7 : the role inventory only lists through :ARG5, but I now see them (and even :ARG8 and :ARG9) in the AMR dictionary and the propbank frames. I think there's no problem to add these.

About 2/3 of the graphs have some type of error. Am I using the library wrong?

Nope, you're using it correctly. The 2/3 are mostly the :wiki roles, which I should add.

Also, since decoding uses the model, is there a simple way to get the errors after calling decode (or possibly load) instead of passing the graph back into model.errors(graph)?

You mean where it checks for errors as it decodes instead of as a separate step? No, that is not provided.

One more thing, when decoding I routinely see warnings such as... WARNING:penman.layout:cannot deinvert attribute: ('l', ':poss-of', '700'). Is this an issue?

This means that Penman tried to deinvert the relation but it could not because it would make a constant (not a node) into the source of the triple, which is not allowed. It is a warning and not an error because you can still proceed, but the triple is then ('l', ':poss-of', '700') and not ('700', ':poss', 'l'). For this to be correct, your parser would need to choose some appropriate non-inverted role, or create a node. For instance, the minimal change might be (l :poss-of (x / 700)). This is what penman.transform.reify_attributes() does, but it applies this to all attributes, which is probably not what you want in this case.

[bjascob]

Thanks for all the info.

FYI... for my use case, I'm looking for "bad" errors so I probably don't care whether it's conforming to a specific revision of the specs / LDC releases. As long as it's a reasonable role, I'm OK with keeping it. What I'm saying is, don't make a bunch of models specific to the LDC releases / specs for me. For now, I'll just create exceptions for the items listed above.

To clarify on the last point... if the parser is producing attributes with X-of roles (and thus the WARNING), would you consider these incorrect graphs? My use case is to use the JAMR parser to produce "silver" data to help train a generator model. I want to throw out anything that is non-conformant. I looked through the spec and could only find the reification section which didn't exactly say this is invalid, only that it can be swapped.

[goodmami]

What I'm saying is, don't make a bunch of models specific to the LDC releases / specs for me.

Got it. But also I think this library could at least document better which version of AMR it supports.

For now, I'll just create exceptions for the items listed above.

Good idea. Note that you cannot just do:

from penman.models.amr import model
model.roles.update({':wiki': {'type': 'wiki'}})  # don't do this

... because when the model is created it creates a regular-expression based index for checking if a role exists. Rather, rebuild the model with the expanded roles. Something like this:

from penman.models import amr
from penman.model import Model
roles = dict(amr.roles)
roles.update({'wiki': {'type': 'wiki'}})
model = Model(
    roles=roles,
    normalizations=amr.normalizations,
    reifications=amr.reifications,
)

if the parser is producing attributes with X-of roles (and thus the WARNING), would you consider these incorrect graphs?

I do, yes. The official documentation is lacking on this point, but this is my conclusion based on what I've seen. For instance, if I were to treat constants like 700 as proper nodes, you might get something like this if the constant were repeated (e.g., "i bought 700 oranges and 700 apples"):

... (o / orange :quant (700 :quant-of (a / apple)) ...

... and most people would find that surprising. So I now have the notion of "attribute" edges which are those where the pre-normalized triple (or tree branch) has a value that is not the source variable of some node. Note that, hypothetically, if we had a role whose primary form has -of (similar to :consist-of, :prep-out-of, etc.) that could take a constant value, this would not generate a warning as it's not actually inverted.

Whether or not you include these in your model is your choice. If you are training an AMR to text generator and the input AMRs from JAMR might contain these bad relations, you might choose to not filter them to increase robustness. But if you only want "proper" data (more relevant if you're training a parser, perhaps?) then you might avoid them.

[bjascob]

I think your conclusion on the X-of attributes not being valid is backed up by the fact that LDC2015E86 doesn't have any of these relationships. The JAMR parser unfortunately produces quite a few of them (~350 in 20K graphs).

Again, thanks for all the great info.

[goodmami]

There were several points raised in this issue, so I gave the issue the "question" label and created #89 specifically for the missing roles. I'll close this now.

Thanks, again!