Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overly rigid treatment of top nodes in SMATCH #12

Closed
oepen opened this issue Jun 15, 2019 · 2 comments

Comments

@oepen
Copy link
Contributor

commented Jun 15, 2019

when comparing our (forthcoming) MCES scorer with SMATCH, it emerged that there is a difference in the count of correct tuples for two test AMRs from the SMATCH distribution:

# ::snt The boy is a hard worker.
(p / person
      :domain (b / boy)
      :ARG0-of (w / work-01
            :manner (h / hard)))
# ::snt The boy is a hard worker.
(w / worker
      :mod (h / hard)
      :domain (b / boy))

SMATCH correctly establishes three node correspondences: the ‘b’ and ‘h’ nodes from both graphs, and ‘p’ from the first graph with ‘w’ from the second. on this view, the two graphs have the same top node, share two node labels (‘boy’ and ‘hard’), as well as one edge (:domain). however, SMATCH appears to represent the top properties as triples that include the node label, which results in an undesirable double penalty. i wonder whether this treatment in SMATCH is background to the critique by Anchieta et al. (2019)?

@danielhers

This comment has been minimized.

Copy link
Collaborator

commented Jun 17, 2019

At least according to the example in the smatch paper, the top triple is not part of the scoring... but it seems ‘attribute’ triples (including top) were probably just introduced later.

@oepen

This comment has been minimized.

Copy link
Contributor Author

commented Aug 2, 2019

closing this issue in mtool (for now), since we decided we wanted to maintain compatibility with the ‘official’ version of SMATCH.

@oepen oepen closed this Aug 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.