Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why the left and right results differ so much? #68

Closed
wanjunhong0 opened this issue Sep 16, 2020 · 1 comment
Closed

Why the left and right results differ so much? #68

wanjunhong0 opened this issue Sep 16, 2020 · 1 comment
Labels

Comments

@wanjunhong0
Copy link

Based on my understanding, your left evalution is to (h ,r, ?) predict tail entity and right is (?, r, t) to predict head entity.
I understand why you have to use reverse triple.
But why the right results are so much worse than the left part? We train the reverse triple as well, and the left/right results should not be differ that much. It does not make sense to me.

My results as example:
Mean reciprocal rank left: 0.41416008121148684
Mean reciprocal rank right: 0.20748184567705122
Mean reciprocal rank: 0.310820963444269

@TimDettmers
Copy link
Owner

This has mostly to do with the data and how it is constructed. Since in many languages are read from left-to-right, including English, the dataset language, most relationships in knowledge graphs are read in a left-to-right manner. For example, you would see and expect (Tom Hanks, won, Academy Award for Best Actor) rather than (Academy Award for Best Actor, won, Tom Hanks). As you can see in this example, left-to-right and subject-verb-object relationships are both the main reasons why triples in English datasets often have a specific natural order.

The second part of the issue, if you reverse it, you still have a valid problem, but it is often much more difficult. It is common, especially in the datasets used in this case, for heads to be subjects or people, and for tails to be objects or things. Since people are unique and things are often not for (h, r, x) you usually predict fewer things and these fewer things are dissimilar to each other, while predicting (x, r, t) is more difficult because a thing is often shared by many people and these people have quite similar other attributes.

For example, if you need to predict (x, won, Academy Award for Best Actor) there are so many possibilities of actors all looking quite similar although only a few of them won the award. (Tom Hanks, won, x) is a much easier problem since their are fewer awards and it is likely that Tom Hanks won the award if he already won many other awards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants