🚚🤖 Add Transformer Interaction

[mberr]

This PR adds the Transformer-based interaction from the StarE paper.

cf.

• https://aclanthology.org/2020.emnlp-main.596/
• https://github.com/migalkin/StarE

Note: this PR doesn't add the citation since it's already in the references.rst because we referenced it before.

[cthoyt]

Is the concept of the transformer interaction an idea proposed by galkin et al. (2020)? Wondering if I should add the citation in

[cthoyt]

Is there a blocker for adding a StarE model appropriate for regular KGs?

[mberr]

Is there a blocker for adding a StarE model appropriate for regular KGs?

StarE collapses to CompGCN if you remove all the hyper-relational parts 🙂

(StarE is "only" the encoder part of what is described in the publication; the decoder is this interaction function)

[cthoyt]

So then it's fine to finish this PR without adding a new model, but it would be important to write some narrative documentation explaining how one might use this interaction to construct a new model

[mberr]

So then it's fine to finish this PR without adding a new model, but it would be important to write some narrative documentation explaining how one might use this interaction to construct a new model

This should be the place to put it? To slightly modify the example there and create an "anonymous" model instead of declaring a new class:

from pykeen.models import ERModel
from pykeen.nn import EmbeddingSpecification
from pykeen.losses import BCEWithLogitsLoss
model = ERModel(
    triples_factory=...,
    loss=BCEWithLogitsLoss(), # TODO: we should support passing a string here, and use the loss-resolver to resolve it
    interaction="transformer",
    entity_representations=EmbeddingSpecification(embedding_dim=64),
    relation_representations=EmbeddingSpecification(embedding_dim=64),
)

[migalkin]

Thanks Max!
I debugged the test case and the interaction function seems to work as expected 🚀

A few minor comments:

• Position embedding is initialized randomly self.position_embeddings = nn.Parameter(torch.rand(2, input_dim)) while we used xavier init. It would probably not matter for just 2 elements (instead of 43 in the StarE case), the original motivation was to initialize entities, relations, and positional encodings in the same scale (range) such that there won't be sudden gradient explosions
• Since the final operation is the dot product, sLCWA training with margin loss doesn't make a lot of sense. LCWA + BCE is ok, sLCWA would work with BCE+sigmoid or softplus. Shall we put some warning if a user creates a pipeline with this interaction and margin loss?

[mberr]

Thanks Max! I debugged the test case and the interaction function seems to work as expected 🚀

A few minor comments:

Position embedding is initialized randomly self.position_embeddings = nn.Parameter(torch.rand(2, input_dim)) while we used xavier init. It would probably not matter for just 2 elements (instead of 43 in the StarE case), the original motivation was to initialize entities, relations, and positional encodings in the same scale (range) such that there won't be sudden gradient explosions

I changed the init to xavier with 08ae09a

Since the final operation is the dot product, sLCWA training with margin loss doesn't make a lot of sense. LCWA + BCE is ok, sLCWA would work with BCE+sigmoid or softplus. Shall we put some warning if a user creates a pipeline with this interaction and margin loss?

This PR only adds the interaction function, not a full model configuration with loss etc. I see that we lose the geometric interpretation of margin loss, but is there any other reason why dot product should not work with margin? After all, margin encourages a relative ordering of scores, which could also be achieved by dot-product based interaction, if I am not mistaken.

[cthoyt]

@PyKEEN-bot test

[cthoyt]

Sorry didn't mean to approve - I still want to see some added tutorial functions like you mentioned