🦋 🏇 Label-based initialization #652
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mberr
commented
Nov 26, 2021
trigger ci
trigger ci
trigger ci
trigger ci
…n into label-based-representations
trigger ci
trigger ci
cthoyt
reviewed
Nov 29, 2021
cthoyt
reviewed
Nov 29, 2021
Co-authored-by: Charles Tapley Hoyt <cthoyt@gmail.com>
cthoyt
reviewed
Dec 9, 2021
trigger ci
trigger ci
trigger ci
|
what to do about the warning from here Warning, treated as error:
/home/runner/work/pykeen/pykeen/.tox/docs/tmp/source/reference/nn/utils.rst:document isn't included in any toctree
ERROR: InvocationError for command /home/runner/work/pykeen/pykeen/.tox/docs/bin/python -m sphinx -W -b html -d /home/runner/work/pykeen/pykeen/.tox/docs/tmp/build/doctrees /home/runner/work/pykeen/pykeen/.tox/docs/tmp/source /home/runner/work/pykeen/pykeen/.tox/docs/tmp/build/html (exited with code 2) |
Fixed in 40cb229 |
|
@mberr thanks for adding the tutorial! is there anything else you're planning to add, or do you want me to clean it up and finish this PR? |
Trigger CI
Trigger CI
mberr
commented
Dec 11, 2021
cthoyt
approved these changes
Dec 11, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds label-based initialization and representations using pre-trained language models from the
transformerslibraryLabelBasedInitializeradds a new initializer, which computed vectors from the entities' labels. After initialization, the vectors may be trained and from that point lose its connection to the language model.LabelBasedTransformerRepresentation, in contrast, maintain the connection between the labels and the representations, but allow to fine-tune the language model behind it.The second option requires significantly more memory and computational cost, since the parameters of the language model require gradients, too.
Typically, you would not directly pass the label-based representations to simple (functional) interaction functions, such as
TransE,RotatEorDistMult, but rather prefer parametric ones, e.g.,ERMLP,which allow further projections into a more suitable space.