Skip to content

Latest commit

 

History

History
494 lines (421 loc) · 29.6 KB

experiments.rst

File metadata and controls

494 lines (421 loc) · 29.6 KB
Raw

Performance

Predictive Performance

We report AmpliGraph filtered MR, MRR, Hits@1,3,10 results for the most common datasets used in literature.

Note

AmpliGraph 1.x Benchmarks. AmpliGraph 1.x predictive power report is available here.

FB15K-237

Model MR MRR Hits@1 Hits@3 Hits@10 Hyperparameters
TransE 222 0.31 0.22 0.35 0.49 k: 400; epochs: 4000; eta: 30; loss: multiclass_nll; regularizer: LP; regularizer_params: lambda: 0.0001; p: 2; optimizer: adam; optimizer_params: lr: 0.0001; embedding_model_params: norm: 1; seed: 0; batches_count: 5;
DistMult 211 0.30 0.21 0.33 0.48 k: 300; epochs: 4000; eta: 50; loss: multiclass_nll; regularizer: LP; regularizer_params: lambda: 0.0001; p: 3; optimizer: adam; optimizer_params: lr: 0.00005; seed: 0; batches_count: 50;
ComplEx 204 0.31 0.22 0.34 0.49 k: 350; epochs: 4000; eta: 30; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 0.00005; seed: 0; regularizer: LP; regularizer_params: lambda: 0.0001; p: 3; batches_count: 10;
HolE 190 0.30 0.21 0.33 0.48 k: 350; epochs: 4000; eta: 50; loss: multiclass_nll; regularizer: LP; regularizer_params: lambda: 0.0001; p: 2; optimizer: adam; optimizer_params: lr: 0.0001; seed: 0; batches_count: 64;
RotatE 162 0.31 0.22 0.35 0.51 k:350; epochs': 700; eta: 20; loss: self_adversarial; loss_params: {'margin': 5, 'alpha': 1.0}; optimizer: 'adam'; learning_rate: 1e-05; regularizer: LP; regularizer_params: {'p': 3, 'lambda': 0.001}; batches_count: 55

Note

FB15K-237 validation and test sets include triples with entities that do not occur in the training set. We found 8 unseen entities in the validation set and 29 in the test set. In the experiments we excluded the triples where such entities appear (9 triples in from the validation set and 28 from the test set).

WN18RR

Model MR MRR Hits@1 Hits@3 Hits@10 Hyperparameters
TransE 3143 0.22 0.03 0.38 0.52 k: 350; epochs: 4000; eta: 30; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 0.0001; regularizer: LP; regularizer_params: lambda: 0.0001; p: 2; seed: 0; embedding_model_params: norm: 1; batches_count: 150;
DistMult 4832 0.47 0.43 0.48 0.54 k: 350; epochs: 4000; eta: 30; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 0.0001; regularizer: LP; regularizer_params: lambda: 0.0001; p: 2; seed: 0; batches_count: 100;
ComplEx 4356 0.51 0.47 0.52 0.58 k: 200; epochs: 4000; eta: 20; loss: multiclass_nll; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; seed: 0; regularizer: LP; regularizer_params: lambda: 0.05; p: 3; batches_count: 10;
HolE 7072 0.47 0.44 0.49 0.54 k: 200; epochs: 4000; eta: 20; loss: self_adversarial; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; seed: 0; batches_count: 50;
RotatE 1839 0.52 0.47 0.53 0.61 k: 350; epochs: 350; eta: 20; loss: self_adversarial; loss_params; {'margin': 5, 'alpha': 0.5}; optimizer: adam; learning_rate: 0.000; regularizer: 'LP'; regularizer_params; {'p': 3, 'lambda': 1e-05}; batches_count: 18

Note

WN18RR validation and test sets include triples with entities that do not occur in the training set. We found 198 unseen entities in the validation set and 209 in the test set. In the experiments we excluded the triples where such entities appear (210 triples in from the validation set and 210 from the test set).

YAGO3-10

Model MR MRR Hits@1 Hits@3 Hits@10 Hyperparameters
TransE 1210 0.50 0.41 0.56 0.67 k: 350; epochs: 4000; eta: 30; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 0.0001; regularizer: LP; regularizer_params: lambda: 0.0001; p: 2; embedding_model_params: norm: 1; seed: 0; batches_count: 100;
DistMult 2301 0.48 0.39 0.53 0.64 k: 350; epochs: 4000; eta: 50; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 5e-05; regularizer: LP; regularizer_params: lambda: 0.0001; p: 3; seed: 0; batches_count: 100;
ComplEx 3153 0.49 0.40 0.54 0.65 k: 350; epochs: 4000; eta: 30; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 5e-05; regularizer: LP; regularizer_params: lambda: 0.0001; p: 3; seed: 0; batches_count: 100
HolE 6941 0.47 0.39 0.52 0.62 k: 350; epochs: 4000; eta: 30; loss: self_adversarial; loss_params: alpha: 1; margin: 0.5; optimizer: adam; optimizer_params: lr: 0.0001; seed: 0; batches_count: 100
RotatE 1318 0.43 0.33 0.48 0.63 k: 350; epochs: 2850; eta: 30; loss: multiclass_nll; loss_params: {alpha: 1, margin: 1}; optimizer: adam; lr: 0.0001; regularizer: LP; regularizer_params: {'p': 3, 'lambda': 0.0}; seed: 0; batches_count: 110

Note

YAGO3-10 validation and test sets include triples with entities that do not occur in the training set. We found 22 unseen entities in the validation set and 18 in the test set. In the experiments we excluded the triples where such entities appear (22 triples in from the validation set and 18 from the test set).

FB15K

Warning

The dataset includes a large number of inverse relations, and its use in experiments has been deprecated. Use FB15k-237 instead.

Model MR MRR Hits@1 Hits@3 Hits@10 Hyperparameters
TransE 45 0.62 0.48 0.72 0.84 k: 150; epochs: 4000; eta: 10; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 5e-5; regularizer: LP; regularizer_params: lambda: 0.0001; p: 3; embedding_model_params: norm: 1; seed: 0; batches_count: 100;
DistMult 227 0.71 0.66 0.75 0.80 k: 200; epochs: 4000; eta: 20; loss: self_adversarial; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; seed: 0; batches_count: 50;
ComplEx 199 0.73 0.68 0.77 0.82 k: 200; epochs: 4000; eta: 20; loss: self_adversarial; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; regularizer: LP; regularizer_params: lambda: 0.0001; p: 3; seed: 0; batches_count: 100;
HolE 238 0.73 0.67 0.77 0.82 k: 200; epochs: 4000; eta: 20; loss: self_adversarial; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; seed: 0; batches_count: 20;
RotatE 222 0.70 0.59 0.80 0.88 k: 200; batch_size: 97, epochs: 1425, eta: 20, loss: self_adversarial, loss_params: {'margin': 5, 'alpha': 1.0}, regularizer_params: {'p': 3, 'lambda': 0.001}, optimizer: adam, learning_rate: 1e-05

WN18

Warning

The dataset includes a large number of inverse relations, and its use in experiments has been deprecated. Use WN18RR instead.

Model MR MRR Hits@1 Hits@3 Hits@10 Hyperparameters
TransE 278 0.66 0.42 0.88 0.95 k: 150; epochs: 4000; eta: 10; loss: multiclass_nll; optimizer: adam; optimizer_params: lr: 5e-5; regularizer: LP; regularizer_params: lambda: 0.0001; p: 3; embedding_model_params: norm: 1; seed: 0; batches_count: 100;
DistMult 699 0.82 0.71 0.92 0.95 k: 200; epochs: 4000; eta: 20; loss: nll; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; seed: 0; batches_count: 50;
ComplEx 713 0.94 0.93 0.95 0.95 k: 200; epochs: 4000; eta: 20; loss: nll; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; seed: 0; batches_count: 20;
HolE 676 0.94 0.93 0.94 0.95 k: 200; epochs: 4000; eta: 20; loss: self_adversarial; loss_params: margin: 1; optimizer: adam; optimizer_params: lr: 0.0005; seed: 0; batches_count: 50;
RotatE 222 0.95 0.94 0.96 0.97 k: 200; epochs: 1425; k: 200; eta: 20; loss: self_adversarial loss_params: {'margin': 5, 'alpha': 1.0}; optimizer: 'adam'; learning_rate: 1e-05; regularizer: LP; regularizer_params: {'p': 3, 'lambda': 0.001}; batches_count: 29

To reproduce the above results:

$ cd experiments
$ python predictive_performance.py

Note

Running predictive_performance.py on all datasets, for all models takes ~34 hours on an an Intel Xeon Gold 6226R, 256 GB, equipped with Tesla A100 40GB GPUs and Ubuntu 20.04.

Note

All of the experiments above were conducted with early stopping on half the validation set. Typically, the validation set can be found in X['valid']. We only used half the validation set so the other half is available for hyperparameter tuning.

The exact early stopping configuration is as follows:

  • x_valid: validation[::2]
  • criteria: mrr
  • x_filter: train + validation + test
  • stop_interval: 4
  • burn_in: 0
  • check_interval: 50

Note that early stopping can save a lot of training time, but it also adds some computational cost to the learning procedure. To lessen it, you may either decrease the validation set, the stop interval, the check interval, or increase the burn in.

Experiments can be limited to specific models-dataset combinations as follows:

$ python predictive_performance.py -h
usage: predictive_performance.py [-h] [-d {fb15k,fb15k-237,wn18,wn18rr,yago310}]
                                 [-m {complex,transe,distmult,hole,rotate}]

optional arguments:
  -h, --help            show this help message and exit
  -d {fb15k,fb15k-237,wn18,wn18rr,yago310}, --dataset {fb15k,fb15k-237,wn18,wn18rr,yago310}
  -m {complex,transe,distmult,hole,rotate}, --model {complex,transe,distmult,hole,rotate}

Loading Pre-Trained Models

If you want to load the pre-trained models used to obtain the above performance, have a look at :meth:`~ampligraph.pretrained_models.load_pretrained_model`.

Runtime Performance

Training the models on FB15K-237 (k=100, eta=10, batches_count=10, loss=multiclass_nll), on an Intel Xeon Gold 6226R, 256 GB, equipped with Tesla A100 40GB GPUs and Ubuntu 20.04 gives the following runtime report:

model seconds/epoch
ComplEx 0.18
RotatE 0.19
TransE 0.09
DistMult 0.10
HolE 0.18