Logical Message Passing Networks with One-hop Inference on Atomic Formula

Implementation for ICLR 2023 paper:

Logical Message Passing Networks with One-hop Inference on Atomic Formula

see the arXiv version and the OpenReview version.

In this documentation, we detail how to reproduce the results in the paper based on existing checkpoints released by other researchers.

• Logical Message Passing Networks with One-hop Inference on Atomic Formula
  • Requirement of this repository
  • Version
  • Preparation
    • (A) Prepare the dataset
    • (B1) Pretrain KGE checkpoints with external submodules
      • Choice 1: Pretrain with uma-pi1/kge
        • Step 0: Prepare the environment and config
        • Step 1: Prepare the dataset
        • Step 2: Train the checkpoint
      • Choice 2: Pretrain with facebookresearch/ssl-relation-prediction
        • Step 0: Prepare the environment
        • Step 1: Prepare the dataset
        • Step 2: Train the checkpoint
    • (B2) Convert pretrained KGE checkpoints into the acceptable format
      • Sources 1. uclnlp/cqd
  • Train LMPNN
  • Answering Existential First Order (EFO) queries
  • Summarize the results from log files
    • Sample training trajectories of LMPNN with command provided above.
    • (depreciated) LMPNN trajectory without freezed KGE (weak implementation)
  • Citing this paper

Requirement of this repository

• pytorch
• jupyter
• tqdm

Requirement of other submodules will be discussed accordingly.

My conda environment could be found at ENV1.yaml

Version

1. Update 2023/07/20. We fix the non-freezed KG embedding issue before the commit id e83a95a. New version fixes the KG embedding and is now fully consistent as the paper.

** This repo is under construction for usability. The key results for the paper can already be reproduced. **

Todo features:

• Implement CQD CO Extended, CQD(E) in the paper
• Introduce several ways to train KGE checkpoints with released repositories, which can be backbones for LMPNN.

Preparation

(A) Prepare the dataset

Please download the dataset from snap-stanford/KGReasoning.

Specifically, one can run:

mkdir data
cd data
wget http://snap.stanford.edu/betae/KG_data.zip # a zip file of 1.3G
unzip KG_data.zip

Then the data folder will contain the following folders and files:

FB15k-237-betae
FB15k-237-q2b
FB15k-betae
FB15k-q2b
KG_data.zip
NELL-betae
NELL-q2b

We rearange them into different subfolders:

mkdir betae-dataset
mv *betae betae-dataset
mkdir q2b-dataset
mv *q2b q2b-dataset

Finally, run convert_beta_dataset.py to convert the data into the graph forms for LMPNN. One can find the new dataset folders in ./data.

An example converted dataset format is

data/FB15k-237-betae
  - kgindex.json
  - train_kg.tsv
  - valid_kg.tsv
  - test_kg.tsv
  - train-qaa.json
  - valid-qaa.json
  - test-qaa.json

where

• kgindex.json file stores the relation/entity names and their ids,
• {train/valid/test}_kg.tsv stores the triples in three knowledge graphs (triples in train_kg.tsv is the subset of those in valid_kg.tsv, and triples valid_kg.tsv is also the subset of those in test_kg.tsv)
• {train/valid/test}-qaa.json stores the Query, easy Answers and hard Answers for train, valid, and test set.

(B1) Pretrain KGE checkpoints with external submodules

We consider two different repositories to pretrain the KGE checkpoints. Including

1. uma-pi1/kge
2. facebookresearch/ssl-relation-prediction

To initialize these modules, please run

git submodule update

How to train the checkpoints with these submodules is discussed in this section.

Generally, there are two steps, once the KG is prepared:

1. Convert the KG triples into the format that can be used in each submodule.
2. Train the checkpoints.

Choice 1: Pretrain with uma-pi1/kge

Step 0: Prepare the environment and config

To run libkge submodule, one need editable installation.

cd kge
pip install -e .

Step 1: Prepare the dataset

Running the following code to convert BetaE dataset into ./kge/data.

python convert_kg_data_for_kge.py

Step 2: Train the checkpoint

We provide a config in config/kge/fb15k-237-complex.yaml. Tailor the config to train checkpoints with libkge.

kge start config/kge/fb15k-237-complex.yaml --job.device cuda:0

The obtained checkpoints can be found at kge/local.

Choice 2: Pretrain with facebookresearch/ssl-relation-prediction

Step 0: Prepare the environment

pip install ogb networkx wandb

Step 1: Prepare the dataset

Running the following code to convert BetaE dataset into ./ssl-relation-prediction/data.

python convert_kg_data_for_ssl.py

Notably, in the ssl-relation-prediction/src/main.py file, the command arg parser prohibits external sources of datasets in line 47-50.

parser.add_argument(
    '--dataset', choices=datasets,
    help="Dataset in {}".format(datasets)
)

Let's change it into

parser.add_argument(
    '--dataset'
)

Step 2: Train the checkpoint

The training process can be initialized by running

cd ssl-relation-prediction
python src/main.py --dataset FB15k-237-betae \
               --score_rel True \
               --model ComplEx \
               --rank 1000 \
               --learning_rate 0.1 \
               --batch_size 1000 \
               --lmbda 0.05 \
               --w_rel 4 \
               --max_epochs 100 \
               --model_cache_path ./ckpts/FB15k-237-complex/

The obtained checkpoints can be found at ssl-relation-prediction/ckpts/FB15k-237-complex.

(B2) Convert pretrained KGE checkpoints into the acceptable format

We convert external KGE checkpoints into the format that can be loaded by LMPNN. We consider three sources of external checkpoints

1. Checkpoints released by uclnlp/cqd.
2. Checkpoints released / pretrained by uma-pi1/kge
3. Checkpoints released by facebookresearch/ssl-relation-prediction

The pretrained checkpoints are managed in the folder pretrain.

mkdir pretrain

Sources 1. uclnlp/cqd

This source of checkpoints is used to repreduced the results shown in the paper.

cd pretrain
wget http://data.neuralnoise.com/cqd-models.tgz # a .tgz file of 4.8G
tar xvf cqd-models.tgz
mv models raw_cqd_pretrain_models

Then we can convert the checkpoints into the format used in this repo.

python convert_cqd_pretrain_ckpts.py

Train LMPNN

Sample usage at FB15k-237

python3 train_lmpnn.py \
  --task_folder data/FB15k-237-betae \
  --output_dir log/FB15k-237/lmpnn-complex1k-default \
  --checkpoint_path pretrain/cqd/FB15k-237-model-rank-1000-epoch-100-1602508358.pt \
  --embedding_dim 1000 \
  --device cuda:0

Sample usage at FB15k

python3 train_lmpnn.py \
  --task_folder data/FB15k-betae \
  --checkpoint_path pretrain/cqd/FB15k-model-rank-1000-epoch-100-1602520745.pt \
  --device cuda:1 \
  --hidden_dim 8192 \
  --output_dir log/FB15k/lmpnn-complex1k-default

Sample usage at NELL

python3 train_lmpnn.py \
  --task_folder data/NELL-betae \
  --checkpoint_path pretrain/cqd/NELL-model-rank-1000-epoch-100-1602499096.pt \
  --device cuda:2 \
  --hidden_dim 8192 \
  --temp 0.1 \
  --batch_size 512 \
  --batch_size_eval_dataloader 8 \
  --batch_size_eval_truth_value 1 \
  --output_dir log/NELL/lmpnn-complex1k-default

Answering Existential First Order (EFO) queries

In this repository, the capability of answering EFO-1 queries is implemented by the reasoners.

• CQD-CO is implemented as GradientEFOReasoner, which is refered as CQD(E) in the paper.
• LMPNN is implemented as GNNEFOReasoner with LogicalGNNLayer

python train_lmpnn.py \
  --reasoner gradient \
  --device cuda:1 \
  --checkpoint_path pretrain/cqd/FB15k-237-model-rank-1000-epoch-100-1602508358.pt

Summarize the results from log files

We use script read_eval_from_log.py to summarize the results from the log file.

For example, the results on FB15k-237 in file log/FB15k-237/pretrain_complex1000-default/output.log can be summarized by the following piece of code.

python3 read_eval_from_log.py --log_file log/FB15k-237/pretrain_complex1000-default/output.log

Then the code will output the markdown table of the trajectory over valid and test set.

Sample training trajectories of LMPNN with command provided above.

For FB15k-237, a possible output trajectory could be like

Validation Set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	43.1522	8.96845	7.30686	28.5481	42.3203	12.6179	10.7356	10.8988	7.9376	5.15343	9.14102	6.64968	3.48541	3.33012	19.1651	5.55193
(10, 'mrr')	43.1469	9.72417	7.70923	29.5263	44.0089	14.4818	12.1041	11.0914	8.02962	5.83145	9.66503	7.01406	3.67668	3.64304	19.9803	5.96605
(15, 'mrr')	43.8283	9.78142	7.81307	29.8871	44.3047	15.0133	12.4008	11.139	8.22536	5.83979	9.86778	7.0962	3.70381	3.79685	20.2659	6.06089
(20, 'mrr')	43.8607	10.5252	8.27492	30.7076	45.3577	16.3521	13.0123	11.2875	8.52308	6.13657	10.185	7.29428	3.74871	3.91729	20.8779	6.25637
(25, 'mrr')	44.0598	10.411	8.37461	30.2869	45.1289	16.2044	12.8736	11.2198	8.30653	6.02084	9.82021	7.21201	3.9356	4.02344	20.7628	6.20242
(30, 'mrr')	43.8415	10.4777	8.47792	30.7736	45.5773	16.8041	13.0142	11.0647	8.35672	6.05398	10.335	7.39058	3.73573	3.95478	20.932	6.29401
(35, 'mrr')	44.0388	10.4904	8.66033	30.8991	45.9508	17.6397	13.2279	11.2514	8.44008	6.16027	10.3354	7.46953	3.80763	3.92426	21.1776	6.33942
(40, 'mrr')	43.7431	10.7154	8.9294	30.9381	46.091	17.4797	13.3665	11.1077	8.54201	6.10274	10.445	7.4784	3.8803	4.01805	21.2125	6.38489
(45, 'mrr')	44.0037	10.8814	8.71511	31.0536	46.4162	17.7432	13.4061	11.2816	8.40346	6.2916	10.5165	7.46814	3.88711	3.94484	21.3227	6.42164
(50, 'mrr')	44.1079	10.8688	8.62299	31.3121	46.5769	17.9518	13.4356	11.3143	8.59079	6.23409	10.398	7.76916	3.77844	3.99303	21.4201	6.43454
(55, 'mrr')	44.2142	11.2075	9.15606	31.7812	47.2926	19.3201	13.9342	11.3463	8.5981	6.29677	10.6873	7.93258	4.02479	4.05711	21.8722	6.59971
(60, 'mrr')	44.1963	11.2091	9.26216	31.7813	47.5042	19.5144	13.9952	11.271	8.60634	6.35608	10.7975	7.85765	4.06059	4.00284	21.9267	6.61492
(65, 'mrr')	44.1878	11.2833	9.23951	31.784	47.5272	19.8171	14.0538	11.2238	8.56164	6.31577	10.7261	7.89386	4.12592	3.97752	21.9642	6.60783
(70, 'mrr')	44.1557	11.3497	9.26543	31.8273	47.6473	19.8378	14.0356	11.1839	8.57884	6.28838	10.7356	7.88885	4.12011	4.00915	21.9868	6.60841
(75, 'mrr')	44.1301	11.3565	9.37637	31.8551	47.6066	19.965	14.0906	11.1939	8.55513	6.30744	10.7699	7.76835	4.11392	3.93539	22.0144	6.57901
(80, 'mrr')	44.1289	11.3645	9.34517	31.865	47.8433	20.0669	14.1551	11.2427	8.65353	6.30529	10.7471	7.92971	4.12401	3.95507	22.0739	6.61224
(85, 'mrr')	44.1127	11.4596	9.25204	31.7441	47.8254	19.9052	14.126	11.1764	8.62984	6.32005	10.7557	7.89449	4.16561	3.90074	22.0257	6.60732
(90, 'mrr')	44.1258	11.3737	9.32294	31.9144	47.8743	20.0821	14.1917	11.2108	8.62529	6.34314	10.7944	7.92542	4.1587	3.94927	22.0801	6.63419
(95, 'mrr')	44.1069	11.4469	9.35151	32.0017	47.8526	20.354	14.2962	11.1435	8.56965	6.39189	10.7905	7.83577	4.21076	3.96381	22.1248	6.63854
(100, 'mrr')	44.1638	11.41	9.49996	31.8933	47.9468	20.2255	14.3075	11.1681	8.64452	6.34371	10.73	7.96422	4.20087	3.91509	22.1399	6.63077

Test set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	44.8991	10.3212	8.21212	31.3772	43.8674	14.1751	13.3347	13.6166	9.15165	7.003	10.8464	6.3307	3.73892	4.149	20.995	6.41361
(10, 'mrr')	44.7485	11.0422	9.01639	32.5366	45.5268	16.815	14.8415	13.68	9.72893	7.78859	11.6098	6.77476	4.03303	4.59245	21.9929	6.95973
(15, 'mrr')	45.4805	11.4756	9.00117	32.716	46.0811	17.2183	15.2297	13.7089	9.82214	8.14234	11.7426	6.82549	3.95403	4.8102	22.3037	7.09494
(20, 'mrr')	45.5168	11.8394	9.10475	33.2281	46.7741	18.0997	15.6644	13.8107	10.1112	8.58388	12.3831	6.96503	4.17977	4.99503	22.6832	7.42136
(25, 'mrr')	45.6835	11.8071	9.46537	33.4615	46.7437	18.3839	15.7297	13.6642	10.0942	8.47826	12.0139	7.21188	4.31563	4.96695	22.7815	7.39732
(30, 'mrr')	45.5894	11.8349	9.42184	33.2727	47.1798	19.1685	16.0285	13.5988	10.0361	8.26477	12.2796	7.02654	4.05282	5.24071	22.9034	7.37289
(35, 'mrr')	45.6687	12.2878	9.57961	33.8039	47.5376	19.7294	16.4946	13.7893	9.99798	8.55637	12.4467	7.06137	4.13864	5.08787	23.2099	7.45819
(40, 'mrr')	45.401	12.1917	9.29751	33.5219	47.5335	19.7207	16.4359	13.6639	10.03	8.39218	12.417	7.24734	4.20833	5.00166	23.0885	7.45331
(45, 'mrr')	45.8186	12.365	9.92397	34.0337	47.9003	20.0247	16.6008	13.6968	9.96648	8.62486	12.8463	7.27353	4.38398	5.05679	23.37	7.63709
(50, 'mrr')	45.633	12.3816	10.1212	33.9538	48.0138	20.1938	16.5077	13.6692	10.2213	8.70553	12.7164	7.41705	4.48108	4.98651	23.4106	7.6613
(55, 'mrr')	45.8942	12.7871	10.2142	34.3762	48.621	21.5511	17.0483	13.6615	10.2328	8.79375	12.9544	7.61405	4.55091	5.12093	23.8207	7.80681
(60, 'mrr')	45.8769	13.0076	10.3082	34.548	48.4693	21.8219	17.2902	13.6215	10.2962	8.82371	13.0351	7.62129	4.6169	5.07511	23.9156	7.83441
(65, 'mrr')	45.8871	12.8002	10.2563	34.5614	48.7442	22.0247	17.1587	13.7247	10.1597	8.81356	13.0102	7.63786	4.55925	5.04153	23.9241	7.81248
(70, 'mrr')	45.915	12.9762	10.3503	34.6932	48.8386	21.9847	17.2467	13.6548	10.1373	8.75037	13.1279	7.60491	4.55571	5.04706	23.9774	7.8172
(75, 'mrr')	45.9074	12.9137	10.2342	34.7737	48.7164	22.0299	17.3165	13.663	10.1987	8.74679	13.062	7.66346	4.63074	5.10834	23.9726	7.84227
(80, 'mrr')	45.9094	12.9309	10.376	34.7534	48.8564	22.3067	17.2492	13.6232	10.1518	8.752	13.0546	7.64423	4.68432	5.12836	24.0174	7.85271
(85, 'mrr')	45.8954	12.9381	10.273	34.7352	48.873	22.3331	17.2259	13.5965	10.1396	8.73849	13.0252	7.67358	4.59914	5.08056	24.0011	7.8234
(90, 'mrr')	45.84	12.8973	10.3091	34.8627	49.0908	22.4607	17.3106	13.5644	10.1515	8.77689	13.1038	7.59491	4.66412	5.04967	24.0541	7.83788
(95, 'mrr')	45.8448	12.8836	10.3528	34.8965	49.0815	22.5072	17.3211	13.4828	10.119	8.78041	13.0549	7.62073	4.70466	5.098	24.0544	7.85175
(100, 'mrr')	45.8614	13.0062	10.4779	34.9732	49.0683	22.5373	17.3939	13.578	10.2351	8.69003	13.0802	7.68441	4.70692	5.01153	24.1257	7.83462

For FB15k dataset, a possible training trajectory could be

Validation set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	72.764	18.1599	14.3685	50.1483	63.4033	20.3927	21.7411	24.0656	17.3729	16.0573	14.3387	8.96579	5.63852	8.01369	33.6018	10.6028
(10, 'mrr')	73.2114	20.7276	16.2229	50.9103	64.7822	24.5783	24.0674	24.2767	18.4504	16.0651	14.6001	9.98227	6.18293	9.6944	35.2475	11.305
(15, 'mrr')	74.041	21.7555	16.8663	51.4346	65.7271	26.7591	25.3408	24.7065	18.9669	16.5432	14.7984	10.4507	6.67602	10.1124	36.1775	11.7161
(20, 'mrr')	74.4192	22.4445	17.3096	51.8621	66.3508	28.1335	26.2893	25.1039	19.3605	17.0578	15.3382	10.7319	6.7229	10.4774	36.8081	12.0657
(25, 'mrr')	74.6472	23.0718	17.8927	52.3799	66.7333	29.4432	27.0306	25.102	19.7324	17.0759	15.3593	11.0009	7.12063	10.7602	37.337	12.2634
(30, 'mrr')	74.6737	23.4607	17.8685	52.2627	66.8866	30.3678	27.282	25.3029	19.5946	17.0104	15.299	11.007	7.04523	10.8055	37.5222	12.2334
(35, 'mrr')	74.9351	23.8835	18.3777	52.3996	67.0009	30.8495	27.6875	25.4789	20.0324	16.8463	15.297	11.1668	7.17088	10.9909	37.8495	12.2944
(40, 'mrr')	75.1099	24.4404	18.5153	52.6634	67.2669	31.201	28.1146	25.4405	20.0116	17.1899	15.7448	11.5347	7.43448	11.1738	38.0848	12.6155
(45, 'mrr')	75.175	24.4533	18.7198	52.7764	67.4387	32.2673	28.5103	25.4906	20.0293	17.2964	15.8747	11.5726	7.28784	11.2956	38.3179	12.6654
(50, 'mrr')	75.2242	24.5531	18.645	53.0605	67.4554	32.5306	28.4636	25.503	20.0741	17.2653	15.9408	11.5878	7.41323	11.3201	38.3899	12.7055
(55, 'mrr')	75.7348	25.3315	19.4048	53.5306	68.1116	34.0854	29.3051	25.6027	20.3288	17.5561	16.1879	11.9783	7.51217	11.4066	39.0484	12.9282
(60, 'mrr')	75.861	25.5841	19.5346	53.7438	68.2576	34.5876	29.5732	25.6164	20.4787	17.6198	16.3073	12.0556	7.65011	11.4113	39.2486	13.0088
(65, 'mrr')	75.9349	25.7382	19.6538	53.6697	68.3454	34.8048	29.6301	25.6537	20.4627	17.6294	16.3302	12.139	7.6883	11.4152	39.3215	13.0404
(70, 'mrr')	75.9818	25.7803	19.7334	53.8455	68.6032	35.0239	29.7125	25.6243	20.5497	17.6473	16.4369	12.1637	7.76578	11.4374	39.4283	13.0902
(75, 'mrr')	76.0647	25.8406	19.8401	53.9193	68.5568	35.2081	29.7754	25.6366	20.5592	17.6685	16.3975	12.1659	7.8314	11.3745	39.489	13.0875
(80, 'mrr')	76.0764	25.815	19.8477	53.8205	68.5533	35.353	29.7676	25.6507	20.5044	17.6077	16.4779	12.226	7.8704	11.3885	39.4876	13.1141
(85, 'mrr')	76.1621	25.8629	19.9085	53.8178	68.594	35.352	29.8925	25.6998	20.6121	17.5867	16.5415	12.2125	7.92277	11.3823	39.5446	13.1292
(90, 'mrr')	76.1505	25.8591	19.941	53.8449	68.7232	35.3287	30.0221	25.7584	20.5822	17.6385	16.4569	12.2411	7.92856	11.4002	39.5789	13.1331
(95, 'mrr')	76.1381	25.9548	19.9634	53.8797	68.7455	35.6256	29.9943	25.7073	20.6724	17.622	16.6394	12.2738	7.97295	11.4501	39.6312	13.1916
(100, 'mrr')	76.2425	25.9843	20.0225	53.9112	68.7771	35.6986	30.0373	25.8397	20.8087	17.6471	16.7057	12.2231	7.98628	11.414	39.7024	13.1952

|Test set| 1p | 2p | 3p | 2i | 3i | pi | ip | 2u | up | 2in | 3in | inp | pin | pni | epfo mean | Neg mean |

	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	80.0214	26.1631	20.6792	63.1397	70.4824	26.7026	31.8146	35.0759	25.2013	26.7359	26.8873	10.0949	6.86781	11.2488	42.1422	16.3669
(10, 'mrr')	80.6118	29.9317	23.0558	64.1709	71.9207	32.2211	35.1749	34.2256	27.085	26.0799	27.0576	11.7576	7.76882	13.5857	44.2664	17.2499
(15, 'mrr')	81.6128	32.087	24.2246	64.7785	72.8758	35.1365	36.9434	35.6952	28.4065	27.7164	27.5466	12.3355	8.14011	14.713	45.7512	18.0903
(20, 'mrr')	82.1223	32.9022	24.7883	65.2956	73.2986	36.3822	37.6886	36.0638	28.7675	29.0385	27.9598	12.9415	8.56375	15.3342	46.3677	18.7676
(25, 'mrr')	82.4252	34.2613	25.7724	65.5153	73.8097	38.1548	38.6816	36.4394	29.2493	28.9039	28.0133	13.1606	8.90367	15.7501	47.1454	18.9463
(30, 'mrr')	82.6365	34.6101	25.5823	65.5032	74.0651	38.9272	39.0842	36.2058	29.4809	28.8655	28.0349	13.4159	8.92127	15.4652	47.3439	18.9406
(35, 'mrr')	82.8815	35.4994	26.4092	65.9638	74.3764	40.0052	39.5615	36.8258	29.8588	28.8328	27.9218	13.6509	9.0599	15.8999	47.9313	19.0731
(40, 'mrr')	83.0528	35.6612	26.4634	66.229	74.5047	40.617	39.6554	36.4271	29.9298	28.8021	28.2275	13.9195	9.4794	16.4576	48.06	19.3772
(45, 'mrr')	83.136	36.7407	26.5522	66.3764	74.7943	42.0012	41.0161	36.5704	30.1945	29.4151	28.6116	13.789	9.48379	16.541	48.598	19.5681
(50, 'mrr')	83.3121	36.643	26.7993	66.705	74.9823	42.057	40.0862	36.6947	30.2306	28.8638	28.5118	13.9804	9.23777	16.38	48.6122	19.3948
(55, 'mrr')	84.0474	38.0631	28.1092	67.6499	76.0039	44.2674	41.6678	36.5543	30.9187	29.0836	29.0125	14.499	9.72798	16.6574	49.698	19.7961
(60, 'mrr')	84.2658	38.3675	28.4186	67.8149	76.2868	44.6406	41.99	36.5638	31.2279	29.1775	29.1025	14.6062	9.85346	16.714	49.9529	19.8907
(65, 'mrr')	84.3822	38.5743	28.5948	67.9548	76.3456	44.9868	42.1976	36.7303	31.1506	29.1546	29.2271	14.7821	9.87208	16.7348	50.1019	19.9541
(70, 'mrr')	84.4594	38.726	28.5233	67.9601	76.3959	45.4252	42.4645	36.632	31.2578	29.2664	29.2771	14.7912	9.95378	16.6008	50.2049	19.9779
(75, 'mrr')	84.5264	38.7699	28.6037	68.0511	76.5039	45.4822	42.545	36.7839	31.2311	29.1764	29.2714	14.7585	9.94432	16.6274	50.2775	19.9556
(80, 'mrr')	84.6096	38.6981	28.5022	68.0766	76.4818	45.5297	42.3989	36.7637	31.2135	29.102	29.3295	14.8571	9.99067	16.6122	50.2527	19.9783
(85, 'mrr')	84.6776	38.6621	28.4035	68.0234	76.4476	45.7052	42.4011	36.6235	31.1659	29.134	29.289	14.8435	9.99	16.5764	50.2344	19.9666
(90, 'mrr')	84.7225	38.9814	28.4973	68.0495	76.5384	45.9682	42.7049	36.6288	31.2611	29.2248	29.3559	14.8518	9.99594	16.5391	50.3725	19.9935
(95, 'mrr')	84.7482	39.0329	28.4649	68.1035	76.3899	46.021	42.7329	36.7371	31.1983	29.1817	29.2837	14.8702	10.0342	16.5788	50.381	19.9897
(100, 'mrr')	84.8463	39.3027	28.729	68.1155	76.6316	46.1163	42.9275	36.8473	31.4271	29.1328	29.2369	14.852	10.0518	16.6147	50.5493	19.9776

For NELL dataset, a possible training trajectory could be

Validation set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	51.9866	16.4089	13.3687	31.5865	43.7179	18.1878	19.2869	14.4951	11.9546	3.18955	9.70108	10.7595	3.79393	4.22667	24.5548	6.33414
(10, 'mrr')	58.0504	18.9486	16.2135	35.7542	49.4412	22.1829	22.2309	15.9189	13.0717	8.04612	10.5261	12.1496	4.10794	4.4874	27.9791	7.86342
(15, 'mrr')	58.1901	19.5215	16.9156	36.2875	50.5697	23.5127	23.0488	15.886	13.1331	8.38497	10.8321	12.4829	3.99677	4.58691	28.5628	8.05675
(20, 'mrr')	58.1312	19.7626	17.1061	36.6888	51.3984	24.2156	23.8144	15.744	13.2207	8.42841	10.7424	12.3675	4.01913	4.49603	28.898	8.0107
(25, 'mrr')	58.2135	19.3948	16.7125	37.0836	51.5466	23.8573	23.4707	15.7727	13.1974	8.13385	10.7643	12.3361	4.0185	4.51245	28.8055	7.95303
(30, 'mrr')	58.3422	19.9796	16.9005	37.3361	52.2049	25.0439	24.258	15.8504	13.529	8.25035	11.0144	12.8912	3.94366	4.48501	29.2716	8.11691
(35, 'mrr')	58.3317	19.8246	17.0437	37.0317	52.0876	24.9918	24.1882	15.7776	13.2863	8.11579	11.0137	12.5773	3.95141	4.43461	29.1737	8.01855
(40, 'mrr')	58.4374	20.0042	17.2376	37.5069	52.4436	25.5753	24.3363	15.8403	13.2857	8.04121	11.0236	12.65	4.00914	4.48272	29.4075	8.04135
(45, 'mrr')	58.3851	20.0087	17.1159	37.2356	52.586	25.8343	24.3349	15.7666	13.269	7.99957	10.8519	12.4951	3.96799	4.50756	29.3929	7.96441
(50, 'mrr')	58.4344	20.066	17.2659	37.5026	52.583	25.8256	24.5214	15.8159	13.3767	8.00606	10.9744	12.7093	3.98905	4.44717	29.4879	8.02518
(55, 'mrr')	58.5214	20.3003	17.4971	37.7731	53.2195	26.6214	24.9088	15.8859	13.3979	7.99773	11.0119	12.8666	4.00092	4.609	29.7917	8.09724
(60, 'mrr')	58.5588	20.2791	17.4278	37.794	53.4732	26.7336	24.8551	15.9778	13.3923	8.05524	10.9982	12.7967	4.03883	4.58863	29.8324	8.09553
(65, 'mrr')	58.5393	20.3012	17.3731	37.7334	53.5801	26.7743	25.0042	15.9129	13.4806	7.98234	10.9909	12.871	4.0504	4.57675	29.8555	8.09429
(70, 'mrr')	58.5464	20.3899	17.636	37.8725	53.5272	26.8348	25.017	15.8722	13.3682	8.00063	10.9787	12.8349	4.09206	4.54549	29.896	8.09037
(75, 'mrr')	58.5244	20.3452	17.5419	37.9096	53.6987	26.9608	24.9611	15.8747	13.3377	8.0151	10.9958	12.8542	4.07362	4.58094	29.906	8.10395
(80, 'mrr')	58.4666	20.3634	17.4013	37.864	53.7288	26.872	25.0944	15.8941	13.4476	8.00644	11.0202	12.889	4.03659	4.56708	29.9036	8.10385
(85, 'mrr')	58.5246	20.3625	17.4951	37.8034	53.7097	26.8793	25.0511	15.7963	13.332	8.00327	10.9794	12.8392	4.04728	4.55166	29.8838	8.08415
(90, 'mrr')	58.5385	20.2787	17.4881	37.8201	53.708	26.9801	25.1586	15.8513	13.3464	8.01791	10.9962	12.8424	4.04243	4.56984	29.9078	8.09376
(95, 'mrr')	58.5262	20.3174	17.407	37.9197	53.704	27.007	24.9786	15.8147	13.3705	8.03061	11.0391	12.7607	4.0563	4.51442	29.8939	8.08023
(100, 'mrr')	58.4876	20.3739	17.5426	37.8685	53.7853	27.0437	25.0912	15.8391	13.3677	8.03276	11.0813	12.8224	4.02579	4.43816	29.9333	8.08009

Test set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	53.3178	17.4509	13.921	34.3557	42.5244	18.9655	19.9409	15.5624	14.1614	3.09049	9.33451	10.4117	3.55259	3.61193	25.5778	6.00025
(10, 'mrr')	60.0538	20.7879	16.6129	38.033	47.0486	24.1357	23.079	17.3772	15.8912	8.24614	10.7019	11.5071	3.87626	4.72569	29.2244	7.81142
(15, 'mrr')	60.0531	21.0174	17.3599	38.6041	48.159	25.4717	23.4817	17.3495	15.6388	8.76846	10.9286	11.972	3.92587	4.70564	29.6817	8.06013
(20, 'mrr')	60.0223	21.3595	17.3279	38.7731	48.7933	25.9714	24.0966	17.2853	15.8172	8.98501	10.8567	11.8679	4.01083	4.63997	29.9385	8.07209
(25, 'mrr')	60.1999	21.4644	17.2901	38.9674	49.2416	26.0032	24.1828	17.2671	15.7017	8.6811	11.0215	11.7445	3.92222	4.81752	30.0353	8.03736
(30, 'mrr')	60.39	21.9168	17.0032	39.2801	49.9484	27.3184	24.6839	17.1173	15.9443	8.57809	10.8312	12.1397	3.83815	4.77605	30.4003	8.03264
(35, 'mrr')	60.3503	21.7325	17.2967	39.1128	49.7495	27.0008	24.6514	17.0884	15.861	8.59063	10.7803	12.1338	3.84769	4.80637	30.3159	8.03174
(40, 'mrr')	60.4626	22.0204	17.3283	39.5281	50.6019	27.4767	24.6566	17.0303	15.7524	8.46947	10.864	12.333	3.92515	4.89782	30.5397	8.09789
(45, 'mrr')	60.3293	22.1564	17.5824	39.4073	50.1337	27.8943	25.0868	16.9976	15.9754	8.70057	10.8894	12.1503	3.98509	4.82957	30.6181	8.111
(50, 'mrr')	60.5006	21.9947	17.398	39.4997	50.1538	27.7933	24.8773	16.97	15.7931	8.42343	10.8244	12.2081	3.92818	4.73758	30.5534	8.02432
(55, 'mrr')	60.5633	22.3015	17.636	39.9431	50.9257	28.5783	25.3599	17.0022	15.833	8.50931	10.8985	12.411	4.0115	4.83301	30.9048	8.13267
(60, 'mrr')	60.5973	22.2343	17.6546	39.9555	50.9812	28.7155	25.2521	17.0157	15.7824	8.48907	10.9854	12.2893	3.96769	4.85918	30.9098	8.11811
(65, 'mrr')	60.5589	22.3278	17.6482	40.0772	51.1738	28.8743	25.255	16.9732	15.8177	8.4574	10.9471	12.3755	3.99949	4.80515	30.9673	8.11693
(70, 'mrr')	60.5047	22.2669	17.734	40.0701	51.2012	28.9174	25.2583	16.8859	15.7568	8.42383	11.0485	12.3892	4.00967	4.80891	30.955	8.13604
(75, 'mrr')	60.5458	22.2862	17.6367	40.2007	51.2726	28.9536	25.2717	16.8917	15.7468	8.43721	10.9711	12.4433	3.97777	4.82057	30.9784	8.13
(80, 'mrr')	60.5429	22.2779	17.5793	40.0852	51.1626	28.9895	25.3941	16.8825	15.7659	8.48395	10.9586	12.4836	3.98999	4.7884	30.9644	8.14091
(85, 'mrr')	60.5592	22.352	17.6074	40.2472	51.2478	28.9383	25.3371	16.908	15.7983	8.41977	10.9511	12.4619	4.01124	4.80731	30.9995	8.13026
(90, 'mrr')	60.5871	22.3782	17.4865	40.2134	51.3309	29.0312	25.318	16.9407	15.744	8.39682	10.9253	12.4119	4.03235	4.80307	31.0033	8.11389
(95, 'mrr')	60.5521	22.3003	17.5146	40.4396	51.5141	28.99	25.3643	16.8287	15.7321	8.39906	11.0132	12.4137	4.01778	4.77633	31.0262	8.12402
(100, 'mrr')	60.5551	22.4637	17.6964	40.2327	51.3726	29.0957	25.449	16.8115	15.7569	8.42328	10.9956	12.3207	4.04476	4.85645	31.0482	8.12816

Results from versions before e83a95a are obtained from the training process where both the KG embedding and the MLP is trainable, which is an accidental mistake and NOT what proposed in the paper. The results with trainable KGE is be weaker than those with freezed KGE which is proposed in the paper. Click to see the sample output trajectory of trainable KGE (results of weak implementation of LMPNN)

(depreciated) LMPNN trajectory without freezed KGE (weak implementation)

For FB15k-237, a possible output trajectory could be like

Validation Set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	43.3374	9.3963	7.36107	28.8171	43.2848	12.7592	11.338	10.9825	8.06806	4.89454	9.03825	6.84014	3.57948	3.43009	19.4827	5.5565
(10, 'mrr')	43.5847	10.2061	8.38958	30.1158	44.8668	13.7348	12.4519	11.1331	8.40302	5.50171	9.92851	7.19045	3.68003	3.68664	20.3207	5.99747
(15, 'mrr')	43.1136	9.93098	8.41377	30.7293	45.7028	14.1373	12.4096	11.17	8.48811	5.90808	9.97395	7.14056	3.78556	3.88563	20.455	6.13876
(20, 'mrr')	43.6212	10.5731	8.92229	30.8057	45.9869	15.2658	13.1256	11.0792	8.84129	5.74387	10.4465	7.32727	3.87902	3.76653	20.9135	6.23263
(25, 'mrr')	43.5993	10.6286	9.14751	30.7887	46.2739	16.2113	13.8994	11.2555	8.81353	5.98244	10.6118	7.51721	3.7699	3.55735	21.1798	6.28774
(30, 'mrr')	43.6251	10.8747	9.03938	30.9388	46.5563	16.4211	13.543	11.2324	9.15598	5.83169	10.5604	7.63332	3.9878	3.64097	21.2652	6.33083
(35, 'mrr')	43.4778	11.0762	9.3528	31.3629	47.202	17.1742	13.701	11.2211	9.16062	5.97082	10.7614	7.62482	3.90409	3.7512	21.5254	6.40247
(40, 'mrr')	43.4053	11.0612	9.56793	31.3726	47.2989	16.9117	13.6576	10.8811	9.21064	5.86364	10.6366	7.91589	3.90675	3.52537	21.4852	6.36965
(45, 'mrr')	43.2611	11.0465	9.57849	31.2661	47.44	17.705	13.9036	11.0584	9.11347	5.98817	10.6902	7.86165	3.93147	3.75511	21.5969	6.44531
(50, 'mrr')	42.9117	11.4134	9.45842	31.4618	47.5948	17.8229	13.9	11.0091	9.40493	5.89662	10.722	7.65385	4.07289	3.60841	21.6641	6.39074
(55, 'mrr')	43.2503	11.6637	9.83592	31.7507	48.095	19.3408	14.4881	11.0594	9.40362	6.04338	10.8817	8.20831	4.18475	3.58321	22.0986	6.58026
(60, 'mrr')	43.3788	11.7293	10.0087	31.8454	48.1824	19.6966	14.5867	11.0317	9.52452	6.06797	10.8322	8.18048	4.23077	3.65369	22.2205	6.59303
(65, 'mrr')	43.265	11.6121	10.0383	32.0212	48.3501	19.8917	14.6462	11.0356	9.44041	6.03938	10.9391	8.08408	4.3106	3.65148	22.2556	6.60492
(70, 'mrr')	43.2776	11.6904	9.98802	32.0637	48.5154	20.0188	14.7324	11.0485	9.54078	5.98941	11.0414	8.21996	4.22696	3.7204	22.3195	6.63963
(75, 'mrr')	43.3091	11.7581	10.0377	32.1079	48.6767	20.1973	14.8317	11.0657	9.52549	6.03904	11.0798	8.28582	4.34577	3.63607	22.39	6.6773
(80, 'mrr')	43.3248	11.7603	9.98525	32.2027	48.5901	20.2498	14.8366	11.0489	9.5386	5.98985	11.0667	8.26597	4.35415	3.66858	22.393	6.66906
(85, 'mrr')	43.2465	11.7877	10.0583	32.1016	48.7363	20.4814	14.9262	10.9962	9.51707	6.02979	11.2297	8.3184	4.39278	3.72155	22.4279	6.73846
(90, 'mrr')	43.376	11.6823	10.0461	32.241	48.6458	20.4281	14.8565	11.0369	9.5371	6.03664	11.1348	8.34831	4.32066	3.62985	22.4278	6.69404
(95, 'mrr')	43.291	11.7701	9.97534	32.3046	48.8179	20.5445	14.8694	11.1149	9.53749	6.02634	11.2056	8.25653	4.34473	3.6933	22.4695	6.70531
(100, 'mrr')	43.2896	11.8108	9.95793	32.3719	48.9575	20.4634	14.8929	11.0351	9.56184	6.06656	11.1147	8.34275	4.35363	3.69552	22.4823	6.71463

Test set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	44.9465	10.4843	8.6229	31.7221	44.4341	14.2661	14.1401	13.6675	9.80541	6.86755	11.2436	6.54754	3.84052	4.55932	21.3432	6.61171
(10, 'mrr')	45.1145	11.2116	9.19404	32.7899	46.5547	15.7508	15.1701	13.5092	9.95767	7.41947	11.6497	6.58225	3.93812	4.52801	22.1392	6.8235
(15, 'mrr')	44.7565	11.6183	9.65737	33.0242	46.9324	16.0322	15.6126	13.9755	10.2088	7.83579	12.0131	6.88489	4.00784	4.6131	22.4242	7.07094
(20, 'mrr')	45.3177	12.0233	9.57721	33.2067	47.472	17.4125	16.2467	13.6488	10.4001	8.05987	12.2843	7.14386	4.14163	4.45509	22.8117	7.21695
(25, 'mrr')	45.1459	12.1309	9.8724	33.4177	47.7942	17.7643	16.4459	13.4371	10.4981	7.80966	12.3091	7.22651	4.23409	4.35564	22.9452	7.187
(30, 'mrr')	45.2453	12.516	10.2946	33.8225	47.6455	18.1683	16.3805	13.5795	10.5048	7.82047	12.0024	7.65289	4.37242	4.55625	23.1285	7.28089
(35, 'mrr')	44.9832	12.6714	10.3581	33.9382	48.1022	19.0562	16.7391	13.4728	10.3644	7.80266	12.4172	7.50164	4.27899	4.18954	23.2984	7.23801
(40, 'mrr')	45.0497	12.9713	10.5534	33.9162	48.2494	19.4182	17.0646	13.3876	10.7647	7.76955	12.0863	7.76417	4.58433	4.18861	23.4861	7.27859
(45, 'mrr')	44.8709	12.7598	10.6207	34.0123	48.4076	20.1168	16.7293	13.4826	10.7217	7.7207	12.2151	7.79834	4.38069	4.2472	23.5246	7.27242
(50, 'mrr')	44.4331	12.9428	10.3695	33.9687	48.3719	20.2204	17.002	13.2265	10.7709	7.51472	12.3501	7.86527	4.52629	4.30918	23.4784	7.31311
(55, 'mrr')	44.8656	13.2094	10.7748	34.5006	48.9649	21.6445	17.5028	13.3423	10.9163	7.63899	12.6289	8.30786	4.61518	4.30892	23.969	7.49997
(60, 'mrr')	44.9011	13.3243	10.8495	34.6471	49.0874	22.1415	17.4582	13.3771	10.8835	7.62007	12.5587	8.2811	4.68471	4.34016	24.0744	7.49695
(65, 'mrr')	44.882	13.4375	10.8931	34.7594	49.126	22.3102	17.6696	13.408	10.9255	7.63332	12.6301	8.24105	4.70683	4.2976	24.1568	7.50179
(70, 'mrr')	44.8828	13.651	11.0718	34.7735	49.2482	22.703	17.7788	13.4791	11.0766	7.68911	12.7907	8.35583	4.76243	4.40152	24.2961	7.59992
(75, 'mrr')	44.9174	13.555	11.012	34.906	49.3564	22.7873	17.6738	13.396	11.0057	7.63671	12.7132	8.41619	4.61401	4.29353	24.2899	7.53473
(80, 'mrr')	44.9689	13.694	11.0733	34.9882	49.5035	22.9782	17.7274	13.3799	11.0732	7.6816	12.7935	8.37103	4.75101	4.35969	24.3763	7.59136
(85, 'mrr')	44.8746	13.5736	11.0516	35.0654	49.612	23.1731	17.7409	13.4019	11.0125	7.63303	12.8699	8.32122	4.76494	4.31249	24.3895	7.58032
(90, 'mrr')	44.963	13.6078	11.0434	35.0212	49.539	22.9025	17.6933	13.3906	10.9815	7.68771	12.8578	8.27871	4.70053	4.26779	24.3491	7.5585
(95, 'mrr')	44.8846	13.665	11.0756	35.1892	49.8396	23.0492	17.7944	13.3872	11.0303	7.66467	12.8382	8.39552	4.71225	4.3251	24.435	7.58714
(100, 'mrr')	44.9048	13.6668	11.1112	35.2322	49.6774	22.859	17.8386	13.3444	11.0184	7.70513	12.857	8.45513	4.74044	4.26359	24.4059	7.60425

For FB15k dataset, a possible training trajectory could be

Validation set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	72.5264	18.9707	15.167	50.393	64.2903	21.3955	22.4212	23.7883	17.7087	15.89	14.4604	9.5854	6.1095	8.80865	34.0735	10.9708
(10, 'mrr')	73.4743	21.3927	16.8809	51.2315	65.4991	25.2254	25.0429	24.3405	18.9479	16.2819	14.5931	10.1545	7.01239	10.0011	35.7817	11.6086
(15, 'mrr')	74.0354	22.2156	17.726	51.3907	65.5969	27.1767	25.7572	24.2847	19.393	16.4947	14.6412	10.7316	7.59753	10.4718	36.3974	11.9873
(20, 'mrr')	74.1509	23.2951	18.3861	52.2457	66.8295	28.6767	26.7481	24.5087	20.1395	16.8086	15.0185	10.912	8.17509	10.3257	37.22	12.248
(25, 'mrr')	74.3886	23.6987	18.7169	52.2513	66.6926	29.6802	27.3806	24.6967	20.545	16.4154	15.0412	11.3016	8.45187	10.6056	37.5612	12.3631
(30, 'mrr')	74.5874	24.1382	18.9701	52.1197	67.0708	31.0522	27.5473	24.7312	20.6051	16.5712	15.397	11.4983	8.42595	10.7639	37.8691	12.5313
(35, 'mrr')	74.6497	24.6292	19.136	52.0571	66.9698	32.1509	28.2427	24.8105	20.8558	16.3211	15.2092	11.6967	8.96677	10.7479	38.1669	12.5883
(40, 'mrr')	74.8573	24.9687	19.3029	52.2527	67.0853	33.2529	28.3276	24.8803	21.0188	16.4776	15.1542	11.8346	9.1182	11.0053	38.4385	12.718
(45, 'mrr')	74.8077	25.2605	19.5295	52.5364	67.5456	33.4174	28.8868	24.7764	20.9093	16.5528	15.2427	11.7762	9.26389	11.0442	38.63	12.776
(50, 'mrr')	74.7147	25.3948	19.6865	52.1841	66.9479	33.9548	28.9421	24.589	21.1452	16.2819	15.3491	11.9337	9.17793	10.933	38.6177	12.7351
(55, 'mrr')	75.2392	26.0655	20.4374	52.7575	68.0126	35.7402	29.763	24.9457	21.5412	16.7172	15.8523	12.292	9.61107	11.1913	39.3892	13.1328
(60, 'mrr')	75.338	26.2711	20.4958	53.005	68.2882	36.3051	29.9832	24.9604	21.5683	16.7698	16.0298	12.4912	9.97094	11.293	39.5795	13.311
(65, 'mrr')	75.4915	26.3433	20.7189	53.1031	68.4577	36.5135	30.125	24.9532	21.6608	16.7159	16.1205	12.445	10.1391	11.28	39.7075	13.3401
(70, 'mrr')	75.6082	26.5294	20.858	53.2512	68.4007	36.8932	30.2989	25.08	21.7732	16.8102	16.2661	12.5809	10.1644	11.3312	39.8548	13.4306
(75, 'mrr')	75.6264	26.779	20.9627	53.4878	68.5542	36.972	30.3147	25.0093	21.8702	16.8719	16.4148	12.5987	10.1621	11.4264	39.9529	13.4948
(80, 'mrr')	75.6596	26.7891	20.9883	53.5604	68.6648	37.0346	30.4991	25.0008	21.9547	16.9228	16.4881	12.6153	10.5	11.3379	40.0168	13.5728
(85, 'mrr')	75.7871	26.9072	21.0725	53.4992	68.6661	37.3166	30.5706	25.0393	21.9667	16.9707	16.5363	12.6744	10.5678	11.3997	40.0917	13.6298
(90, 'mrr')	75.7685	26.9004	21.1033	53.5543	68.6671	37.4762	30.6839	25.0264	21.9165	16.9988	16.5336	12.6405	10.6902	11.2698	40.1219	13.6266
(95, 'mrr')	75.8367	27.112	21.1882	53.6209	68.8099	37.4012	30.7478	25.1237	22.027	17.024	16.577	12.7249	10.588	11.4293	40.2075	13.6687
(100, 'mrr')	75.821	26.9972	21.1739	53.5756	68.8016	37.3668	30.7325	25.0958	21.967	16.9533	16.6115	12.6655	10.5621	11.4197	40.1702	13.6424

Test set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	79.6252	27.0495	22.0475	63.0068	71.0271	26.8145	31.631	33.8454	26.428	25.348	26.4778	11.1268	7.37001	12.2931	42.3861	16.5231
(10, 'mrr')	80.8636	30.8524	24.7873	64.5688	72.6059	32.4962	34.8987	34.8972	28.6328	26.4557	26.8073	12.2922	8.45513	14.3868	44.9559	17.6794
(15, 'mrr')	81.5185	32.2857	25.5074	64.8339	72.9823	34.3122	36.1444	34.3031	29.9918	26.6691	27.0838	13.1753	9.03323	14.2974	45.7644	18.0518
(20, 'mrr')	81.9271	34.1768	26.5718	65.453	73.8918	36.7933	37.8788	34.451	31.1975	26.7253	26.6824	13.0502	9.25748	14.8176	46.9268	18.1066
(25, 'mrr')	82.1761	35.3178	27.2472	65.4745	74.0307	38.0896	38.8817	34.9898	31.5301	26.2991	26.4839	13.6902	9.53723	14.8901	47.5264	18.1801
(30, 'mrr')	82.3555	35.6731	27.6514	65.0369	73.8001	40.1415	38.939	34.6648	31.4103	25.8967	26.2618	14.1097	9.66685	14.7401	47.7414	18.135
(35, 'mrr')	82.5593	36.5284	27.9251	65.3617	74.0039	40.9399	39.6183	34.3864	31.5693	25.4552	26.0457	14.1313	9.76101	14.6804	48.0992	18.0147
(40, 'mrr')	82.9059	37.9752	28.8003	65.5398	74.1289	42.1209	40.5559	34.4631	32.6308	25.4917	25.8993	14.6716	10.0891	14.8624	48.7912	18.2028
(45, 'mrr')	82.8705	37.8872	29.0582	65.5965	74.1744	42.4366	40.5069	34.3821	32.6076	25.2364	25.7732	14.508	10.2368	14.9662	48.8356	18.1441
(50, 'mrr')	82.9409	38.4537	29.3314	65.409	74.1168	43.6435	40.9284	34.5042	32.8697	24.8741	25.5653	14.8343	10.3195	15.0117	49.1331	18.1209
(55, 'mrr')	83.6995	40.1008	30.5288	66.6587	75.1619	45.9406	42.3054	34.6511	33.7162	25.3503	26.1966	15.3033	10.573	15.2319	50.307	18.531
(60, 'mrr')	83.8872	40.7408	30.9499	67.0044	75.4608	46.6269	42.8369	34.5836	34.0223	25.475	26.432	15.5052	10.7476	15.2947	50.6792	18.6909
(65, 'mrr')	84.0518	40.849	30.9894	67.2783	75.6766	46.6758	42.9262	34.6531	34.0006	25.4424	26.5234	15.5519	10.7439	15.2883	50.789	18.71
(70, 'mrr')	84.2106	41.3092	31.257	67.3251	75.8018	47.0875	43.2535	34.654	34.1175	25.3795	26.5757	15.6222	10.8237	15.2704	51.0018	18.7343
(75, 'mrr')	84.2636	41.5525	31.5103	67.5114	75.7793	47.4208	43.6347	34.6501	34.2698	25.5468	26.6772	15.6955	10.8787	15.3086	51.1769	18.8214
(80, 'mrr')	84.3512	41.571	31.4847	67.5161	75.975	47.2654	43.582	34.6848	34.3082	25.5773	26.7497	15.7046	10.9515	15.3087	51.1932	18.8584
(85, 'mrr')	84.4401	41.7911	31.6551	67.6595	76.0263	47.5891	43.8328	34.6937	34.4209	25.6277	26.8003	15.8049	10.9324	15.4512	51.3454	18.9233
(90, 'mrr')	84.4638	41.8142	31.7417	67.8218	76.1267	47.763	43.5757	34.6339	34.3527	25.4679	26.857	15.781	11.0279	15.3128	51.3659	18.8893
(95, 'mrr')	84.6069	41.7902	31.7677	67.8683	76.2519	47.8811	43.8575	34.5829	34.5235	25.4988	26.8536	15.815	11.0273	15.3696	51.4589	18.9129
(100, 'mrr')	84.5962	41.8732	31.7129	67.8121	76.2344	47.7903	43.8667	34.6276	34.5437	25.6159	26.8515	15.8538	11.0693	15.3367	51.4508	18.9455

For NELL dataset, a possible training trajectory could be

Validation set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	58.1204	17.1394	14.8171	35.1191	48.1434	18.216	19.8213	15.6274	12.3501	7.47669	10.6017	11.8062	4.0412	4.38074	26.5949	7.66129
(10, 'mrr')	57.9101	17.9723	15.7368	35.6915	49.0143	19.5558	20.8033	15.7196	12.6459	8.01705	10.6385	12.3644	4.12027	4.34362	27.2277	7.89678
(15, 'mrr')	57.9716	19.0338	16.2968	35.9256	49.8854	22.1367	21.8386	15.4901	12.8566	7.80956	10.5861	12.4902	4.04185	4.31369	27.9373	7.84828
(20, 'mrr')	57.4168	18.7561	16.7115	35.977	49.8357	20.4362	21.6066	15.3487	12.7226	7.66178	10.4122	12.2952	4.02626	4.42086	27.6457	7.76325
(25, 'mrr')	57.0351	19.1695	16.6744	36.701	51.1331	23.5681	22.4403	15.3076	12.8107	7.91373	10.5872	12.7712	4.07904	4.35931	28.3155	7.94209
(30, 'mrr')	57.1963	19.2891	16.9377	36.7586	51.0399	23.756	22.522	15.2679	12.7877	7.5976	10.5313	12.6761	4.02105	4.47263	28.395	7.85975
(35, 'mrr')	56.9236	19.3807	17.0099	36.9825	51.3761	23.6143	22.3366	15.2227	13.0573	7.32792	10.6499	12.4046	4.1504	4.44791	28.4337	7.79616
(40, 'mrr')	56.8403	19.4833	16.8577	36.5935	51.3677	23.9328	22.81	15.1275	13.0253	7.45651	10.7676	12.6684	4.27722	4.56407	28.4487	7.94677
(45, 'mrr')	56.5277	19.4434	16.9731	36.6514	50.8131	23.8585	22.523	14.9389	13.0073	7.48076	10.4497	12.7325	4.09825	4.52448	28.304	7.85714
(50, 'mrr')	56.5823	19.3082	17.0722	36.9401	51.1826	24.1982	22.5559	14.8032	12.8865	7.30333	10.58	12.8101	4.07694	4.2912	28.3921	7.81231
(55, 'mrr')	56.7491	19.6617	17.3232	37.4124	51.9701	25.0041	23.0953	14.9818	12.9978	7.29917	10.6776	13.0981	4.1533	4.38353	28.7995	7.92233
(60, 'mrr')	56.747	19.7013	17.3851	37.4219	51.9863	25.0253	23.2136	15.0225	13.0185	7.31288	10.7236	13.1495	4.09421	4.39861	28.8357	7.93576
(65, 'mrr')	56.7386	19.7831	17.3879	37.5423	51.9313	25.2353	23.2883	14.9899	13.0879	7.33864	10.685	13.1793	4.1111	4.405	28.8872	7.9438
(70, 'mrr')	56.7031	19.809	17.4155	37.5206	52.1556	25.4837	23.2963	15.0402	13.174	7.32878	10.7495	13.2333	4.09398	4.40422	28.9553	7.96194
(75, 'mrr')	56.6914	19.7878	17.407	37.4702	52.2481	25.3543	23.3908	15.0409	13.1152	7.31722	10.7442	13.1681	4.13749	4.43393	28.9451	7.96018
(80, 'mrr')	56.7012	19.8272	17.4449	37.5907	52.2383	25.4022	23.4482	14.989	13.0254	7.35492	10.777	13.1616	4.14352	4.39464	28.963	7.96634
(85, 'mrr')	56.7686	19.8192	17.5098	37.4502	52.1744	25.5558	23.3911	15.0243	13.064	7.28512	10.8985	13.197	4.1342	4.45095	28.973	7.99316
(90, 'mrr')	56.7139	19.8626	17.4418	37.6441	52.174	25.4519	23.5212	14.9292	13.0997	7.30827	10.9621	13.1915	4.17801	4.43536	28.982	8.01505
(95, 'mrr')	56.6114	19.8485	17.6015	37.5606	52.3552	25.4303	23.4222	14.9639	13.0974	7.3714	10.9159	13.2139	4.17094	4.42943	28.9879	8.02032
(100, 'mrr')	56.5304	19.837	17.5056	37.557	52.61	25.68	23.4784	14.9258	13.0932	7.39956	10.8602	13.1604	4.17135	4.40587	29.0242	7.99949

Test set	1p	2p	3p	2i	3i	pi	ip	2u	up	2in	3in	inp	pin	pni	epfo mean	Neg mean
(5, 'mrr')	59.9925	18.7431	15.3488	37.131	45.7525	19.6135	20.6219	17.2678	14.6512	7.65617	10.6113	11.183	3.77651	4.2455	27.6802	7.4945
(10, 'mrr')	59.3839	19.619	15.7485	37.7121	46.9139	21.1344	21.3127	16.9185	14.8788	8.27667	10.6757	11.5308	3.65816	4.32677	28.1802	7.69362
(15, 'mrr')	59.515	20.5792	16.5123	38.2234	47.3892	23.91	22.5951	16.8399	15.4843	8.11972	10.6834	12.132	3.79222	4.60354	29.0054	7.86619
(20, 'mrr')	59.0143	20.6289	17.0332	38.2814	47.7713	21.9448	22.3699	16.8168	15.3931	7.85623	10.8544	11.9253	3.80776	3.9679	28.806	7.68231
(25, 'mrr')	58.7151	21.418	17.1238	38.6686	48.7802	24.9183	22.9127	16.5135	15.4847	8.29464	10.7124	12.1862	3.90752	4.75549	29.3927	7.97125
(30, 'mrr')	58.7759	21.42	17.1072	38.729	48.6958	24.9884	22.9775	16.5603	15.5697	7.92933	10.7024	12.1284	4.05308	4.57014	29.4249	7.87666
(35, 'mrr')	58.5786	21.377	17.0758	38.9681	48.9368	25.2203	22.4404	16.4391	15.3977	7.92863	10.5248	12.2395	3.94388	4.69192	29.3815	7.86574
(40, 'mrr')	58.3774	21.4076	17.0121	39.0265	49.08	25.3503	23.0431	16.1658	15.5395	7.9038	10.4779	12.3383	3.98636	4.66744	29.4447	7.87476
(45, 'mrr')	58.0577	21.2919	17.2616	38.8111	48.8938	25.5967	23.1196	16.1437	15.38	7.93512	10.43	12.1186	3.95103	4.61853	29.3951	7.81065
(50, 'mrr')	58.2683	21.2357	17.0527	39.0259	48.6985	25.8154	22.7295	16.2029	15.108	7.81959	10.5743	11.8353	3.9347	4.52063	29.3486	7.7369
(55, 'mrr')	58.4405	21.8109	17.518	39.4844	49.5178	26.5499	23.3775	16.2771	15.5987	7.91742	10.5566	12.1564	3.93595	4.59566	29.8417	7.8324
(60, 'mrr')	58.4016	21.886	17.6116	39.6262	49.5139	26.4611	23.2744	16.2556	15.4958	7.90128	10.5776	12.3606	3.93631	4.58478	29.8362	7.87211
(65, 'mrr')	58.3851	21.8665	17.5549	39.5868	49.6965	26.753	23.4442	16.2629	15.5942	7.90126	10.5395	12.3257	4.00136	4.59561	29.9049	7.87268
(70, 'mrr')	58.3645	21.8921	17.6507	39.5146	49.9237	26.8391	23.553	16.2191	15.5815	7.85621	10.5537	12.3672	3.96091	4.63597	29.9487	7.87479
(75, 'mrr')	58.3917	21.9669	17.7499	39.7617	49.9059	26.797	23.6986	16.1471	15.7068	7.90412	10.5253	12.3506	3.97828	4.63258	30.014	7.87817
(80, 'mrr')	58.3979	22.1004	17.8992	39.8158	49.8562	27.1052	23.7746	16.2492	15.7949	7.94933	10.5536	12.3678	3.97498	4.61852	30.1104	7.89284
(85, 'mrr')	58.3532	22.0636	17.7837	39.9643	50.0419	27.1929	23.7004	16.2874	15.6986	7.92375	10.549	12.439	3.98941	4.65393	30.1207	7.911
(90, 'mrr')	58.3807	22.1138	17.7913	39.807	50.0646	27.2867	23.715	16.2238	15.7525	7.8916	10.5542	12.402	4.04165	4.60234	30.1262	7.89835
(95, 'mrr')	58.3548	21.9695	17.8834	39.8838	50.0381	27.0707	23.5618	16.1987	15.5789	7.87723	10.5658	12.3075	4.06027	4.62175	30.06	7.88651
(100, 'mrr')	58.2411	22.1544	17.8277	39.9782	50.2438	27.1149	23.6624	16.2382	15.6717	7.91782	10.6511	12.4242	3.99122	4.63306	30.1258	7.92348

Citing this paper

@inproceedings{LMPNN,
  author    = {Zihao Wang and
               Yangqiu Song and
               Ginny Y. Wong and
               Simon See},
  title     = {Logical Message Passing Networks with One-hop Inference on Atomic Formulas},
  booktitle = {The Eleventh International Conference on Learning Representations, {ICLR} 2023, Kigali Rwanda, May 1-5, 2023},
  publisher = {OpenReview.net},
  year      = {2023},
  url       = {https://openreview.net/forum?id=SoyOsp7i_l},
}