Different test results by using dgl_eval

[jiag19]

Hi there,

I found I couldn't reproduce the test results after training.

After training, I got:

-------------- Test result --------------
Test average MRR : 0.2607070246959733
Test average MR : 847.37625
Test average HITS@1 : 0.16041666666666668
Test average HITS@3 : 0.2991666666666667
Test average HITS@10 : 0.45958333333333334

But when I use dgl_eval command to evaluate the saved embeddings, I got:

-------------- Test result --------------
Test average MRR: 0.09140389248502755
Test average MR: 5974.63
Test average HITS@1: 0.034583333333333334
Test average HITS@3: 0.08208333333333333
Test average HITS@10: 0.21875

The command I use is:

dglke_eval --model_name RotatE --dataset Mydata --hidden_dim 200 --gamma 12.0 --batch_size_eval 16
--gpu 0 1 2 3 4 5 6 7 --model_path ./ckpts/Mydata/RotatE_Mydata_0 --data_path ./Mydata/ --format raw_udd_hrt --data_files train.txt valid.txt test.txt

Could you please help me figure out this?

Besides, I also encountered the out-of-memory issue on a larger dataset using dgl_eval command, but it works fine on the same amount of GPUs using dgl_train.

Thanks.

[zheng-da]

what are your arguments for training?

[jiag19]

My training command is:

dglke_train --model_name RotatE --data_path ./Mydata/ --data_files train.txt valid.txt test.txt --format raw_udd_hrt --batch_size 1024 --log_interval 1000 --neg_sample_size 128 --regularization_coef 1e-07 --hidden_dim 200 --gamma 12.0 --lr 0.001 --batch_size_eval 256 --test -adv -de --max_step 250000 --num_thread 8 --neg_deg_sample --mix_cpu_gpu --num_proc 8 --gpu 0 1 2 3 4 5 6 7 --async_update --rel_part --force_sync_interval 1000 --save_path ./ckpts/Mydata --dataset Mydata --neg_sample_size_eval 10000

[classicsong]

Did you add --neg_sample_size_eval 10000 command during dglke_eval?

[jiag19]

Did you add --neg_sample_size_eval 10000 command during dglke_eval?

Thanks for your kind suggestion. It seems to work after adding "--neg_sample_size_eval 10000 ". But the results are slightly different from the training, which shows as below:

-------------- Test result --------------
Test average MRR: 0.2569992566366979
Test average MR: 846.05375
Test average HITS@1: 0.15541666666666668
Test average HITS@3: 0.2941666666666667
Test average HITS@10: 0.4608333333333333

In fact, the test results seem to be changing for each time I run the dgl_eval command, please see the following results. Does this due to the choice of random seed? Can we keep it same as the training process? Thanks.

-------------- Test result --------------
Test average MRR: 0.25908662648355973
Test average MR: 846.9891666666666
Test average HITS@1: 0.16083333333333333
Test average HITS@3: 0.2941666666666667
Test average HITS@10: 0.45666666666666667

[classicsong]

Because when using --neg_sample_size_eval 10000, it randomly sampled 10000 negative edges.

[jiag19]

Because when using --neg_sample_size_eval 10000, it randomly sampled 10000 negative edges.

Got it. But it would be better if there is an argument for user to specify the random seed. Thanks.