Can't Reproduce LINKX Full Batch results

[zrhhhhh123]

Hi, I try to reproduce LINKX Full-Batch results on Actor/Cornell/Texas/arxiv-year dataset, but I only get 32 test accuracy for Actor dataset, 64 for Cornell dataset, 67 for Texas dataset, 54 for arxiv-year dataset.

I follow the parameters as you show in the paper: hidden_channels ∈ {16, 32, 128, 256}, MLP_final ∈ {1, 2, 3}, MLP_A ∈ {1, 2}, MLP_X ∈ {1, 2}, learning rate ∈ {0.05, 0.01, 0.002}, dropout ∈ {0.0, 0.5}, I try many times to Combination these parameters, But I failed to reproduce the results. Could you please show the parameters used in these datasets?

[cptq]

Are you running it using our script experiments/linkx_exp.sh? In particular, are you using --runs 5, which runs it over 5 seeds? Some of these experiments (especially on the smaller datasets from other papers) can have high variance.

[zrhhhhh123]

yes, I just download the code and don't modify any parameters, and then run experiments/linkx_exp.sh, but the results are far from what you show in the paper. Another question is that in your code, don't --runs 5 represents standard deviations over 5 train/val/test splits?

[cptq]

Yes, I mispoke, you are right that 5 runs is over 5 label splits.

Thank you for the information. I presume you did not use our install.sh for package versions? Could you give me information about your environment?

[zrhhhhh123]

yes, I don't use your install.sh for package versions, my environment is:
python==3.7.13
torch==1.12.1
scipy==1.7.3
ogb==1.3.4
gdown==4.5.3
torch-scatter==2.0.9
torch-sparse==0.6.14
torch-cluster==1.6.0
torch-spline-conv==1.2.1
torch-geometric==2.1.0

[cptq]

Hi, why did you close this issue? Did you find some fix or some mistake?