Dataset split is not the standard, like in GCN.

[rllm-project]

Dear authors,
I have noticed that you use DGL to load Cora/Citeseer/Pubmed dataset. But the corresponding data split is not the correct split used in GCN paper (as pointed out in old questions in MVGRL ). Therefore, it is not fair for the compared baselines (like GCN and GAT).
Moreover, even with the same DGL split setting (as pointed out in your baseline paper [11]), MVGRL[11] also beats yours, but you did not compare this one?
For example, MVGR get 86.8% node classification on Cora, while you only get 84.2?
Are there any mistakes?

[THINK2TRY]

Hi @BeckVision , thanks for your attention to our work and careful consideration.

1. We noticed this problem before but found that the impact is minor to the performance of GraphMAE. As discussed in questions in pyg, the split of Cora in PyG and DGL is distinct. We implement GraphMAE using both DGL and PyG, and they produce almost the same results (the PyG implementation can be found in this branch).
   Cora (DGL): 84.2±0.4
   Cora (PyG): 84.2±0.6

2. For MVGRL, the performance on Cora (>86%) cannot be reproduced even with the official code, as discussed in questions in this issue. DGL also provides an implementation of MVGRL, but the result (83.5%) on Cora is still much worse than the reported result. Therefore, the ''excellent'' performance of MVGRL cannot be attributed to the DGL split.

3. We follow the setting used in many baselines to have a fair comparison, including CCA-SSG, GRACE, and BGRL. Theri results are all from DGL implementation.

If you have any other questions, welcome further discussion.

[rllm-project]

Thanks for your quick reply. However, it does seem like too much of a coincidence, with completely different data splits.
Could you add the detailed hyper-parmeters in your running cmd, and also report the classification results (with PyG) on its readme.md file?
Therefore, we can run your code and reproduce your reports easily.

[THINK2TRY]

The hyper-parameters are totally the same for different implementations and are listed here. You can refer to this branch for PyG implementation and run the same scripts to reproduce the results.
As for the results with different data splits, the reason may be that GraphMAE pretrains the GNN in a self-supervised manner, and the labeled data is used only to train a linear classifier.