Problems of results of LightGCN

[ShadowTinker]

Hi, I've run LightGCN in QRec recently. But I find that the performance of it is not good. The results I get on Yelp are Precision:0.0252', 'Recall:0.0575', 'F1:0.0350', 'NDCG:0.0465 after 1000 epochs. While the results of SEPT on the same dataset are 'Precision:0.0310', 'Recall:0.0712', 'F1:0.0432', 'NDCG:0.0583'] after 30 epochs. And I also set if epoch > self.maxEpoch / 3: to if epoch > self.maxEpoch: in SEPT, which means SEPT will behave like LightGCN. But the results, which are ['Precision:0.0289', 'Recall:0.0641', 'F1:0.0399', 'NDCG:0.0532'] after 30 epochs, are also much better than the original one. BTW, the initial rec_loss of the degraded SEPT is much lower than the original LightGCN, ~300/batch vs ~1000/batch.
So I want to know how to make the original LightGCN better. I've already tried the same normalization and initiliazation as SEPT, but it doesn't work.

[Coder-Yu]

For SEPT, the best results are reported because after each epoch, the immediate results are recored.

QRec/model/ranking/SEPT.py

Line 312 in 2483ad0

self.ranking_performance(epoch)

For LightGCN, you can use the same way to get the best performance. According to my experience, LightGCN needs hundreds of epochs to get converged. The way to speedup the training is applying L2 normalization at each layer.

The results reported in the paper were based on python 2. We later upgraded QRec and transplated it to python3. So, there would be some trivial differences on performance. We conducted 5-fold cross-validation. Not sure if you applied the same experimental setting. BTW, you can try our pytorch implementation of LightGCN at https://github.com/Coder-Yu/SELFRec/blob/main/model/graph/LightGCN.py

[ShadowTinker]

Thank you for your reply.

I just find changing reduce_mean to reduce_sum in LightGCN as SEPT can make the original LightGCN behave similarly as SEPT in the early training phase. I think this may be the reason.

QRec/model/ranking/LightGCN.py

Line 19 in 2483ad0

all_embeddings = tf.reduce_mean(all_embeddings, axis=0)

Thank you for your time and great repo!

[Coder-Yu]

"changing reduce_mean to reduce_sum in LightGCN as SEPT can make the original LightGCN behave similarly as SEPT in the early training phase"

Exactly. I forgot to pinpoint this cause. I wish you a good luck on your study.

[ShadowTinker]

Thanks a lot!