Can't reproduce reported results

[ruoshiliu]

Hi, I ran the exact argument you provided in the README.md file
python run.py --data_dir=./data --num_epochs=30 --log_dir=./logs --batch_size=1024 --num_layers=2 --cell_type=hyp_gru
which yields a loss of 0.678 and precision of ~56%
I also ran
python run.py --data_dir=./data --num_epochs=30 --log_dir=./hyp_gru_hyp_decision --batch_size=1024 --num_layers=2 --cell_type=hyp_gru --decision_type hyp
which yields a loss of 0.693 and precision of ~50%
Do you know any potential reason for that?

[ferrine]

hmm, that is strange. We had updates in Geoopt and implementation of Poincare model is different now, this might affect the results. Did you check the old version of geoopt (before Stereographic is merged)?

[ruoshiliu]

Yes, I installed version 0.1.2 of geeopt which is before the Stereographic update.

[ferrine]

That's a problem smth really seems to be broken. How does the training curve look like?

[ruoshiliu]

Please check the following tensorboard results, with orange being train curves and blue being valid curves. I obtained the following results with python run.py --data_dir=./data --num_epochs=30 --log_dir=./logs --batch_size=1024 --num_layers=2 --cell_type=hyp_gru

[SkyFishMoon]

It occurs also in the new version. But I found that if decode using method in another paper named Hyperbolic GCN, the precision will arise a lot, but still can't get the result in original paper

[ferrine]

From the curves I can't say the model has converged, did you try training more?

[ferrine]

Other suggestion is to increase learning rate a bit.

[SkyFishMoon]

This is the validation

and train result

a little rise in precision, but I didn't use the dist2plane as the last layer. I just use a linear + logmap0 + logsoftmax + nllloss as the last layer.
Another problem is I find that some parameters in your model are general torch.Parameter rather than geoopt.ManifoldParameter. Only the bias parameters are geoopt.ManifoldParameter. Does this matter?