Reproducing "binge" result

[nonamestreet]

How to get similar result to https://github.com/maciejkula/binge as in your paper?

from spotlight.cross_validation import random_train_test_split
from spotlight.datasets.movielens import get_movielens_dataset
from spotlight.evaluation import mrr_score
from spotlight.factorization.implicit import ImplicitFactorizationModel
from spotlight.factorization.explicit import ExplicitFactorizationModel
from torch import optim
dataset = get_movielens_dataset(variant='1M')

train, test = random_train_test_split(dataset,test_percentage=0.2)
print train
print test

model = ImplicitFactorizationModel(n_iter=12,embedding_dim=32,batch_size=2048,
                                learning_rate=0.001,l2=1e-6,loss='adaptive_hinge',use_cuda=True)
model.fit(train,verbose=True)
mrr = mrr_score(model, test)
print mrr.mean()

The result is around 0.035 which is far from 0.07 in the paper. I was using the same hyper-parameters as in "movielens_1M_validation.log" in your binge repository, what am I missing?

[maciejkula]

Off the top of my head, it could be that you need to pass the train argument into MRR score to exclude known interactions from ranking. However, I don't think this would account for the entirety of the difference.

I will have a look at reproducing this and get back to you in the next couple of days.

[nonamestreet]

Thank your for the reply!

I just tried to put the train argument to MRR score, the performance is now at the 0.68-0.69 range which is quite close to the paper's result. But there is still a small gap left as you said.