Stopping criterion of ALS

[stegben]

Is it a good idea to stop ALS by validation dataset based on some criteria(RMSE, etc.)? The paper use probe datasets as validation-set. Once the RMSE is less than 1e-9, they stop the iteration.

[benfred]

It usually doesn't hurt to run more iterations - for instance this post run this code and calculated p@5 out at each iteration and found that it converged around 15 iterations or so (which is the default here).

I'm going to add some code for evaluation here (calculating training loss, and maybe MAP) at some point, but I don't think that using that for early termination is worthwhile - it can be slow to calculate things like MAP or P@K because we have to sort the predictions : while the training itself can leverage sparsity of the dataset, sorting means there is a cost for missing items in the validation phase. (the paper you linked to is on the explicit case where you can safely ignore missing items).

[benfred]

I added code to calculate the loss at each iteration here da1a7fa
I compared this per iteration in this post: http://benfrederickson.com/fast-implicit-matrix-factorization/ and showed that the training loss per iteration converges after 15 or so iterations. Closing this for now, since I don't think early stopping will make this any faster or more accurate