CFTest takes an hour in debug mode

[rcurtin]

Reported by rcurtin on 29 Sep 44900790 20:36 UTC
According to Jenkins, CFTest takes nearly an hour in debug mode and half an hour in non-debug mode:

http://big.cc.gt.atl.ga.us/job/mlpack%20-%20nightly%20matrix%20build/arch=linux-amd64,armadillo=armadillo-svn,buildmode=debug/lastCompletedBuild/testReport/MLPACKTest/CFTest/

http://big.cc.gt.atl.ga.us/job/mlpack%20-%20nightly%20matrix%20build/arch=linux-amd64,armadillo=armadillo-svn,buildmode=release/lastCompletedBuild/testReport/MLPACKTest/CFTest/

(or if those links don't work, you can look at the nightly matrix build, select a configuration, then click on the test result and narrow down to CFTest, and it should show how long it takes)

I think that testing is quite important so we should be certain that the test is comprehensive, but I suspect that we can significantly cut down the runtime of that particular test suite (with each test taking 10 or 20 minutes).

A first guess would be that the tolerance of the NMF ALS factorizer is too tight, or simply that NMF-ALS is slow. So, what should happen is this ticket is that we should investigate the cause of the slowdown (either using the Timer class and displaying the output) or with gprof, then determine if the real issue is a bottleneck in NMFALSFactorizer, or if the convergence tolerance is needlessly tight, or if NMFALSFactorizer is even the best choice for these tests and as the default CF factorizer.

Siddharth and Sumedh, I've CC'ed you guys because you were significantly involved with the CF module development in the past and might have some insight here.

Migrated-From: http://trac.research.cc.gatech.edu/fastlab/ticket/372

[rcurtin]

Commented by siddharth.950 on 13 Apr 44901861 07:53 UTC
Ryan, I cannot comment on the working of NMF-ALS factorizer, as I have no idea how it works.

Replacing the default factorizer with RegSVD might be a good idea, since it is a standard method for CF and from what I remember works quite fast with the GroupLens dataset.

[rcurtin]

On my system I can accelerate the tests to 3 seconds without debugging symbols and 20 seconds with, by loosening the default convergence criterion for the SimpleResidueTermination class; committed in 658962f. This should make the test sufficiently fast on the Jenkins slaves. But I guess big.cc.gt.atl.ga.us will have to come back up before we find out... either way, the change should solve this issue.