Implementations of shilling detection algorithm in collaborative filtering, based on breeze, Mahout and Spark respectively
Algorithm adapted in this project comes from paper Mehta. 2009, the main idea is that using Principal Component Analysis to work on the user-rating matrix, and pick out users which have low covariance which are marked as possible spam users.
In general, we use SVD to calculate the main component of the user-rating matrix, which equals to PCA.
As scala, mahout and breeze library may not be included in the runtime, recommended built
mvn assembly:assembly to generate assembly jar package.
PCA with mahout:
GenerateDataclass on ml-10M data to generate spam user data locally, and upload to HDFS for next step's input.
PCAMahoutclass to execute PCA in MapReduce and get possible spam users.