Add min threshold to textstat_dist()?

[koheiw]

[kbenoit]

Not exactly a clear feature request, but if you mean an option to return a reduced size dist object based on the dist values, this is not really workable since the threshold would differ wildly across the matrix.

But maybe I've misunderstood. Are there examples of this for other dist operations? (from proxy for instance?)

[koheiw]

I did not have time to explain until now. I noticed that people are struggling to calculate pairwise similarity between large number of sentences or n-grams to detect reuse of texts because it generates large dense matrix.

We can make it more efficient if we floor values lower than a threshold to zero. Flooring does not have any impact on the analysis because people are usually only interested in very high similarity. I haven't seen the code really, but could be done either in textstat_simil or textstat_dist.

[koheiw]

This is where flooring will should happen:

quanteda/src/dist_mt.cpp

Line 218 in 76e2cbf

rmat(j,i) = (double)sum(bb) * (double)ncol/(double)non_zeros.n_elem;

[kbenoit]

OK, I see. So in the row/column loop in C++ we would just return a zero if the distance were below the minimum, and that would be one less cell in the sparse object returned by the C++ functions, so it is a smaller object returned to R.

The current return to R is a dist object, which is an efficient lower-diagonal vector of a distance matrix, basically a vector with a separate record of the object dimensions. But a zero in that object would still be a zero numeric, so we would have to change the return from textstat_dist() object to be something like that the fcm class is currently. Would also need an as.dist() coercion method for that.

The question is whether this is really needed. Is the number of zero cells caused by a removing distances below a minimum significant enough to warrant this solution? Current object format for an fcm is a sparse lower diagonal matrix. Current object format for dist is the same, except that it records small values too. What would the "sparsity" level of a typical dist return?

Finally, are there precedents for doing this with distance computations? Something whose practice we could emulate?

[koheiw]

Nicolas Merz is facing this issue right now, so we can ask him to test if flooring helps. I also have a plant to perform huge pairwise distance calculation in near future for "fake news". Analysis of text reuse has been restricted due to the lack of an efficient tool, but the new argument might create a new trend in text analysis.

wordspace::dist.matrix returns a matrix, but can be coerced to a dist object.
https://www.rdocumentation.org/packages/wordspace/versions/0.2-0/topics/dist.matrix

spam::nearestdist has delta as a threashold.
https://www.rdocumentation.org/packages/spam/versions/2.1-1/topics/nearestdist

It seems that not many functions that take sparse matrix as an input return dense dist as an output.

[koheiw]

After being asked about how to large-N pair-wise similarity computation, I am keen on adding dist_floor or something like that. However, I noticed that the most popular distance measures (cosine and correlation) are calculated by Matrix::crossprod(). So we need to write its equivalent in parallel C++ with floor argument.

quanteda/R/textstat_simil.R

Lines 122 to 164 in 57a3921

	cosine_simil <- function(x, y = NULL, margin = 1) {
	if (!(margin %in% 1:2)) stop("margin can only be 1 (rows) or 2 (columns)")
	if (margin == 1) x <- t(x)
	S <- rep(1, nrow(x))
	N <- Matrix::Diagonal(x = sqrt(colSums(x ^ 2)) ^ -1)
	x <- x %*% N
	if (!is.null(y)) {
	if (margin == 1) y <- t(y)
	N <- Matrix::Diagonal(x = sqrt(colSums(y ^ 2)) ^ -1)
	y <- y %*% N
	return(as.matrix(Matrix::crossprod(x, y)))
	} else
	return(as.matrix(Matrix::crossprod(x)))
	}
	# Pearson correlation
	correlation_simil <- function(x, y = NULL, margin = 1) {
	if (!(margin %in% 1:2)) stop("margin can only be 1 (rows) or 2 (columns)")
	func_cp <- if (margin == 2) Matrix::crossprod else Matrix::tcrossprod
	func_tcp <- if (margin == 2) Matrix::tcrossprod else Matrix::crossprod
	func_sum <- if (margin == 2) colSums else rowSums
	n <- if (margin == 2) nrow(x) else ncol(x)
	mux <- if (margin == 2) colMeans(x) else rowMeans(x)
	if (!is.null(y)) {
	stopifnot(if (margin == 2) nrow(x) == nrow(y) else ncol(x) == ncol(y))
	muy <- if (margin == 2) colMeans(y) else rowMeans(y)
	covmat <- (as.matrix(func_cp(x,y)) - n * tcrossprod(mux, muy)) / (n - 1)
	sdvecX <- sqrt((func_sum(x ^ 2) - n * mux ^ 2) / (n - 1))
	sdvecY <- sqrt((func_sum(y ^ 2) - n * muy ^ 2) / (n - 1))
	cormat <- covmat / tcrossprod(sdvecX, sdvecY)
	} else {
	covmat <- (as.matrix(func_cp(x)) - drop(n * tcrossprod(mux))) / (n - 1)
	sdvec <- sqrt(diag(covmat))
	cormat <- covmat / tcrossprod(sdvec)
	}
	cormat
	}

[kbenoit]

Let's look at some other packages to see how they deal with this issue, e.g. scipy - see http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.pairwise_distances.html, or http://na-o-ys.github.io/others/2015-11-07-sparse-vector-similarities.html.

[koheiw]

I saw these but there is no indication that they have flooring option or something similar. This is why many people who are in trouble.

[kbenoit]

Understood, but I'd like to see some other approach that implements the floor, or a paper justifying it, so that we fully understand the substantive consequences of zeroing small values on all of the metrics that we implement. This could be a test suite for instance. But it's possible that not all measures are indifferent to near-zero values being treated as zero.

The R implementation formerly known as Revolution Analytics uses a parallelized and optimised version of LAPACK for speed, and it has a better crossprod implementation, I believe. We cannot use that directly but we can learn from the source. See http://blog.revolutionanalytics.com/2014/10/revolution-r-open-mkl.html.

[koheiw]

This guy takes top-n for similar purpose:
https://medium.com/@ingwbaa/https-medium-com-ingwbaa-boosting-selection-of-the-most-similar-entities-in-large-scale-datasets-450b3242e618