Implemented Ng et al.'s dissimilarity measure #44

benandow · 2017-06-07T01:21:42Z

Implemented Ng et al.'s dissimilarity measure[1] including necessary modifications to the base libraries. Feel free to pull into the main branch.

[1] Michael K. Ng, Mark Junjie Li, Joshua Zhexue Huang, and Zengyou He, "On the Impact of Dissimilarity Measure in k-Modes Clustering Algorithm", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 3, January, 2007

coveralls · 2017-06-07T01:59:54Z

Coverage decreased (-1.6%) to 95.134% when pulling 9785a5b on benandow:master into a30da80 on nicodv:master.

nicodv · 2017-06-07T04:20:57Z

This is awesome, thank you!

I have 2 remarks, though:

Do we need separate dissimilarity measures between the initialization and the main loop of the algorithm? I think it makes the API less clear, and I don't see the added value.
The new dissimilarity measure needs to be properly unit tested before we can merge this.

benandow · 2017-06-07T16:01:26Z

Not a problem! It's a very useful library. Thanks for releasing it!

To address your remarks:

That's a fair point. I'll revert it to use the same dissimilarity measure for the initialization and main loop. The main reason why I implemented it with different parameters for dissimilarity measures is that it is more efficient to just invoke the simple matching_diss measure for initialization, as Ng. et al.'s measure inherently produces the same result due to the clusters being empty at that point.
- Explanation: The benefit of Ng et al.'s dissimilarity measure is that it essentially accounts for cluster impurity by penalizing attribute mismatches within a cluster. When the attribute value of the current record x_i,j matches the corresponding attribute value of the cluster centroid z_l,j, you look at the current members of the cluster and calculate phi(z_l,j, x_i,j) as 1 - |C_l,j,r| / |C_l}|, where |C_l| is the number of objects in the l^th cluster, and |C_l,j,r| is the number of objects in the l^th cluster whose j^th attribute has the same attribute value of the current record x_i,j. If the cluster is empty and the attribute value of the centroid matches the attribute value of the record (i.e., x_i,j == z_l,j), then there should be no penalty (i.e., 1 - |C_l,j,r| / |C_l}| = 0, as both |C_l,j,r| and |C_l}| equals 0). If x_i,j != z_l,j, then the penalty is 1. Therefore, when the clusters are empty, such as during centroid initialization, Ng. et al.'s dissimilarity measure produces the same result as the simple matching_diss measure.
I'll try to find some free time to include unit tests.

Best,
-Ben

…diss, and added error checking to dissimilarity function for membship array

coveralls · 2017-06-07T20:41:38Z

Coverage decreased (-0.1%) to 96.602% when pulling 5c08222 on benandow:master into a30da80 on nicodv:master.

coveralls · 2017-06-07T20:41:38Z

Coverage decreased (-0.1%) to 96.602% when pulling 5c08222 on benandow:master into a30da80 on nicodv:master.

coveralls · 2017-06-07T20:41:38Z

Coverage decreased (-0.1%) to 96.602% when pulling 5c08222 on benandow:master into a30da80 on nicodv:master.

benandow · 2017-06-07T20:48:21Z

As discussed above, I've reverted k-modes and k-prototypes to use the same dissimilarity measure for initialization and the main loop. I added unit tests for the Ng dissimilarity function, and also error checking in the Ng dissimilarity measure to ensure that the 'membship' array has the correct shape.

nicodv · 2017-06-09T05:12:48Z

kmodes/util/dissim.py

+    xcids = np.where(np.in1d(memj.ravel(), [1]).reshape(memj.shape))
+    return float((np.take(X, xcids, axis=0)[0][:, idr] == b[idr]).sum(0))
+
+def calc_dissim(b, X, memj, idr, idj):


idj is not used in the function at all?

Good catch! I removed that parameter. I had implemented support for a weighting function, which was removed for the PR and that parameter was a leftover remnant.

…ant from weighting function)

nicodv · 2017-06-09T05:26:24Z

kmodes/kmodes.py

@@ -373,6 +376,9 @@ def fit_predict(self, X, y=None, **kwargs):
        """
        return self.fit(X, **kwargs).labels_

+    def genMembshipArray(self):


Let's make this a function in utils/init.py

Moved it and updated the references.

coveralls · 2017-06-09T05:26:27Z

Coverage decreased (-0.1%) to 96.602% when pulling 7e4e865 on benandow:master into a30da80 on nicodv:master.

nicodv · 2017-06-09T05:26:46Z

kmodes/kprototypes.py

@@ -415,6 +416,9 @@ def fit(self, X, y=None, categorical=None):
                                                    self.verbose)
        return self

+    def genMembshipArray(self):


Let's make this a function in utils/init.py

Moved it and updated the references.

nicodv · 2017-06-09T05:35:19Z

@benandow , in test_kmodes and test_kprototypes, we need to add tests that use this dissimilarity measure to prove that the end-to-end flow works, and that the results are as expected.

coveralls · 2017-06-09T06:00:03Z

Coverage decreased (-0.1%) to 96.585% when pulling 1d9526a on benandow:master into a30da80 on nicodv:master.

…ck of initial feature vector for attribute frequency calculation in ng_diss. Although this means that the encoded FV will not be freed by the GC! Alternative is to build a map of attribute frequencies at the beginning, but this adds unnecessary memory overhead if not predicting new data (e.g., just invoking fit or fit_predict). Therefore, I believe that maintaining a reference to the encoded FV is the best choice.

…ure and results are as expected.

benandow · 2017-06-09T15:39:24Z

@nicodv I added the test cases in test_kmodes and test_kprototypes to show that the end-to-end flow works and the results are expected when using the Ng dissimilarity measure.

After testing, I also submitted a bug fix that occurred when invoking fit and then trying to predict new data. I overlooked that we need to keep track of initial encoded feature vector for the attribute frequency calculation in Ng's dissimilarity measure. I implemented the patch. However, this means that the encoded FV will not be freed by the GC. An alternative solution is to build a map of attribute frequencies at the beginning, so that we can allow the encoded FV to be freed, but this adds unnecessary memory overhead if not predicting new data (e.g., if you're just invoking fit or fit_predict). Therefore, I believe that maintaining a reference to the encoded FV is the best choice.

coveralls · 2017-06-09T15:42:34Z