Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-6517][mllib] Implement the Algorithm of Hierarchical Clustering #5267

Closed
wants to merge 77 commits into from
Closed

[SPARK-6517][mllib] Implement the Algorithm of Hierarchical Clustering #5267

wants to merge 77 commits into from

Commits on Oct 21, 2015

  1. [SPARK-6517][mllib] Implement the Algorithm of Hierarchical Clustering 

    Thank you for your great cooperation, RJ Nowling(@rnowling), Jeremy Freeman(@freeman-lab), Xiangrui Meng(@mengxr) and Sean Owen(@srowen).
    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    295bdde View commit details
    Browse the repository at this point in the history

  2. Fix the some comments

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    c51017c View commit details
    Browse the repository at this point in the history

  3. Remove parentheses for getters and add a test for HierarchicalCluster… 

    …ing.setNumClusters
    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    a8cd7ab View commit details
    Browse the repository at this point in the history

  4. Remove the static train() method from HierarchicalClustering object

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    306f603 View commit details
    Browse the repository at this point in the history

  5. Add a test for HierarchicalClustering.findClosestCenter()

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    b2d0369 View commit details
    Browse the repository at this point in the history

  6. Add a function to compute Within Set Sum of Squared Error into Scala/… 

    …Java/Python
    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    0ddfcfb View commit details
    Browse the repository at this point in the history

  7. Change the visibility of constructer parameters of HierarchicalCluste… 

    …ring and ClusterTree from public to private
    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    ecb3fd7 View commit details
    Browse the repository at this point in the history

  8. Rename getSubIterations to getMaxIterations

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    08f0101 View commit details
    Browse the repository at this point in the history

  9. Modify how to broadcast variables

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    2a14900 View commit details
    Browse the repository at this point in the history

  10. Remove unnecessary parentheses

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    38f07bd View commit details
    Browse the repository at this point in the history

  11. Add toLinkageMatrix() and toAdjacencyList()

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    99e703b View commit details
    Browse the repository at this point in the history

  12. Add a java test file for HierarchicalClustering

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    344d14e View commit details
    Browse the repository at this point in the history

  13. Support save and load functions in Java

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    e7256f5 View commit details
    Browse the repository at this point in the history

  14. Change the specification of HierarchicalClusteringModel.save()

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    e294795 View commit details
    Browse the repository at this point in the history

  15. Format code and modify the comments

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    1c66e09 View commit details
    Browse the repository at this point in the history

  16. Format the code because there is a long line

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    59480d3 View commit details
    Browse the repository at this point in the history

  17. Fix some comments for HierarchicalClustering in Scala

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    ec9f85f View commit details
    Browse the repository at this point in the history

  18. Sort import statements in HierarchicalClustering.scala and 

    HierarchicalClusteringModel.scala
    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    58999db View commit details
    Browse the repository at this point in the history

  19. Format HierarchicalClusteringSuite and HierarchicalClusteringModelSuite

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    a077e99 View commit details
    Browse the repository at this point in the history

  20. Rename ClusterTree to ClusterNode

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    fa74f20 View commit details
    Browse the repository at this point in the history

  21. Remove save/load from HierarchicalClusteringModel

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    1143920 View commit details
    Browse the repository at this point in the history

  22. Fix some mislenious code pointed out by IntelliJ

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    16cc823 View commit details
    Browse the repository at this point in the history

  23. Remove python API. We will implement it at another issue.

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    c02134e View commit details
    Browse the repository at this point in the history

  24. Rename HierarchicalClustering to BisectingKMeans

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    def81e2 View commit details
    Browse the repository at this point in the history

  25. Remove the unnecessary parentheses

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    707609a View commit details
    Browse the repository at this point in the history

  26. Change the way how to initialize the children centers

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    4e1653d View commit details
    Browse the repository at this point in the history

  27. Change the criterion for building a cluster tree from variance vector… 

    …s to avg. costs.
    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    6a51b12 View commit details
    Browse the repository at this point in the history

  28. Change toArray to avoid the TimSort error

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    c8a2a19 View commit details
    Browse the repository at this point in the history

  29. Format the code, since there are some validation problems

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    5f899b3 View commit details
    Browse the repository at this point in the history

  30. Remove unnecesary comment and import

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    313e87f View commit details
    Browse the repository at this point in the history

  31. Fix a typo and a few comments

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    3f6b14a View commit details
    Browse the repository at this point in the history

  32. Add a new line above spark project classes

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    52b4704 View commit details
    Browse the repository at this point in the history

  33. Arrange the order of import statements

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    fe87715 View commit details
    Browse the repository at this point in the history

  34. Use isDefined, instead of != None

    @yu-iskw
    yu-iskw committed Oct 21, 2015
    Configuration menu
    Copy the full SHA
    eeef1e7 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2015

  1. Update the bisecting k-means 

    - Add the description about the algorithm and the criterion for build a cluster tree
- Extract utility methods to `BisectingKMeans` object
- Make `BisectingKMeans` `private[clustering]`
- Add `@Since` tags to public methods, parameters and variables
    @yu-iskw
    yu-iskw committed Oct 22, 2015
    Configuration menu
    Copy the full SHA
    3156dd7 View commit details
    Browse the repository at this point in the history

  2. Improve a performance

    @yu-iskw
    yu-iskw committed Oct 22, 2015
    Configuration menu
    Copy the full SHA
    31623ea View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2015

  1. Remove toAdjacencyList and toLinkageList 

    These will be merge at following PR.
    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    052c9d6 View commit details
    Browse the repository at this point in the history

  2. Remove an unnecessary constructor arg: clusterMap

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    e13e47f View commit details
    Browse the repository at this point in the history

  3. Add import BisectingKMeans._ inside of BisectingKMeans class

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    ad6b9e2 View commit details
    Browse the repository at this point in the history

  4. Rename rddArray to updatedDataHistory in order to make the name m… 

    …ore descraptive
    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    043f5f3 View commit details
    Browse the repository at this point in the history

  5. Modify math.log10 to math.log

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    31b05ec View commit details
    Browse the repository at this point in the history

  6. Add getMinimumNumNodeInTree to calculate the minimum number of node… 

    … in a binary tree
    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    12a60cf View commit details
    Browse the repository at this point in the history

  7. Rename leafClusters to leafClusterStats

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    a13a404 View commit details
    Browse the repository at this point in the history

  8. Modify BisectingKMeans.updateClusterIndex

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    75564b5 View commit details
    Browse the repository at this point in the history

  9. Move checking whether there are dividable clusters or not to below

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    084b992 View commit details
    Browse the repository at this point in the history

  10. Make sure the input data keeps the storage level and unpersist unnece… 

    …ssary RDDs
    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    b6a952d View commit details
    Browse the repository at this point in the history

  11. Remove closestCenter from findClosestCenter because it was a unne… 

    …cessary variable
    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    1ba4e45 View commit details
    Browse the repository at this point in the history

  12. Modify summarizeClusters

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    cb4fbfe View commit details
    Browse the repository at this point in the history

  13. Modify a type

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    fbcb9ea View commit details
    Browse the repository at this point in the history

  14. Replace criterion with cost

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    ffbe399 View commit details
    Browse the repository at this point in the history

  15. Change the constructor args of BisectingClusterStats

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    6f37028 View commit details
    Browse the repository at this point in the history

  16. Convert Long to BigInt

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    010fd2c View commit details
    Browse the repository at this point in the history

  17. Modify BisectingKMeansModel.predict

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    e39f69a View commit details
    Browse the repository at this point in the history

  18. Organize import statements

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    2e8226d View commit details
    Browse the repository at this point in the history

  19. Modify the comment inside of updateClusterIndex

    @yu-iskw
    yu-iskw committed Oct 26, 2015
    Configuration menu
    Copy the full SHA
    622499e View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2015

  1. Change the default value of k to 2

    @yu-iskw
    yu-iskw committed Oct 27, 2015
    Configuration menu
    Copy the full SHA
    267908a View commit details
    Browse the repository at this point in the history

  2. Change 10E-4 to 10e-4

    @yu-iskw
    yu-iskw committed Oct 27, 2015
    Configuration menu
    Copy the full SHA
    a664f6d View commit details
    Browse the repository at this point in the history

  3. Remove toLinkageMatrix and toAdjacencyList from BisectingKMeans

    @yu-iskw
    yu-iskw committed Oct 27, 2015
    Configuration menu
    Copy the full SHA
    165e191 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2015

  1. Change 10e-4 to 1e-4

    @yu-iskw
    yu-iskw committed Oct 28, 2015
    Configuration menu
    Copy the full SHA
    c417dd1 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2015

  1. Fix a test for the default value of k

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    e2f6966 View commit details
    Browse the repository at this point in the history

  2. Replace mapPartition(...).reduceByKey(...) with aggregateByKey

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    69e0910 View commit details
    Browse the repository at this point in the history

  3. Change sumOfSquares vector to scholar at divideClusters

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    675bafb View commit details
    Browse the repository at this point in the history

  4. Replace a chain of mapPartition and reduceByKey with `aggregateBy… 

    …Key` at `divideClusters`
    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    704e145 View commit details
    Browse the repository at this point in the history

  5. Modify getMinimumNumNodesInTree with 1 << multiplier

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    ee3ea62 View commit details
    Browse the repository at this point in the history

  6. Rename numClusters parameter to k

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    73e2c7a View commit details
    Browse the repository at this point in the history

  7. Rename a variable in BisectingKMeansModelSuite

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    a876ba2 View commit details
    Browse the repository at this point in the history

  8. tmp

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    1985fea View commit details
    Browse the repository at this point in the history

  9. Make this implementation more simple

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    629f897 View commit details
    Browse the repository at this point in the history

  10. Reorganize import statements and adjust parameters and return values

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    1f84ded View commit details
    Browse the repository at this point in the history

  11. Rename WSSSE to computeCost

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    12b3223 View commit details
    Browse the repository at this point in the history

  12. Remove updateClusterIndex

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    ef4a3e8 View commit details
    Browse the repository at this point in the history

  13. Remove BisectingClusterStat object

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    57b06ba View commit details
    Browse the repository at this point in the history

  14. Fix minors

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    5da05d3 View commit details
    Browse the repository at this point in the history

  15. Improve initNextCenters

    @yu-iskw
    yu-iskw committed Oct 29, 2015
    Configuration menu
    Copy the full SHA
    a50689a View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2015

  1. refactor

    @mengxr
    mengxr committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    d422be7 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #4 from mengxr/SPARK-6517 

    Refactor bisecting k-means
    @yu-iskw
    yu-iskw committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    75ca2a0 View commit details
    Browse the repository at this point in the history

  3. Remove a magic number 63 for level limitation

    @yu-iskw
    yu-iskw committed Nov 9, 2015
    Configuration menu
    Copy the full SHA
    29ccdf9 View commit details
    Browse the repository at this point in the history