StreaKHC is a novel incremental hierarchical clustering algorithm for efficiently mining massive streaming data. It uses a scalable point-set kernel to measurethe similarity between an existing cluster in the cluster tree and a new point in a stream. It also has an efficient hierarchical structure updating mechanism to continuously maintain a high-quality cluster tree in real-time. Technical details and analysis of the algorithm can be found in paper.
Download and Install Anaconda's Python3
https://docs.continuum.io/anaconda/install
Install numba
conda install numba
Set environment variables:
source bin/setup.sh
If want to visulize the build tree, install Graphviz
sudo apt install graphviz
Run test on data set:
./bin/run_grid_evaluation.sh
The evaluation result is shown in /exp_out/ default. For each of the randomly shuffled data of a specified data set, the dengrogram purity result and figure of built tree is shown in score.tsv and tree.png, respectively.
- If do not need to visualize the generated tree, you can comment out the corresponding code in the /bin/run_evaluation.sh.
- Perl is used to shuffle the data.You'll need perl installed on your system to run experiment shell scripts. If you can't run perl, you can change this to another shuffling method of your choice.
- The scripts in this project use environment variables set in the setup script. You'll need to source this set up script in each shell session running this project.
- Most of the program running time is used to calculate dendrogram purity.
If you have used this codebase in a scientific publication and wish to cite it, please use the following publication (Bibtex format):
@inproceedings{HZTZL22Streaming,
author = {Han, Xin and Zhu, Ye and Ting, Kai Ming and Zhan, De-Chuan and Li, Gang},
title = {Streaming Hierarchical Clustering Based on Point-Set Kernel},
year = {2022},
isbn = {9781450393850},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3534678.3539323},
doi = {10.1145/3534678.3539323},
pages = {525–533},
numpages = {9},
keywords = {streaming data, hierarchical clustering, isolation kernel},
location = {Washington DC, USA},
series = {KDD '22}
}
Apache License, Version 2.0