Text_Mining_Project

To reproduce the results of the paper you can do the following steps:

cd Algorithms/

Pre-processing and documents to vector format

./pipeline_preprocessing.sh

it will call the "pipeline" execution of preprocessing_tools.py to do the pre-processing of the raw texts and the "base" execution of matrix_tools.py to make the bow and tf-idf-l2 matrices.

Word embedding

Then we need to create the BERT embeddings of the words of each dataset. We recommend to install bert-as-service, to download a pre-trained model (we have chosen this one) and finally run a service with:

bert-serving-start -model_dir=PRE_TRAINED_BERT_MODEL_PATH -max_seq_len=NONE -verbose

to create the embeddings:

./embedding_bert_all.sh

Co-clustering

./pipeline_coclustering.sh

it will run the co-clustering algorithms (CoclustInfo, CoclustMod and CoclustSpecMod) on all the versions of each dataset with coclustering.py then keep the best version of each dataset (according the NMI, ARI and clustering accuracy) with result_manager_row.py and finally get the results of the columns clustering with coclustering_col.py.

Then you can call result_manager_col.py to print the result of the columns clustering.