-
Notifications
You must be signed in to change notification settings - Fork 0
Home
-
Directories to analyse: this selection is mandatory and must have at least one element otherwise; an error message will pop up. Here the directories selected will act as root directories, gathering all documents of accepted mime type within their tree hierarchy.
-
Exclusion: this allows selecting as many directories the user wishes to exclude from the search. Exclusion of one directory will also exclude not only its content but also its whole structure of subfolders. This selection is optional.
Each has a panel below showing all the current directories selected, with a Del button to remove said selection.
Immediately below, on the left a selection comprising of two radio buttons can be used to select a stop words file containing words used excluded during the tokenization process, for example for being too common in the language and purify the text. The “default” selection is already referring to a file included in the project, but the stop words can come from a custom file, which can be selected by clicking on the other selection. This will open a file choosing box, which will allow selecting a file to be used as stop words. If no selection was made, the “default” option will be marked again.
On the opposite side, there is a text field that allow defining a custom regex to use during the tokenisation process. If left blank, it will use a default regex which will split the text using a 1-gram model (single words).
Drop-down menu used for choosing the type of clustering one desires between two options: Flat and Hierarchical. Choosing the Flat option, the clustering will make use of a proportional algorithm to generate a sparse viewing of the documents in a flat space. This choice will make appear the “Number of Clusters” selection bar, in place of the default “Merging criteria”, as long as the Flat option is being selected. Selecting flat will switch the view to show the indicated input box. In addition, when the option is selected, cosine sim in Scoring Metric and no dictionary in Features Modelling are not available.
Choosing the Hierarchical option will allow to obtain a tree structure for the resulting clusters. This selection will also make appear a “Merging criteria” selection bar below, instead of “Number of cluster”, to choose what link will be the criteria for merging the clusters.
Select a weighting scheme. Selecting Raw-Bag-Of-Words will preclude to select any option in the selection bar on its left, except “Normal” and “No Dictionary”.
The selection on the left allows choosing a Document Frequency counterweight. Normal is the equivalent of an empty selection, while No Dictionary will prevent the creation of a dictionary with all the consequences; it is advised to use only on specific cases.
This selection can be found at the rightmost side of the GUI. Distance/Similarity metrics are the criteria for what each document will be compared to each other for similarity of their content. These methods will allow the creation of a Vector Space Model to calculate the score of comparison between every pairing of vectors.
This selection bar will be appearing only while the clustering type is hierarchical. As mentioned before, this choice will determine how the clusters will merge. This too can heavily affect the result.
A text field will appear in the middle of the GUI when Flat option is selected in the clustering selection. This area accepts only digits and is dedicated to set the number of clusters desired when performing a partitioning clustering algorithm. This field needs to be filled, to avoid the default option of one singleton cluster being used. Typing in zero will also trigger the algorithm to use the option of a single cluster (see K-means).
View Result will show the graphical representation of the clusters after the process is completed. It shows something different than a blank window only when at least one process has run since the opening of the application. It keeps the output in memory (so it can be reopened anytime) until another analysis has run and has completed, substituting the old output. Error logs will open to DC_logs in HOME (e.g.: /Desktop in Windows OSs) directory in the local system. It contains the logs produced from errors in the program. Start will kick off a new process based on the selections in the GUI set in that moment. Stop is currently unavailable and clicking it will open an error message.
Plotting the hierarchical clustering will produce a Dendrogram, a type of tree diagram where each converging leaf represent a document (vector) and nodes represent the merge of two elements (either documents or clusters). All clusters and documents keep merging until all elements belong to one cluster. A horizontal line represents each merge. The y-coordinate of the horizontal line is the similarity of the two clusters that were merged, where documents are viewed as singleton clusters. We call this similarity the of the merged cluster combination similarity. (Christopher D. Manning, Raghavan, & Schütze, 2009)
Each purple node is a point where a merging occurred. When clicking on those nodes a new window appears which focus on the branch in the Dendrogram starting from that point down to the documents.
Once entered this new window, if the mouse exits the window disappears. The grey nodes at the ending points are the document vectors themselves. Clicking those nodes will make appear a small window:
- The title is the name of the vector and if clicked will open the original document, unless it was moved from the indicated position.
- View Vector shows the features of the vector: the terms and their weight. It cannot collapse.
- Main Topic will visualise as a popup the highest-ranking term in the vector. The box cannot be resized, but it can be dragged (in only the right half of the window though). In addition, a button will be available on the top-right corner of the window:
The “Generate file system” button when click will open a dialog that will allow choosing a directory in the user’s file system.
If a selection is made, the program will proceed to generate a new file structure from the resultant clusters. The merges will become folder and subfolders. The documents will be copied in the new structure accordingly to how they were merged in the Dendrogram. As such, a new file system is generated reproducing the result of the hierarchical clustering. A message will show when the process is completed. In addition, a message will appear if the process could not finish doing to some critical error. The program should not be close during this process.
The Generate file system button, appearing always in the same position, may cover some part of the draw. It is, though, possible to drag the button in a more convenient position.
TO DO: Future Graphical Implementation At the current state, no dense/sparse plot is produces due to a lack of time and for not being a top priority in the current iteration of the project. The current graphic will show a window containing a list of names clusters, each listing the path of the documents that form said cluster.
Future Improvement
Due to the original didactical and academical nature of the project, custom algorithms and implementations were made. However, future implementations will include the use of third party machine learning libraries (e.g. spark) adpated to the project in question. Also it will require improvements on the parallel processes taking place and overall perfomance. The current way has been achieved with the best available in the standard library and carefully measures. However streaming library such fs2 are under consideration as future works. Also there is room for improvements on the data types and overall code base.
For more information and to require the complete reading for this project please mail federicocdev@gmail.com.