Quantum Gaussian Mixture Models
Gaussian Mixture Model (GMM) is widely used in computer vision as a state-of-the-art clustering algorithm. This project proposes Quantum Gaussian Mixture Model (QGMM) for Quantum Clustering and it is originally proposed in the paper. In this project, we implemented QGMM and conducted some experiments to see if how fast it trains, how better it models the data, what edge cases there are, and there is anything we can improve.
We conducted researches to find out the strengths and weaknesses QGMM has.
According to the original paper, the probability equation of QGMM is that
where is normalized
In addition, has an effect on the mixture case and QGMM and GMM are the same when
. Therefore, we checked its interference phenomena by visualizing it in 3D plotting.
From the above images, we can check the interference phenomena as changed. In addition, we can see when
, QGMM is the same with GMM.
In the original paper, the objective function is that
In addition, the objective function means the expectation of the complete-data log likelihood, and we'll call it as log likelihood in this report.
However, in the probability equation, because is unnormalized, we can't guarantee
. Thus, we newly defined the objective function as an indicator of the training status.
Because is an unnormalized Gaussian in QGMM, we defined the new objective function like Lagrangian multiplier for constraint optimization. Therefore, the new objective function is Negative Log Likelihood (NLL) +
* Approximation Constraint and using an optimizer, we'll minimize it. With the objective function, we conduct several experiments to check if it works properly.
From the above images, we can see the training works properly except for the right one (In the next research, we'll dig into the failed case).
From the validity of the objective function research, we figured out it works properly. In addition, the higher value means the optimization is more constrained. Therefore, in this research, we checked the impact of . Generally, the initial
can be calculated by NLL / approximation constraint from the objective function, but when the initial
are almost zero, we can't calculate NLL. Therefore, we set the initial value of
manually.
The above images are the training process and the graph of the constraint. The left is with 100 and the right is with
1,000. From that, we found out that with
100, the constraint was unstable and there are some cases in which the training works only when using the more-constrained optimization. However, the high
is not always desirable because we also found out that the too high
rather interferes with the convergence of the objective function.
According to the original paper, can be calculated from that
However, when the initial are almost zero, the
is too large, exceeding the bound,
, and it results in the unstable training process. Therefore, we changed it to a trainable variable and the results in this final document were made after changing it. As the original paper mentioned, the
difference is calculated from
Thus, we checked the training results with the different initial values of .
In the above images, the left, center, and right are with the initial values of 0 (45 - 45), 90 (45 - (-45)), and 180 (90 - (-90)) respectively. When we set the initial
as 0, the values weren't changed, whereas in the cases of
90 and 180, they were changed. From some experiments, we found out that the two distributions get father as
is positive, while they get closer as
is negative.
Using mlpack's GMM class, we generated the data set for the mixed clusters to see if how QGMM works. To generate the mixture, we drew a circle between the two clusters and generated observations randomly.
Using the above data sets, we trained QGMM and GMM. Especially, we investigated two cases for QGMM with the initial 0 and 90 to check the impact of the initial
.
From the above results, we found out the results between QGMM and GMM are totally different. Furthermore, even between QGMMs, the results vary depending on .
In this research, we compared QGMM with GMM. As the indicator of the training performance, we use the percentage of the convergence on the clusters of the observations. We conducted 100 experiments with different initial means and the initial means were randomly generated between -1 and 1 from the maximum and minimum of x coordinates of the data set, and between -10 and 10 from the maximum and minimum of y coordinates of the data set. Besides, we used the augmented Lagrangian multiplier method for constrained optimization. Like other researches, we didn't use any initial clustering like K-means.
From the table above, there are 4 and 20 failed cases for QGMM with initial 0 and 90 respectively and 6 failed cases for GMM. Especially, among failed cases, there is a case that the training doesn't work, so we ran it again with other hyperparameters.
In the images above, the left is one of the failed cases of QGMM with initial 0, and the right is followed up by changing its hyperparameter.
We conducted another 100 experiments. In this time, the initial means were randomly generated between -0.5 and 0.5 from the maximum and minimum of x coordinates of the data set, and between -5 and 5 from the maximum and minimum of y coordinates of the data set.
From the above table, we can see the performance of QGMM with initial 0 increased a bit. We also checked some cases by changing
.
The above images are the case that the two distributions are overlaid initially. The left is with initial 0, and the right is with initial
180. From the images, we can see that in the left image, the two distributions moved to the same cluster first before moving to each cluster, on the other hand, the two distributions moved to each cluster. Therefore, we checked again that
has an effect on the training process.
In this research, we checked the performance of QGMM in multiple clusters.
In the images above, the left is with the sequence of , [0, 0, 0, 0, 0] and the right is with the sequence of
, [45, -45, 45, -45, 45]. For the multiple clusters cases, it's tricky to set the initial sequence of
to model the data properly.
In this project, we found some errors in the original QGMM, tried to correct them, and made some improvements in its performance while we looked into the property of it through the various trials. Before implementing QGMM, we simply visualized and checked the 3D probability space of QGMM to investigate its impact and come up with methods to normalize the probability to make the integral of it one.
Also, we found an error in the derivation of the covariance in the original approach, so we newly defined the objective function with the approximation constraint for the probability normalization, and checked it works properly.
While looking into the training states, we found the value of the cosine of is too large, so we changed
as a trainable variable. As a result, training performance became stable than before.
As we saw in the comparison with GMM research, QGMM showed flexible performance by adjusting the hyperparameters. In other words, we should set the proper hyperparameters to model the data correctly, but sometimes it would be not easy to do. Furthermore, from some several experiments, we found out that has a significant effect on the training process. In particular, it's hard to set the initial sequence of
when more than 3 clusters cases. Therefore, the current QGMM needs to come up with how to control
to generalize its performance.
- Convert DiagonalGMMs into GMMs in
mlpack_gmm_train - Make
GradientTransposedConvolutionLayerTestrobust. - Synchronize parameter's name with @param in
distsandgmmdirectory. - Resolve gcc parameter reorder warning for clean build.
- Edited to comply with the Style Checks
- Implement wrapper for diagonally-constrained GMM HMMs.
- Changed the file name
acrobat.hpptoacrobot.hpp. - Edited mountain_car.hpp to bound to goal position.
- Fix/Change Windows installer
- Fixed DBSCAN isn't using PointSelectionPolicy issue
Massive thanks Sumedh. He gave me great support and guidance for the project. Whenever I got stuck with problems, he presented possible solutions with enough description. While doing this project with him, I got many impressions from his inventive ideas and insightful approaches to the researches and learned a lot from him. Lastly, there are many proficient developers and researchers in mlpack community. It's my pleasure to contribute to this great machine learning library and I'll continue to contribute to mlpack actively. Thanks again Sumedh, and everyone. :)