Implement wrapper for diagonally-constrained GMM HMMs

[rcurtin]

This is a follow-up to issue #1499. Specifically, there I wrote

I can write up a nice wrapper class that will do this, and I can also incorporate the precalculation of forward and backward probabilities. We can see how much this improves things. :)

But I was too optimistic, and I don't think I have time to do this. However I thought it might be a nice task for someone looking to contribute. So I will try to detail the steps to do this below. Overall, our goal is to make it easy for users to train Hidden Markov Models with Gaussian Mixture Models where each Gaussian has the constraint that the covariance matrix is diagonal. Right now, we provide (through the bindings) discrete, gaussian, and gmm but gmm has a full covariance matrix, not a diagonal one. So you could see this proposed diagonal_gmm as something that lives "between" gaussian and gmm in terms of expressivity.

1. Make sure you're at least reasonably familiar with HMMs and GMMs. I'd suggest playing with the mlpack_hmm_train, mlpack_hmm_viterbi, mlpack_hmm_loglik and mlpack_hmm_generate command-line programs (or corresponding Python bindings) to see how a user might use these. Also familiarize yourself with the mlpack::hmm::HMM class and mlpack::gmm::GMM class. It's worth looking at the documentation in src/mlpack/methods/hmm/hmm.hpp and src/mlpack/methods/gmm/gmm.hpp, as well as the tests in src/mlpack/tests/hmm_test.cpp and src/mlpack/tests/gmm_test.cpp.

2. Note that we can train a Gaussian mixture model with a diagonal constraint with the following code:

GMM gmm(gaussians, dimensionality); // for some value of gaussians and dimensionality
gmm.Train<EMFit<kmeans::KMeans<>, DiagonalConstraint>>(data);

But that template parameter FittingType to GMM::Train() is not a class member. In addition, because of EMFit::ArmadilloGMMWrapper(), when GMM::Train() is called with DiagonalConstraint, we internally use the Armadillo gmm_diag support. We should write a new class DiagonalGMM that entirely wraps the gmm_diag class, but provides the same API as the existing GMM class.

3. Once you have DiagonalGMM, write some tests for it in src/mlpack/tests/gmm_test.cpp to make sure that a GMM trained in this way has diagonal covariance. You might consider adapting GMMTrainEMOneGaussian.

4. Now, we can turn to the HMM implementation. Modify src/mlpack/methods/hmm/hmm_model.hpp so that it can also support HMM<DiagonalGMM>. One thing that you will have to do is make sure that the serialize() function here is backwards-compatible; see LSHSearch::serialize() in src/mlpack/methods/lsh/lsh_search_impl.hpp to get an idea of how the second const unsigned int version parameter can be used to make sure things are reverse-compatible.

5. Adapt hmm_train_main.cpp, hmm_viterbi_main.cpp, hmm_loglik_main.cpp, and hmm_generate_main.cpp to also support an HMM type diagonal_gmm.

6. Adapt the tests in src/mlpack/tests/main_tests/hmm_*_test.cpp to also test the diagonal_gmm HMM type.

7. Using mlpack_hmm_train (or hmm_train() in Python), ensure that a diagonal GMM HMM trains faster than a regular GMM HMM on some example dataset. If it does, then this all behaves as expected and I think we are done. 👍

[KimSangYeon-DGU]

I'm working this now. :) #1666.

[prateeksingh0001]

@KimSangYeon-DGU are you still working on this issue? else, I can take it up :)

[KimSangYeon-DGU]

@prateeksingh0001
Yes, I'm currently working. :)

[mlpack-bot]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! 👍

[KimSangYeon-DGU]

I want to keep this open. :)

[mlpack-bot]

This issue has been automatically marked as stale because it has not had any recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions! 👍

[KimSangYeon-DGU]

I want to keep this open.

[zoq]

Agreed, let's add the keep open label.

[rcurtin]

Since #1666 is merged this is done. 👍