EM

[trappmartin]

The implementation of the EM algorithm for GMMs is incorrect as it mixes univariate data x and multivariate component pdfs. Moreover, the calculations should be done in the log space and not in the space of probabilities for numerical reasons.

I believe the double sum in the E step should also be simplified to a single for loop.

[mossr]

Could you elaborate on your first comment? I don't agree because the input x is multivariate.

The EM algorithm implemented here is expanded upon in this notebook: https://htmlview.glitch.me/?https://github.com/mossr/PlutoNotebooks/blob/master/html/em_algorithm.html

Reminder that this repo is to highlight the 𝓈𝒾𝓂𝓅𝓁𝒾𝒸𝒾𝓉𝓎 and 𝒷𝑒𝒶𝓊𝓉𝓎 in these algorithms, so things like dealing in log-space for numerical reasons is somewhat out of scope. But I'm open to a PR if you can convince me that it's still beautiful ;) Same thing goes to the double for loop, which I'm not convinced it could be simplified any further without adding unnecessary confusion (again, these are for education and to display the algorithms in a simple/beautiful manner).

[trappmartin]

I see so x is a vector of vectors, that makes sense but is somewhat unexpected at least for me. :)

I'm happy to post an example implementation today or tomorrow of what I refer to.

[mossr]

Aw yes, the Vector{Vector} is not quite clear :) That's why issue #7 is open! Sometimes the "ugly" is tucked away in the behind-the-scenes to allow the "beauty" to shine

[trappmartin]

OK, looking at your notebook it seems that the purpose is for teaching. So in that case, I agree that it is sensible to write out the for loops and so on. It can easily be written as

for j in 1:k
    w[:,j] = pdf.(Ref(MvNormal(μ[j], Σ[j])), x) * ϕ[j]
end
	
w = w ./ sum(w[:,j] for j in 1:k)	

which I'm sure you are aware of. So I'm closing this now. Nice idea to make this repo btw.

[mossr]

Thank you :) It's been very fun to put together—threading the line between concise, educational, correct, and general (usually at the expense of generality).