How to determine optimal number of clusters with K-Mean

[Vijay27anand]

System information

ML.NET 1.4.0

Issue

I need to know how to find optimal number of clusters for K-Mean for the dataset, there are some methods that can be used like Silhouette Method or Gap analysis or Elbow methods. Does ML.NET have any API that i can use for identifying optimal K for the dataset or any way i can find. Attaching my dataset also. It will be very helpful for my use case.

InputforClustering.xlsx

Source code / logs

[antoniovs1029]

Hi, @Vijay27anand

Unfortunately ML.NET doesn't have the tools you've mentioned, so I will tag this as a feature request.

On the other hand, notice that you could run the Elbow method manually. I.e., run different experiments with different K's, and then use the Evaluate method as follows to get the ClusteringMetrics.

var predictions = model.Transform(data);
var metrics = context.Clustering.Evaluate(predictions);

You could use then the AverageDistance from the metrics to determine the "elbow", and then choose the number of clusters.

[Vijay27anand]

@antoniovs1029 thanks for your reply. Yes i had implemented this manually before, but the problem i face is i can plot and somehow find the elbow visually, but is there a programmatic way to find the elbow bend that can be used to choose the K without visual seeing, that is where i am struck. Any insight on how to achieve.

[Vijay27anand]

@antoniovs1029 can you hlep me with any insight on how to achieve?

[najeeb-kazmi]

@Vijay27anand there is no programmatic way to do this in ML.NET.

[Vijay27anand]

@antoniovs1029 can you help if there is any enhancement for ML.net made that can help in finding the optimized K for the dataset for K-Mean clustering. Please let me know.