Possible issue with DynamicTimeWarp kernel class

[sophisma]

Sorry for the long post in advance.
First of all let me start by saying that this is an issue being opened after a brief interaction with Cesar in stackoverflow that can be fount in the following link.

https://stackoverflow.com/questions/43256559/getting-the-class-label-using-dynamictimewarping-using-accord-net

Some parts of this post will be from that thread.
Just to summarize.
I'm trying to create a prototype of something that can be compared to a touchpad, with the ability to identify gestures like swipe left, swipe right, double tap, etc.
While researching for a way to do this I came accross machine learning and Dynamic Time Warping.
Since my project was in .NET I decided to give Accord.NET a shot.
I started by creating a blank project and following the example on this link.

http://accord-framework.net/docs/html/T_Accord_Statistics_Kernels_DynamicTimeWarping.htm

With the help of Cesar I adapted that binary learning problem into a multi-class one.
Next, to be able to reject accidental gestures, I had to use the .Probability or .Score methods of the SVM and accept results above a certain threshold. Cesar suggested calibrating the machine
using ProbabilisticOutputCalibration.
The exampe code I used is from the second example on the following link.

http://accord-framework.net/docs/html/T_Accord_MachineLearning_VectorMachines_Learning_ProbabilisticOutputCalibration.htm

After doing this I came accross some inconsistent results.
The best way to understand what I'm about to explain is to refer back to the code on the bottom.
Let's say I have an array of gestures that I use to train my algorythm.
If after the calibration I calculate the probability of the first gesture using

machine.Probability(gestures[0], out decision1);

I get the right decision, although the probability is only approximatelly 0.67.
If after this I try to evaluate gestures[0] again I get the same result, as expected.
Now comes the weird part, if I create a new instance of a gesture (named swipeLeftGesture in the code) with the same values as gestures[0] the probability lowers to around 0.35.
If I use .Score instead of .Probability I also get inconsistent results.
You can use the following code to see this happening.
I didn't print the values to console, just place a breakpoint at the end of the Main method and inspect de variables.

************************ Source Code ************************

namespace DynamicTimeWarpingTest
{
    public class Program
    {
        public static void Main(string[] args)
        {

            double[][][] gestures =
            {
                new double[][] // Swipe left
                {
                    new double[] { 1, 1, 1 },
                    new double[] { 1, 2, 1 },
                    new double[] { 1, 2, 2 },
                    new double[] { 2, 2, 2 },
                },

                new double[][] // Swipe right
                {
                    new double[] { 1, 10, 6 },
                    new double[] { 1, 5, 6 },
                    new double[] { 6, 7, 1 },
                },

                new double[][] // Double tap
                {
                    new double[] { 8, 2, 5 },
                    new double[] { 1, 50, 4 },
                }
            };

            int[] outputs =
            {
                    0,  // Swipe left
                    1,  // Swipe right
                    2   // Double tap
            };

            var teacher = new MulticlassSupportVectorLearning<DynamicTimeWarping, double[][]>()
            {
                // Configure the learning algorithm to use SMO to train the
                //  underlying SVMs in each of the binary class subproblems.
                Learner = (param) => new SequentialMinimalOptimization<DynamicTimeWarping, double[][]>
                {
                    Complexity = 1.5,
                    Kernel = new DynamicTimeWarping(alpha: 1, degree: 1),
                    //UseKernelEstimation = true
                }
            };

            // Learn a machine
            var machine = teacher.Learn(gestures, outputs);

            // Create the multi-class learning algorithm for the machine
            var calibration = new MulticlassSupportVectorLearning<DynamicTimeWarping, double[][]>()
            {
                Model = machine, // We will start with an existing machine

                // Configure the learning algorithm to use Platt's calibration
                Learner = (param) => new ProbabilisticOutputCalibration<DynamicTimeWarping, double[][]>()
                {
                    Model = param.Model // Start with an existing machine
                }
            };

            // Configure parallel execution options
            calibration.ParallelOptions.MaxDegreeOfParallelism = 1;

            // Learn a machine
            calibration.Learn(gestures, outputs);


            // Validate Results
            double decision1, decision2, decision3, decision4, decision5, decision6;

            // First create new instances for the gestures with the same values as in the sequences array
            var swipeLeftGesture = new double[][]
            {
                new double[] { 1, 1, 1 },
                new double[] { 1, 2, 1 },
                new double[] { 1, 2, 2 },
                new double[] { 2, 2, 2 },
            };

            var swipeRightGesture = new double[][]
            {
                new double[] { 1, 10, 6 },
                new double[] { 1, 5, 6 },
                new double[] { 6, 7, 1 },
            };

            var doubleTapGesture = new double[][]
            {
                new double[] { 8, 2, 5 },
                new double[] { 1, 50, 4 },
            };

            // Check what are the decisions and probabilities of the original sequences array
            var res1 = machine.Probability(gestures[0], out decision1); // decision 0 - Probability 0.66666666570779487 - Score 0.69314717840248374
            var res2 = machine.Probability(gestures[1], out decision2); // decision 1 - Probability 0.57647717384694053 - Score 0.6931471784024833
            var res3 = machine.Probability(gestures[2], out decision3); // decision 2 - Probability 0.66666666570779465 - Score 0.6931471784024833


            // Check what are the decisions and probabilities of the newly created instances
            var res4 = machine.Probability(swipeLeftGesture, out decision4);  // decision 0 - Probability 0.34881631999941815 - Score 0.035525143563626807
            var res5 = machine.Probability(swipeRightGesture, out decision5); // decision 1 - Probability 0.38414838905152121 - Score 0.13802120233728751
            var res6 = machine.Probability(doubleTapGesture, out decision6);  // decision 2 - Probability 0.607525583968486   - Score 0.56625552124018352
        }
    }
}

[sophisma]

Hi Cézar.
Sorry to bother you, will this release be compatible with a PCL?
I know you already have a NetStandard 2.0 compliant version, the thing is that Xamarin Forms is not yet compatible with this version of NetStandard.

[cesarsouza]

Hi @sophisma,

Not directly. There is a project called Portable Accord.NET project that is basically a version of Accord.NET than is compatible with PCLs. However, it seems it has been a while since this project was last updated, so I am not sure if this update will make through it.

What is the lowest NetStandard version that Xamarin Forms can support? I can try to lower the target from NetStandard 2.0 to this version, at least for the math/machine learning/statistics projects.

Regards,
Cesar

[sophisma]

One thing that I completely forgot to mention in the original post was that I was actually using Portable Accord.NET.
Could you try targeting Netstandard 1.4?

[cesarsouza]

Hi @sophisma,

It looks like I can to target NetStandard 1.4, but the biggest problem is that serialization will not be available in this version. If you need to save/load your models, you would need to write your own conversion/serialization classes instead.

I will update this issue once those implementations are done.

Regards,
Cesar

[sophisma]

Thank you very much for trying.
Serialization to me it's not an issue, but might be to other people.
Looking forward to the new version.

[cesarsouza]

Hi @sophisma,

Just out of curiosity, do you think I could ask what application do you currently have in mind for the DynamicTimeWarping class in Xamarin.Forms? It is just that, for a long time, I haven't received many requests to support Xamarin. However, those requests have been increasing since a few months ago, and I think I have missed some big announcement/news in the previous months that made more people look into it.

Regards,
Cesar

[cesarsouza]

Hi @sophisma,

I just wanted to update that I will, in fact, be adding support for .NET Standard 1.4 in the next framework release. I should be able to commit a quite large change in a few hours. Sorry if it took a while. I can only work in the project in my free time.

Regards,
Cesar

[sophisma]

Great news.
You don't have to be sorry for anything, I am the one that should be grateful for having a Framework to begin with.
Thank you very much.

[cesarsouza]

Hi @sophisma,

I've just added .NET Standard 1.4 (pre-release) packages to NuGet. Please let me know if they work for you.

Regards,
Cesar

[sophisma]

Hi @cesarsouza,
Sorry for the delay, I had some issues with NetStandard projects and had to reinstall Visual Studio.
I installed Accord.MachineLearning v3.6.4-alpha and the support for NetStandart 1.4 seems to be working correctly.
The bug I reported originally, related to different instances with the same values having different probabilities, also seems to be corrected.
Although there is one thing, that I'm not sure is giving the correct result.
To better help me explain I'll just copy a very small fraction of the original code here:

var res1 = machine.Probability(gestures[0], out decision1); 
var res2 = machine.Probability(gestures[1], out decision2); 
var res3 = machine.Probability(gestures[2], out decision3); 

Let's say that the gestures array was used as the training data.
If after the training process I calculate the probabilities of each of the gestures on that array, shouldn't the probabilities be close to the value 1?
Right now they are all around 0.66, which seems a bit low.
The values of the decisions are correct though.
I'm sorry for bothering you if this is the expected behaviour.
I'm new to ML and I'm still learning as I go.

[cesarsouza]

I am glad you asked, actually I forgot to precise that you can get better probability estimates by setting the svm.Method property to MulticlassComputeMethod.Voting. If you are using one-vs-rest, there are also other techniques that could be used in this case (I might be committing some techniques for one-vs-rest in the near future).

Anyways, I guess I should take this opportunity to update the documentation. Can you please let me know if using MulticlassComputeMethod.Voting improves the problem you are facing? I have to say that actually the probabilities will not be that much close to 1 due many factors, including the aspects of your dataset (i.e. if samples from different classes are actually similar, then intuitively this probability should not be closer to 1) and the complexity parameters of the SVM (i.e. the SVM should be attempting to find a compromise solution where some training samples can be misclassified hoping that the model can generalize better to unseen samples, therefore it also shouldn't be that much confident about the most likely class decision).

Regards,
Cesar

[cesarsouza]

Different strategies for computing multi-label probabilities by setting the .Method method of MultilabelSupportVectorMachine to one of the possible values of the MultilabelProbabilityMethod enumeration.

[cesarsouza]

The original issue raised in the issue has been addressed in release v3.7.0.