OVA accuracy drop

[yaeldekel]

Running OVA with AveragedPerceptron{lr=0.8} on the iris dataset used to give the following confusion table:

               ||========================

PREDICTED || 0 | 1 | 2 | Recall
TRUTH ||========================
0 || 50 | 0 | 0 | 1.0000
1 || 0 | 47 | 3 | 0.9400
2 || 0 | 0 | 50 | 1.0000
||========================

Now it gives this:

               ||========================

PREDICTED || 0 | 1 | 2 | Recall
TRUTH ||========================
0 || 50 | 0 | 0 | 1.0000
1 || 0 | 20 | 30 | 0.4000
2 || 0 | 0 | 50 | 1.0000
||========================

Note: currently OVA doesn't calibrate its models since in line 126 this

            var calibratedModel = transformer.Model as TScalarPredictor;

should be this instead:

            var calibratedModel = transformer.Model as TDistPredictor;

[TomFinley]

Hi @yaeldekel and @abgoswam I hear that we are investigating this today. It might be of help if we saw the sample in full, so we can appreciate perhaps what is going on. As it is now we don't see why this might be occurring, and it might be as trivial as the root data-view being a text loader (which is not shufflable) vs. the cache data view (which is shufflable). But that's just a guess since I do not know.

Perhaps this came from a unit test, but if it is that's not obvious?

[abgoswam]

Hi @TomFinley . The issue reported above came from the internal repo revival effort (test case : MulticlassReductionTest). We see a difference in accuracy when running OVA + AvgPerceptron. Expected accuracy 0.98 v/s Observed accuracy 0.80

As a part of the investigation, we identified one issue related to calibration. Fix #1512. Even with this fix we do not get the 0.98 accuracy for the unit test above

So, I tried digging deeper into the OVA implementation in ML.NET. I have a unit test with OVA + LR without calibration. Interestingly, I observe this unit test performs quite poorly (accuracy ~0.67). An equivalent pipeline with v3.10 of internal repo gives accuracy ~0.94

abgoswam@f606ebf#diff-43725ce400d2a0bd85f373d9004faf15R14

The goal is to identify OVA bugs (if any) in ML.NET, so we can fix them so we can get the 0.98 accuracy expected by the unit test in the internal repo.

[abgoswam]

Even with the 2 fixes above, OVA + AveragedPerceptron still performs poorly.

Iris dataset.

• Internal repo : 0.97 (micro accuracy)
• ML.NET : 0.8 (micro accuracy)

(Sample test)

    public void OvaAveragedPerceptronWithProbability()
    {
        string dataPath = GetDataPath("iris.txt");
         // Create a new context for ML.NET operations. It can be used for exception tracking and logging, 
        // as a catalog of available operations and as the source of randomness.
        var mlContext = new MLContext(seed: 1);
        var reader = new TextLoader(mlContext, new TextLoader.Arguments()
        {
            Column = new[]
                    {
                        new TextLoader.Column("Label", DataKind.R4, 0),
                        new TextLoader.Column("Features", DataKind.R4, new [] { new TextLoader.Range(1, 4) }),
                    }
        });
         // Data
        var data = reader.Read(GetDataPath(dataPath));
         // Pipeline
        var pipeline = new Ova(
            mlContext,
            new AveragedPerceptronTrainer(mlContext, "Label", "Features"));
         var model = pipeline.Fit(data);
        var predictions = model.Transform(data);
         // Metrics
        var metrics = mlContext.MulticlassClassification.Evaluate(predictions);

[shauheen]

@abgoswam have you investigated this further?

[abgoswam]

@shauheen . I looked into this but have not been able to find the root cause yet (as u had suggested, I have pasted a sample of the test case which shows the accuracy drop).

[abgoswam]

• The default NumIterations for AveragedPerceptron is set to 1. This default value seems to be the main cause of the variability, esp since the notion of seed value varies between the two platforms.

• When I set the NumIterations = 100, both InternalRepo and ML.NET produce similar results for OVA with AveragedPerceptron.

• I tried two datasets. In fact for the Wine dataset, ML.NET has better accuracy than InternalRepo when we use OVA + AvgPerceptron

• Based on the results below, I do not think this is a bug.

---

Wine dataset .

InternalRepo
maml.exe traintest loader=TextLoader{sep=, col=Label:R4:0 col=Features:R4:1-13} tr=OVA{p=AveragedPerceptron{iter=100}} data=wine.data.txt test=wine.data.txt seed=1 norm=No

Accuracy(micro-avg): 0.679775 (0.0000)
Accuracy(macro-avg): 0.625433 (0.0000)
Log-loss: 0.639056 (0.0000)
Log-loss reduction: 41.157163 (0.0000)

ML.NET
dotnet MML.dll TrainTest loader=Text{sep=, col=Label:R4:0 col=Features:R4:1-13} tr=OVA{p=AveragedPerceptron{iter=100}} data=wine.data.txt test=wine.data.txt seed=1 norm=No

Accuracy(micro-avg): 0.735955 (0.0000)
Accuracy(macro-avg): 0.703876 (0.0000)
Log-loss: 0.629104 (0.0000)
Log-loss reduction: 42.073476 (0.0000)

---

Iris dataset.

InternalRepo
maml.exe traintest tr=OVA{p=AveragedPerceptron{iter=100}} data=iris.txt test=iris.txt seed=1 norm=no

Accuracy(micro-avg): 0.960000 (0.0000)
Accuracy(macro-avg): 0.960000 (0.0000)
Log-loss: 0.247560 (0.0000)
Log-loss reduction: 77.466111 (0.0000)

ML.NET
dotnet MML.dll TrainTest loader=Text{col=Label:R4:0 col=Features:R4:1-4} tr=OVA{p=AveragedPerceptron{iter=100}} data=iris.txt test=iris.txt seed=1 norm=No

Accuracy(micro-avg): 0.953333 (0.0000)
Accuracy(macro-avg): 0.953333 (0.0000)
Log-loss: 0.187760 (0.0000)
Log-loss reduction: 82.909336 (0.0000)

@yaeldekel @shauheen @sfilipi