Adding functional tests for all training and evaluation tasks #2646

rogancarr · 2019-02-20T01:02:04Z

This PR adds functional tests for all training and evaluation tasks.

I can evaluate a model trained for any of my tasks on test data. The evaluation outputs metrics that are relevant to the task (e.g. AUC, accuracy, P/R, and F1 for binary classification)
I can get the data that will allow me to plot PR curves (blocked by Cannot produce a Precision-Recall Curve from our Binary Classification Evaluator #2645)

codecov · 2019-02-20T01:46:05Z

Codecov Report

Merging #2646 into master will increase coverage by 0.09%.
The diff coverage is 96.29%.

@@            Coverage Diff             @@
##           master    #2646      +/-   ##
==========================================
+ Coverage   71.59%   71.69%   +0.09%     
==========================================
  Files         805      808       +3     
  Lines      142004   142102      +98     
  Branches    16119    16115       -4     
==========================================
+ Hits       101673   101879     +206     
+ Misses      35891    35788     -103     
+ Partials     4440     4435       -5

Flag	Coverage Δ
#Debug	`71.69% <96.29%> (+0.09%)`	⬆️
#production	`67.96% <ø> (+0.06%)`	⬆️
#test	`85.81% <96.29%> (+0.08%)`	⬆️

Impacted Files	Coverage Δ
...icrosoft.ML.Functional.Tests/Datasets/Sentiment.cs	`0% <0%> (ø)`
test/Microsoft.ML.Functional.Tests/Prediction.cs	`100% <100%> (ø)`	⬆️
test/Microsoft.ML.Functional.Tests/Evaluation.cs	`100% <100%> (ø)`
test/Microsoft.ML.TestFramework/Datasets.cs	`100% <100%> (ø)`	⬆️
...est/Microsoft.ML.Functional.Tests/Datasets/Iris.cs	`100% <100%> (ø)`
test/Microsoft.ML.Functional.Tests/Validation.cs	`100% <100%> (ø)`	⬆️
...ional.Tests/Datasets/TrivialMatrixFactorization.cs	`70% <70%> (ø)`
...soft.ML.Functional.Tests/Datasets/MnistOneClass.cs	`83.33% <83.33%> (ø)`
test/Microsoft.ML.Functional.Tests/Common.cs	`98.55% <94.73%> (-1.45%)`	⬇️
src/Microsoft.ML.Core/Data/ProgressReporter.cs	`70.95% <0%> (-6.99%)`	⬇️
... and 27 more

artidoro · 2019-02-21T20:50:41Z

test/Microsoft.ML.Functional.Tests/Evaluation.cs

+            // Train the model.
+            var model = pipeline.Fit(trainData);
+
+            // Evaulate the model.


Evaulate [](start = 15, length = 8)

I think there is small typo that was copy pasted everywhere, can you fix that? #Resolved

artidoro · 2019-02-21T21:01:18Z

test/Microsoft.ML.Functional.Tests/Common.cs

+        /// </summary>
+        /// <remarks>
+        /// TODO #2644: At times, AnomalyDetection.Evaluate will return a set of NaN metrics.
+        /// </remarks>


Maybe would not add the TODO to a xml comment. I would suggest having it as a regular comment only. #Resolved

Ivanidzo4ka · 2019-02-22T18:13:17Z

test/Microsoft.ML.Functional.Tests/Common.cs

+        /// Check that a <see cref="BinaryClassificationMetrics"/> object is valid.
+        /// </summary>
+        /// <param name="metrics">The metrics object.</param>
+        public static void CheckMetrics(BinaryClassificationMetrics metrics)


BinaryClassificationMetrics [](start = 40, length = 27)

we added CalibratedBinaryClassificationMetrics recently. Strangely you don't have them in your PR. #Closed

Good catch. I hadn't noticed that.

In reply to: 259454652 [](ancestors = 259454652)

artidoro

Once you fix the build and add the check and related test with CalibratedBinaryClassificationMetrics it should be good to go.

…iers

rogancarr · 2019-02-22T20:32:18Z

Once you fix the build and add the check and related test with CalibratedBinaryClassificationMetrics it should be good to go.

This is a weird build error. When the build systems do it, it's not handling the method overloading correctly. I can't repro it locally....

Ivanidzo4ka · 2019-02-22T21:02:46Z

test/Microsoft.ML.Functional.Tests/Evaluation.cs

+            var metrics = mlContext.BinaryClassification.Evaluate(scoredData);
+
+            // Check that the metrics returned are valid.
+            Common.CheckMetrics(metrics);


CheckMetrics [](start = 19, length = 12)

So the reason why I mentioned CalibratedBinaryClassificationMetrics is not only to add them to Common.cs, but also to have test on them.
Maybe have two pipelines in this test. One with calibrated, one without.
#Resolved

Gotcha ;)

In reply to: 259509132 [](ancestors = 259509132)

Ivanidzo4ka · 2019-02-22T21:05:33Z

test/Microsoft.ML.Functional.Tests/Datasets/Iris.cs

+            {
+                output.Label = input.Label;
+                // The standard set used in tests has 150 rows
+                output.GroupId = (ushort)rng.Next(0, 30);


ushort [](start = 34, length = 6)

any reason why it's ushort? #Resolved

Remnants of when I was trying to read directly as a KeyType. (see #2642)

In reply to: 259509894 [](ancestors = 259509894)

Ivanidzo4ka · 2019-02-22T21:10:25Z

test/Microsoft.ML.Functional.Tests/Datasets/TrivialMatrixFactorization.cs

+namespace Microsoft.ML.Functional.Tests.Datasets
+{
+    /// <summary>
+    /// A class containing one property per <see cref="DataKind"/>.


/// A class containing one property per . [](start = 4, length = 63)

I don't get this sentence. What DataKind has to do with MF? Your data is row+col+value triplets for matrix. #Resolved

Copy/Paste error, unfortunately.

In reply to: 259511254 [](ancestors = 259511254)

Ivanidzo4ka

Rogan Carr added 2 commits February 19, 2019 16:55

Adding functional tests for all training and evaluation tasks

2e4d4b0

Updating test summaries.

0b02fb1

rogancarr mentioned this pull request Feb 20, 2019

V1 Scenarios need to be covered by tests #2498

Open

Rogan Carr added 2 commits February 21, 2019 12:37

Merge branch 'master' into 2621_evaluation_scenarios

54dfc2f

Updating comments to end in a period!

acd2f1a

artidoro self-requested a review February 21, 2019 20:45

artidoro reviewed Feb 21, 2019

View reviewed changes

Addressing PR comments.

e9d5bad

rogancarr requested review from sfilipi and yaeldekel February 21, 2019 23:48

Ivanidzo4ka reviewed Feb 22, 2019

View reviewed changes

artidoro approved these changes Feb 22, 2019

View reviewed changes

Updating binary classification check to also check calibrated classif…

7012ea6

…iers

Ivanidzo4ka reviewed Feb 22, 2019

View reviewed changes

Rogan Carr added 5 commits February 22, 2019 15:15

Merge branch 'master' into 2621_evaluation_scenarios

94c23d2

Normalizing names after merge with master.

12d3dfe

Addressing PR comments.

42af510

Merge branch 'master' into 2621_evaluation_scenarios

51ea456

Updating to fix master merge issues around test datasets.

89e44cb

Ivanidzo4ka approved these changes Feb 23, 2019

View reviewed changes

Marking matrix factorization test with the new attribute.

051ba01

rogancarr merged commit 8001ccc into dotnet:master Feb 24, 2019

rogancarr deleted the 2621_evaluation_scenarios branch February 24, 2019 23:11

ghost locked as resolved and limited conversation to collaborators Mar 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding functional tests for all training and evaluation tasks #2646

Adding functional tests for all training and evaluation tasks #2646

rogancarr commented Feb 20, 2019

codecov bot commented Feb 20, 2019 •

edited

Loading

artidoro Feb 21, 2019 •

edited by rogancarr

Loading

artidoro Feb 21, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Feb 22, 2019 •

edited

Loading

rogancarr Feb 22, 2019

artidoro left a comment •

edited

Loading

rogancarr commented Feb 22, 2019

Ivanidzo4ka Feb 22, 2019 •

edited by rogancarr

Loading

rogancarr Feb 22, 2019

Ivanidzo4ka Feb 22, 2019 •

edited by rogancarr

Loading

rogancarr Feb 22, 2019

Ivanidzo4ka Feb 22, 2019 •

edited by rogancarr

Loading

rogancarr Feb 22, 2019

Ivanidzo4ka left a comment

Adding functional tests for all training and evaluation tasks #2646

Adding functional tests for all training and evaluation tasks #2646

Conversation

rogancarr commented Feb 20, 2019

codecov bot commented Feb 20, 2019 • edited Loading

Codecov Report

artidoro Feb 21, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

artidoro Feb 21, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

Ivanidzo4ka Feb 22, 2019 • edited Loading

Choose a reason for hiding this comment

rogancarr Feb 22, 2019

Choose a reason for hiding this comment

artidoro left a comment • edited Loading

Choose a reason for hiding this comment

rogancarr commented Feb 22, 2019

Ivanidzo4ka Feb 22, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Feb 22, 2019

Choose a reason for hiding this comment

Ivanidzo4ka Feb 22, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Feb 22, 2019

Choose a reason for hiding this comment

Ivanidzo4ka Feb 22, 2019 • edited by rogancarr Loading

Choose a reason for hiding this comment

rogancarr Feb 22, 2019

Choose a reason for hiding this comment

Ivanidzo4ka left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 20, 2019 •

edited

Loading

artidoro Feb 21, 2019 •

edited by rogancarr

Loading

artidoro Feb 21, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Feb 22, 2019 •

edited

Loading

artidoro left a comment •

edited

Loading

Ivanidzo4ka Feb 22, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Feb 22, 2019 •

edited by rogancarr

Loading

Ivanidzo4ka Feb 22, 2019 •

edited by rogancarr

Loading