Ignore hidden columns in AutoML schema checks of validation data #4490

daholste · 2019-11-20T23:07:29Z

When the AutoML API consumes data, it validates schema consistency between the train and validation data.

There are two bugs in this logic:

The API asserts that the count of columns in the train and validation data must be equal. This throws an exception if the two data views have the same number of active columns but a different number of hidden columns. This PR updates to assert that the # of active (not hidden) columns in the train and validation data are equal.
If either the train or validation data has a hidden column with a type that differs from an active column of the same name, an exception is thrown. This PR restricts type consistency checks to active columns only.


        Ignore hidden columns in AutoML schema checks of validation data

justinormont

LGTM.

justinormont · 2019-11-21T00:07:27Z

src/Microsoft.ML.AutoML/Utils/UserInputValidationUtil.cs

@@ -183,14 +183,19 @@ private static void ValidateValidationData(IDataView trainData, IDataView valida

            const string schemaMismatchError = "Training data and validation data schemas do not match.";

-            if (trainData.Schema.Count != validationData.Schema.Count)
+            if (trainData.Schema.Count(c => !c.IsHidden) != validationData.Schema.Count(c => !c.IsHidden))
            {
                throw new ArgumentException($"{schemaMismatchError} Train data has '{trainData.Schema.Count}' columns," +
                    $"and validation data has '{validationData.Schema.Count}' columns.", nameof(validationData));
            }

            foreach (var trainCol in trainData.Schema)


Could use a comment...

Suggested change

foreach (var trainCol in trainData.Schema)

// Also indirectly checks for new columns in the validation datasets as we above enforce the column counts are equal

foreach (var trainCol in trainData.Schema)

I was otherwise going to suggest we check the reverse direction. A comment helps future readers to not have to think through it.

codecov · 2019-11-21T00:09:23Z

Codecov Report

❗️ No coverage uploaded for pull request base (master@b7db4fa). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master    #4490   +/-   ##
=========================================
  Coverage          ?   74.85%           
=========================================
  Files             ?      908           
  Lines             ?   159880           
  Branches          ?    17215           
=========================================
  Hits              ?   119678           
  Misses            ?    35384           
  Partials          ?     4818

Flag	Coverage Δ
#Debug	`74.85% <100%> (?)`
#production	`70.2% <100%> (?)`
#test	`90.19% <100%> (?)`

Impacted Files	Coverage Δ
...rosoft.ML.AutoML.Tests/UserInputValidationTests.cs	`100% <100%> (ø)`
...crosoft.ML.AutoML/Utils/UserInputValidationUtil.cs	`91.81% <100%> (ø)`


        comment rev

Ignore hidden columns in AutoML schema checks of validation data

Loading status checks…

41e7d36

daholste added the AutoML label Nov 20, 2019

daholste requested a review from dotnet/mlnet-automl as a code owner Nov 20, 2019

justinormont approved these changes Nov 21, 2019

View changes

justinormont reviewed Nov 21, 2019

View changes

comment rev

Loading status checks…

e24d565

dotnet/machinelearning

Ignore hidden columns in AutoML schema checks of validation data #4490

Ignore hidden columns in AutoML schema checks of validation data #4490

daholste commented Nov 20, 2019 •

edited

justinormont left a comment

This comment has been minimized.

This comment has been minimized.

codecov bot commented Nov 21, 2019 •

edited

dotnet/machinelearning

Join GitHub today

Ignore hidden columns in AutoML schema checks of validation data #4490

Conversation

daholste commented Nov 20, 2019 • edited

justinormont left a comment

This comment has been minimized.

justinormont Nov 21, 2019

This comment has been minimized.

codecov bot commented Nov 21, 2019 • edited

Codecov Report

daholste commented Nov 20, 2019 •

edited

codecov bot commented Nov 21, 2019 •

edited