Settling the gaps

Step 3 — Settling the gaps

This is the step most courses leave out, and the one the order exists for. A feature worked out from a column with a gap is itself a gap — and filling that column later cannot reach back into the feature, because a fill learns its value from the training rows and therefore stands below the split, long after the feature was worked out.

So the gaps a feature will be worked out from are settled here, above the split, with a value nobody learned.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Optional("age", ColumnKind.Number).Number("fare"))
    .SettleGaps(gaps => gaps                         // Zero, Constant(value), Refuse
        .Zero("age"))                                // a nought no row decided
    .AddFeature("older", "age", Arithmetic.Plus, "sibsp")
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("age", "older", "fare")
    .Build()
    .Run();

Console.WriteLine(prepared.Table.NumbersOf("age_was_missing").Count(value => value == 1));   // 177

Why this one may stand above the split. A nought, a number you choose, and a refusal are not values any row could have told you. No row of validation and no row of test taught them anything, so putting them in before the rows are divided leaks nothing — and the step writes nothing into the fitted half of the file, because there is nothing it learned. A replay does what the run did by doing it again.

A nought is not an absence, and the pipeline keeps the receipt. Beside every column it settles it writes <column>_was_missing, one per row, because settling destroys the difference between "absent" and "the value happened to be that" — permanently. That column is the only record left that the difference existed. A verb that marks the same column later adds to it and never overwrites it.

One answer or the other. A column settled here has no gap left, so a .FillMissing of the same column below the split can do nothing — and the pipeline refuses that line where you wrote it rather than run a step that cannot act. Settle it up here for the features, or fill it down there for the model; not both.

Dropping the rows instead is a verb of its own, .DropGaps, and it stands up here too, because rows are taken out before they are divided and never after.

Next: Working out the features · Every strategy and every refusal: Pipeline