Naming the answer

Step 6 — Naming the answer

A pipeline says what a model is asked to predict, and it says it in the declaration rather than in the code that trains. Then the handover knows which column is the answer and hands it apart from the features, and nothing can accidentally feed a model the thing it is meant to guess.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close", "AAPL.Volume"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15, gap: 5)
    .Ahead("AAPL.Close", 5, AheadAs.Return)          // the answer: the return five days on
    .Drop("Date")                                    // what no model reads, before what learns
    .Normalise("AAPL.Close", "AAPL.Volume")
    .Build()
    .Run();

var train = prices.Batch(Part.Train);

Console.WriteLine($"{train.Features.Count} rows, {train.FeatureNames.Count} features, answers apart: {train.Labels is not null}");

It stands right below the split. A return is made from its column as it was read, so no step that changes that column may stand above it. Scale AAPL.Close first and the pipeline refuses the pair at the return: Step 7, 'target.ahead': is a return on 'AAPL.Close' as it was read, and step 5, 'normalise', changes 'AAPL.Close' above it. Put the return above that step. Named right below the split, the answer is valid whatever kind it is, which is why the course puts it there for a class as well, and what no model reads is dropped next, before what learns.

Target and Ahead are two questions, not one verb with a flag. .Target("survived") says the answer is a column that is already there. .Ahead("close", 5, AheadAs.Return) makes the answer out of a column read further down the file — the price five days on, as a return rather than a level — and the rows at the end that have no such future are kept apart rather than guessed at.

A column that is the answer is not also a feature. Leaving close in as a feature while asking for close five days on is fine and often right; leaving the answer itself in is the leak that makes a model look perfect. The handover knows which is which because the declaration said so.

More than one answer is allowed. A pipeline can name several, and the handover gives each row its answers in the order the declaration named them.

What the profile is for. 'alive' hands a model the answer in another word, and 'class' says again what 'pclass' says. A profile of the source names columns like those before you declare them — and then the answer is simply not to declare them.

Next: Filling and scaling · Every output kind and what each refuses: Pipeline