Writing a step of your own

Writing a step of your own

Every verb of a pipeline is a small record that names itself, says what it takes, and can be written to a file and read back. The ones this library ships are written that way, and so is the one the indicators package adds, so a step of your own stands exactly where they stand: it is declared in the chain, it is written into the pipeline's file under its verb, and a program that has never seen your class reads it back once its catalog has been taught the verb.

A step is a record that says four things. Its Name, the verb it is written under — lowercase and dotted, like feature.weeks; its Purpose, one sentence a person reads; its Parameters, each bound to the property that holds it; and ReadFrom, which builds the step back from the JSON it was written as. The parameters are typed keys — a column that must hold numbers, a name for a new column, a number, a share — and one list of them writes the step, reads it, checks it, and draws its form in the notebook, so a key is typed in one place. Make the step a record: two declarations are compared step by step, and a step that compares by reference makes a pipeline read back from its file unequal to the one it came from.

Then it says what kind of step it is, by one capability. A step that does arithmetic on the rows and learns nothing from them is an IAddsColumns: it puts new columns on the table, and it may stand above the split. A step that learns something from the training rows and replays it is an IFittedStep: it fits on the training rows alone, applies what it learned to every row, and may stand only below the split. A step cannot be both, because the run could not place it. IDescribesColumns is how it says which columns it leaves behind, so a declaration can follow the columns down from the schema and refuse a step that reads one that is not there.

A step that adds a column and learns nothing:

using System.Text.Json;
using DeepSharp.Pipelines;

var catalog = StepCatalog.BuiltIn().WithWeeks();

var prepared = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Timestamp("Date").Number("Animals", "Age", "Weight"))
    .Add(new WeeksStep("Age", "AgeInWeeks"))              // learns nothing, so it may stand above the split
    .SplitAtRandom(0.70, 0.15)
    .Target("Weight")
    .Drop("Date", "Age")
    .Build()
    .Run();

// The verb is in the file, and a program that has never met the class reads it back with a catalog taught the verb.
var again = PreparedData.FromJson(prepared.ToJson(), catalog);

public sealed record WeeksStep : IPipelineStep<WeeksStep>, IAddsColumns, IDescribesColumns
{
    private static readonly ColumnParameter FromKey = new("from", "The column that holds days.", "age", ColumnKinds.Numbers);
    private static readonly NewColumnParameter IntoKey = new("into", "What the weeks are called.", "age_in_weeks");

    public WeeksStep(string from, string into)
    {
        From = FromKey.Require(from);
        Into = IntoKey.Require(into)!;
    }

    public string From { get; }

    public string Into { get; }

    public static string Name => "feature.weeks";

    public static string Purpose => "Turns a number of days into a number of weeks.";

    public string Verb => Name;

    public static StepParameters<WeeksStep> Parameters { get; } = new StepParameters<WeeksStep>()
        .With(FromKey, step => step.From)
        .With(IntoKey, step => step.Into);

    public ColumnState After(ColumnState before) => before.With(Into, ColumnKind.Number);

    public void AddTo(Table table) =>
        table.Put(new Column<double>(Into, ColumnKind.Number, table.NumbersOf(From).Select(days => days / 7)));

    public static WeeksStep ReadFrom(JsonElement element) => new(FromKey.Read(element), IntoKey.Read(element)!);
}

public static class WeeksExtensions
{
    extension(StepCatalog catalog)
    {
        public StepCatalog WithWeeks()
        {
            catalog.Register<WeeksStep>();

            return catalog;
        }
    }
}

A catalog is taught a verb, and a file is read with a catalog. StepCatalog.BuiltIn() knows the verbs this library ships; catalog.Register<TStep>() adds one, and writing it as an extension — WithWeeks() here, WithIndicators() in the indicators package — keeps registration to one sentence. A file that names a verb its catalog was never taught is refused by name, with the nearest verb it does know. An application with a host registers the verb once, for every catalog it builds, by offering an IStepContribution to its services before it calls AddDeepSharpPipelines().

A step that learns, and replays what it learned:

using System.Text.Json;
using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Timestamp("Date").Number("Animals", "Age", "Weight"))
    .SplitAtRandom(0.70, 0.15)
    .Target("Weight")
    .Drop("Date")
    .Add(new CentredStep("Age", "Age_centred"))             // below the split: it learns from the training rows
    .Build()
    .Run();

Console.WriteLine(prepared.ToJson().Contains("\"mean\"", StringComparison.Ordinal));   // what it learned travels with the file

public sealed record CentredStep : IPipelineStep<CentredStep>, IFittedStep, IDescribesColumns
{
    private static readonly ColumnParameter FromKey = new("from", "The column to centre.", "age", ColumnKinds.Numbers);
    private static readonly NewColumnParameter IntoKey = new("into", "What the centred column is called.", "age_centred");

    public CentredStep(string from, string into)
    {
        From = FromKey.Require(from);
        Into = IntoKey.Require(into)!;
    }

    public string From { get; }

    public string Into { get; }

    public static string Name => "feature.centred";

    public static string Purpose => "Subtracts the mean of the training rows from a column.";

    public string Verb => Name;

    public static StepParameters<CentredStep> Parameters { get; } = new StepParameters<CentredStep>()
        .With(FromKey, step => step.From)
        .With(IntoKey, step => step.Into);

    public ColumnState After(ColumnState before) => before.With(Into, ColumnKind.Number);

    public FittedStepValues Fit(Table table, IReadOnlyList<Part> parts)
    {
        var values = table.NumbersOf(From);
        var mean = Enumerable.Range(0, table.RowCount)
            .Where(row => parts[row] == Part.Train && values[row] is not null)
            .Average(row => values[row]!.Value);
        var learned = new FittedStepValues();

        learned.Learned("mean", mean);

        return learned;
    }

    public void ApplyTo(Table table, FittedStepValues fitted)
    {
        var mean = fitted.Number("mean");

        table.Put(new Column<double>(Into, ColumnKind.Number, table.NumbersOf(From).Select(value => value - mean)));
    }

    public static CentredStep ReadFrom(JsonElement element) => new(FromKey.Read(element), IntoKey.Read(element)!);
}

Fit is given the table and which part each row is in, and learns from the training rows alone; what it returns is written into the fitted half of the file under the step's key. ApplyTo is given every row and what was learned, and learns nothing more, so validation, test and a row that arrives in production next year are all treated with the same number. A step that some learners do without can say so as well, by Pipeline's IMeetsANeed.

Take care with a step that learns from the answer. ApplyTo is handed every row and does not know which of them were the training rows, so a step cannot treat a training row differently from any other. For a column of numbers about the rows themselves that is exactly right. For a statistic of the answer — a farm's mean weight, a customer's average spend, the history of a group — it is a leak that no test of the step will show: the training row's own answer is inside the group mean it is handed, and a model learns that the feature knows the answer. In a simulation with a farm effect of one and a noise of two per flock, three flocks per farm, such a mean scored an R² of 0.53 on the rows it was fitted on and −0.23 on rows it had not seen; the same mean with each training row left out of its own farm scored 0.06 and 0.05, which is the honest size of what the farm tells. Work such a history out with the row's own answer left out, and for the training rows only, before the data reaches the pipeline, and hand the pipeline the column it made; the pipeline then treats it as the number it is.

Next: Pipeline has every verb the library ships · Saving and replaying says what a file keeps of a fit