Examples

Examples

In short

What each release brought, shown in code: a section to each release, the newest first, what is new and what was improved, each with a line on what it buys you. Every block is written against the library as it is now and compiled by the tests with every other block of this wiki, so what you copy from here builds. A fix, or a change nobody calls, has no example — the changelog says what each release changed and why.

Release	What it brought
0.12.0	A model watched after it ships
0.11.0	The third letter of the flock team, and the words a model is written in.
0.10.0	Everything the parts can be told.
0.9.0	Everything a window can be.
0.8.1	Optimizing: the numbers a network is trained with, searched for and measured by.
0.8.0	A live source, landed: the candles an exchange answers with, fetched once over a closed window and kept as a file that the pipeline then reads like any other.
0.7.1	A course to fill in: every step of a prepared pipeline in the order the steps belong, each present and waiting for what only a person knows, so a pipeline written from nothing starts from the steps rather than from a reader who has to remember which verb comes where.
0.7.0	A trainer from ML.NET, behind the same seam a network stands behind, so a tree and a network are measured by one report on the same rows — which is the only reason the two can honestly be compared.
0.6.2	Where a gap is settled before the features are worked out.
0.6.1	Where one line does one thing to many columns.
0.6.0	Where the model moves into the pipeline.
0.5.0	Where it runs on libtorch and reads what others trained, and where every gap 0.4.0 left between a trained network and a person using it is closed.
0.4.0	Where it learns.
0.3.0	A pipeline you can look at: every step written as a block of a notebook, and the data at any block shown on request.
0.2.0	The data half of the library: everything a set of rows goes through before a model ever sees it, declared once as a file you can save and replay.
0.1.0	The first release.

0.12.0

New

Keep the training rows a served model is measured against. A pipeline read from its file keeps what its steps learned, not the rows they learned it from; .Watch below the split keeps those rows, counted, so the rows a model meets long after can be held up against them.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Timestamp("day").Category("farm").Number("weight", "y"))
    .OrderBy("day")
    .SplitByTime("day", train: 0.7, validation: 0.15)
    .Target("y")
    .Watch(0.1, "weight", "farm")         // keeps the training rows of these columns, counted
    .Build()
    .Run();

File.WriteAllText("flocks.pipeline.json", prepared.ToJson());

Measure a window of served rows against them. Each watched column's distance from the training rows, flagged above the bound, with the rows it was worked out from and when the window was — so a world that moved on is seen, not guessed.

using DeepSharp.Pipelines;

var prepared = PreparedData.FromJson(File.ReadAllText("flocks.pipeline.json"), StepCatalog.BuiltIn());
var window = prepared.Watched(new CsvRowSource("served/2025-02-01.csv"));

foreach (var column in window.Columns.Where(column => column.Above))
{
    Console.WriteLine($"{window.From:yyyy-MM-dd}: {column.Column} is {column.Apart:0.00} away, over {column.WindowRows} rows");
}

Draw the windows one beside another. A line to each column, window by window, under the bound: a column drifting away is a line climbing through it.

using DeepSharp.Charts;
using DeepSharp.Pipelines;

var prepared = PreparedData.FromJson(File.ReadAllText("flocks.pipeline.json"), StepCatalog.BuiltIn());
WindowLikeness[] windows = [.. Directory.GetFiles("served", "*.csv").Order(StringComparer.Ordinal)
    .Select(day => prepared.Watched(new CsvRowSource(day)))];

File.WriteAllText("watched.svg", windows.Distances());

A step of your own can keep what it saw as a list. What a person should see, and no replay may read, can now be a list of words or a run of numbers as well as a count.

using DeepSharp.Pipelines;

var seen = new FittedStepValues();

seen.Saw("gaps", 3);                    // a count, as before
seen.Saw("farms", ["A", "B"]);          // a list of words
seen.Saw("weights", [10.0, 20.0]);      // a run of numbers

Console.WriteLine(string.Join(", ", seen.Counts));

0.11.0

New

The tensor seam has two more operations, Sin and Erf.

using DeepSharp.Tensors;

var backend = new CpuBackend();
var turns = Tensor.From(new Shape(3), [0f, MathF.PI / 2, MathF.PI]);
var spread = Tensor.From(new Shape(3), [-1f, 0f, 1f]);

Console.WriteLine(string.Join(", ", backend.Sin(turns).Values.ToArray()));   // the periodic embedding turns a number round with it
Console.WriteLine(string.Join(", ", backend.Erf(spread).Values.ToArray()));  // what the exact GELU is written in

A run can judge its epochs by the average of its weights.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass", "sibsp").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("pclass", "sibsp", "fare")
    .WithTorch(network => network
        .Dense(16).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy()
        .Average(0.99).JudgedByAverage()               // each epoch is judged by the average of the weights ...
        .StoppingAfter(patience: 10)                   // ... so the best epoch it restores is the best average
        .Run(seed: 20261017, epochs: 100))
    .Build()
    .Train();

File.WriteAllText("titanic.network.json", trained.ToJson());

A search can stop a trial that is not going anywhere, choose a setting only when another is chosen, and go on from trials it ran before.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

Pipeline Passengers(int deal) => Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitAtRandom(0.70, 0.15, seed: 100 + deal, testSeed: 7)
    .Target("survived")
    .Normalise("fare")
    .Build();

var space = new SearchSpace(
[
    new Choices("optimizer", ["sgd", "adam"]),
    new NumberRange("momentum", 0, 0.99) { OnlyWhen = new Chosen("optimizer", "sgd") },   // drawn only for a trial that chose sgd
]);

Study StudyOf(int trials) => new(space, (candidate, network) =>
{
    var layers = network.Dense(8).Relu().Dense(1);

    return (candidate.Choice("optimizer") == "sgd" ? layers.Sgd(0.05, candidate.Number("momentum")) : layers.Adam(0.01))
        .BinaryCrossEntropy().Run(seed: 1, epochs: 20);
})
{
    Trials = trials,
    Pruner = new MedianPruner(startup: 5, warmUp: 1).Prunes,   // stops a trial worse than the median at the same epoch
};

var first = StudyOf(8).Run(Passengers);
var found = StudyOf(16).Run(Passengers, first.Trials);           // goes on from the eight trials it already ran

Console.WriteLine($"{found.Trials.Count(trial => trial.Pruned)} of {found.Trials.Count} pruned; best {found.Best.Score:0.000}");

A number can be written down as how far it fills a few bins.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing(fill => fill.Median("age"))
    .PiecewiseLinear("age", bins: 8, outOfRange: OutOfRange.Clip)   // eight bins placed by the training rows' quantiles
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", prepared.Table.Columns.Select(column => column.Name)));   // age_bin1 … age_bin8 stand in place of age

A running statistic of each key is written beside every row.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Timestamp("date").Integer("round").Category("flock").Number("weight", "y"))
    .OrderBy("date", "round")
    .SplitByTime("date", train: 0.70, validation: 0.15)
    .Target("y")
    .History("flock", "weight", shrink: 5, "date", "round")      // the mean weight of the flock's training rows before this one
    .Build()
    .Run();

var served = PreparedData.FromJson(prepared.ToJson(), StepCatalog.BuiltIn()).Replay(new CsvRowSource("one-flock.csv"));   // one flock, served alone

Console.WriteLine(served["weight_history_flock"].TextAt(0));

The parts of a split can be compared.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Timestamp("date").Integer("round").Category("flock").Number("weight", "y"))
    .OrderBy("date", "round")
    .SplitByTime("date", train: 0.70, validation: 0.15)
    .Target("y")
    .Likeness(0.2, "weight", "flock")                 // flags a distance above a fifth
    .Build()
    .Run();

var likeness = prepared.Evidence.Values.OfType<PartLikeness>().Single();

foreach (var column in likeness.Columns.Where(column => column.Pairs.Any(pair => pair.Above)))
{
    Console.WriteLine($"{column.Column} lies {column.Pairs.Max(pair => pair.Apart):0.00} apart between two parts, above {likeness.WarnAbove}");
}

A layer, a loss and an output say more words.

using DeepSharp.Networks;

var compiled = new Sequential()
    .Dense(32).Gelu().RmsNorm()                  // a smooth activation, then a normalisation by the root mean square
    .Dense(3).Softmax()                          // softmax as a layer: lifted into the loss, which applies it itself
    .Compile(new Adam(0.01), new CrossEntropy { Smoothing = 0.1 });   // labels softened by a tenth

Console.WriteLine(compiled.Loss.Applies(new Softmax()));                                   // True
Console.WriteLine($"{new Poisson(logInput: true).Activation}, {new Hinge().Activation}");  // Exp, Margin

A declared network says the new words in both vocabularies, each filling in its own library's numbers.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Number("age", "temperature", "weight"))
    .SplitAtRandom(0.70, 0.15, seed: 7)
    .Target("weight")
    .Normalise("age", "temperature")
    .Report(report => report.Measure(Metric.Rmse, Metric.Mape).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .WithTensorflow(network => network                    // .WithTorch(…) says the same in PyTorch's words
        .Periodic(sigma: 0.1).Relu()                      // Gorishniy et al.'s PLR; sigma has no default, the data decides it
        .Dense(16).LeakyRelu().RmsNorm()                  // a slope of three tenths here, a hundredth under .WithTorch
        .Dense(1).Adam(0.01).Huber()                      // an amount with outliers; Quantile(0.9), Focal(), Hinge() and Poisson() stand beside it
        .Run(seed: 20261017, epochs: 50))
    .Build()
    .Train();

File.WriteAllText("flocks.network.json", trained.ToJson());

An embedding reads the block a category was written down as.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Category("pclass").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .EncodeCategories()
    .Normalise("fare")
    .WithTorch(network => network
        .Embedding("pclass", width: 2)                    // a vector it learns for each class: stands before any other layer
        .Dense(8).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20261017, epochs: 20))
    .Build();

Console.WriteLine(string.Join(", ", pipeline.Run().CategoryBlock("pclass")));   // the columns the category was written down as

File.WriteAllText("titanic.network.json", pipeline.Train().ToJson());

A report measures F1, the area under the curve, the error as a share of the answer, and the Brier score.

using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Category("sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing(fill => fill.Median("age"))
    .EncodeCategories()
    .Normalise("age", "fare")
    .Report(report => report.Measure(Metric.F1, Metric.Auc, Metric.Brier).On(Part.Validation, Part.Test).As(Shown.Numbers, Shown.Drawn))
    .WithTorch(network => network.Dense(8).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20261017, epochs: 20))
    .Build()
    .Train();

File.WriteAllText("titanic-reliability.svg", trained.Measures!.Reliability());   // the Brier score's curve: the chances said against how often they held

0.10.0

New

A bias, a scale and a shift can be absent, so the properties that hold them can be null.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var weights = Tensor.From(new Shape(3, 2), [0.5f, -0.5f, 0.25f, 0.75f, -1f, 1f]);
var layer = new Dense(weights, null);                       // no bias: the slot is not there
var norm = new BatchNorm(2, new Affine { WithoutScale = true });

Console.WriteLine(layer.Bias is null);                      // True: code that reads a bias asks first
Console.WriteLine(norm.Weight is null);                     // True: no scale
Console.WriteLine(norm.Bias is not null);                   // True: the shift is kept

A layer begins with a Beginning, which takes the seat the draws had.

using DeepSharp.Networks;

var stream = new RandomStream(42);
var keras = new Beginning(stream.Draw("initialise:0", 0, 0)) { Weights = new GlorotUniform(), Bias = new Zeros() };       // Keras's start
var steady = new Beginning(stream.Draw("initialise:1", 0, 0)) { Weights = new Orthogonal(), Bias = new Constant(0.1) };

var network = new LayerStack(new Dense(14, 16, keras), new Relu(), new Dense(16, 16, steady));

Console.WriteLine(network.Slots().Count());

A layer, a normalisation and a padding say what they leave out and what they fill with.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var description = new Sequential()
    .Input(new Shape(64, 3))
    .Pad1D(new Sides(2, 2), Filling.Mirror)             // two places each end, taken from the series itself
    .Conv1D(16, new Window1D(5), new Beginning { WithoutBias = true })
    .BatchNorm(new Affine { WithoutScale = true })      // keeps its shift, leaves its scale out
    .Relu()
    .GlobalMaxPool1D()
    .Dense(1);

var network = description.Lower(new Shape(64, 3), new RandomStream(42));
Console.WriteLine(network.Slots().Count());

A parameter can carry a penalty, a bound and a hold, and the loss a run reports includes the penalty.

using DeepSharp.Networks;

var stream = new RandomStream(42);
var first = new Dense(14, 16, stream.Draw("initialise:0", 0, 0));
var last = new Dense(16, 1, stream.Draw("initialise:2", 0, 0));
var network = new LayerStack(first, new Relu(), last);

first.Hold();                                                     // no step moves its numbers, and the loss is not differentiated by them
last.Weight.Rules = new SlotRules { L2 = 0.0001, MaxNorm = 3 };   // a penalty the loss adds, and a bound after every step

Console.WriteLine(network.Parameters().Count(parameter => parameter.Learns));   // 2: only the last layer's weight and bias
first.Release();                                                  // and the first learns again

Optimizers take the options PyTorch's have.

using DeepSharp.Networks;

var sgd = new Sgd(0.05) { Momentum = 0.9, Nesterov = true, WeightDecay = 0.0001 };
var adam = new Adam(0.001) { AmsGrad = true };
var rms = new RmsProp(0.01) { Centered = true };

sgd.Validate();    // options that do not go together are refused by name, here or wherever the optimizer is made
Console.WriteLine($"{sgd.Nesterov} {adam.AmsGrad} {rms.Centered}");

A run can clip each gradient value, step once every several batches and keep an average of its weights.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("fare", "sibsp", "parch")
    .WithTorch(network => network
        .Dense(16).Relu().Dense(1)
        .Sgd(0.05, momentum: 0.9)
        .BinaryCrossEntropy()
        .ClipValue(1).StepEvery(4).Average(0.99)     // each gradient value clipped, a step from four batches, an average of the weights
        .Run(seed: 20260929, epochs: 100))
    .Build();

File.WriteAllText("titanic.network.json", pipeline.Train().ToJson());

0.9.0

New

A window says its stride for each axis, so Window.Stride is a pair of numbers where it was one.

using DeepSharp.Tensors;

var window = new Window(3, 3) { Stride = new Steps(2, 1) };            // two down, one across
var square = new Window(3, 3) { Stride = 2 };                           // one number still means both axes
var volume = new Window3D(3, 3, 3) { Stride = new Steps3D(1, 2, 2) };  // and a volume's window takes a Steps3D

Console.WriteLine($"{window.Stride.Down} down, {window.Stride.Across} across; {square.Stride}; {volume.Stride}");

A window can say four things it could not, and says none of them when it does not.

using DeepSharp.Tensors;

var window = new Window(3, 3)
{
    Stride = new Steps(2, 1),                                  // a stride of its own for each axis
    Dilation = 2,                                              // cells two places apart
    RoundsUp = true,                                           // the place at the end that does not fit whole counts
    Border = new Margins(new Sides(1, 2), new Sides(0, 1)),    // a border that differs from side to side
};

Console.WriteLine($"{window.RowsOver(28)} rows and {window.ColumnsOver(28)} columns of places over a 28 by 28 image");

A convolution holds a Channels: how many it reads, how many it makes, and in how many groups.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var description = new Sequential()
    .Input(new Shape(28, 28, 8))
    .Conv2D(16, new Window(3, 3), groups: 4)         // eight channels in, in four groups of two, sixteen out
    .Relu()
    .DepthwiseConv2D(2, new Window(3, 3))            // a group for every channel, two channels made of each
    .Relu()
    .GlobalAvgPool2D()
    .Dense(10);

var network = description.Lower(new Shape(28, 28, 8), new RandomStream(42));

foreach (var (path, layer) in network.HeldLayers())
{
    if (layer is Convolution convolution)
    {
        Console.WriteLine($"{path}: {convolution.Channels}, kernel {convolution.Weight.Value.Shape}");   // as small as its groups
    }
}

Three families of layer are new.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var description = new Sequential()
    .Input(new Shape(28, 28, 1))
    .Conv2D(8, new Window(3, 3) { Stride = 2, Padding = 1 })                        // 28 by 28 down to 14 by 14
    .Relu()
    .Conv2DTranspose(1, new Window(3, 3) { Stride = 2, Padding = 1 }, outputPadding: 1)   // gives the 28 by 28 back
    .Pad2D(new Margins(new Sides(2, 2), new Sides(2, 2)))                           // adds places and does nothing else
    .AdaptiveAvgPool2D(4, 4)                                                        // pools to a size, whatever it was handed
    .Flatten()
    .Dense(10);

var network = description.Lower(new Shape(28, 28, 1), new RandomStream(42));
Console.WriteLine(network.Parameters().Count());

A pooling can round the number of places up, and the average that counts its padding counts only the border.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var whole = new Window(2, 2) { Stride = 2 };
var rounded = new Window(2, 2) { Stride = 2, RoundsUp = true };

Console.WriteLine($"{whole.RowsOver(15)} rows of places, or {rounded.RowsOver(15)} counting the one that does not fit whole");

var description = new Sequential()
    .Input(new Shape(15, 15, 3))
    .MaxPool2D(rounded)
    .AvgPool2D(new Window(3, 3) { Stride = 2, Padding = 1 }, countsPadding: true)   // divides by the border it was told to pad with
    .GlobalAvgPool2D()
    .Dense(1);

Console.WriteLine(description.Lower(new Shape(15, 15, 3), new RandomStream(42)).Parameters().Count());

0.8.1

New

AdamW, RMSprop and Nadam.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("fare", "sibsp", "parch")
    .WithTorch(network => network
        .Dense(16).Relu().Dense(1)
        .AdamW(0.001, weightDecay: 0.01)             // or .RmsProp(0.01, alpha: 0.99), or .Nadam(0.002)
        .BinaryCrossEntropy()
        .Run(seed: 20260929, epochs: 50))
    .Build()
    .Train();

Console.WriteLine($"{trained.History!.Epochs.Count} epochs");

Gradient clipping by norm, a warm-up, and an initialiser on a layer.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var rows = new TrainingData(
    Tensor.From(new Shape(4, 2), [0f, 0f, 0f, 1f, 1f, 0f, 1f, 1f]),
    Tensor.From(new Shape(4, 1), [0f, 1f, 1f, 0f]));

var compiled = new Sequential()
    .Dense(8, new GlorotUniform()).Relu().Dense(1)                       // a layer's numbers start another way
    .Compile(new Adam(0.05), new BinaryCrossEntropy(), new LinearWarmup(5, then: new CosineDecay(50)));   // up over five epochs, then down

var history = compiled.Fit(rows, null, new FitOptions(seed: 7)
{
    Epochs = 55,
    GradientClip = new GradientClip(1.0),                                // a step's gradients are clipped to a norm of one together
});

Console.WriteLine(string.Join(", ", history.Epochs.Take(7).Select(epoch => $"{epoch.LearningRate:0.0000}")));

A run can be watched and stopped.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("fare", "sibsp", "parch")
    .WithTorch(network => network.Dense(16).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20260929, epochs: 500))
    .Build();

using var stop = new CancellationTokenSource(TimeSpan.FromMinutes(1));   // a run that takes hours is cut short from outside

// Every epoch is handed over as it ends; a stopped run throws OperationCanceledException and returns nothing.
var trained = pipeline.Train(null, epoch => Console.WriteLine($"{epoch.Number}: {epoch.Loss:0.000}"), stop.Token);

Console.WriteLine($"{trained.History!.Epochs.Count} epochs");

An optimizer of your own.

using System.Text.Json;
using DeepSharp.Networks;
using DeepSharp.Tensors;

var catalog = NetworkCatalog.BuiltIn().Register<PlainSteps>();      // a checkpoint file can now name "plainSteps"
var compiled = new Sequential().Dense(8).Relu().Dense(1).Compile(new PlainSteps(0.05), new MeanSquaredError());

Console.WriteLine($"{compiled.Optimizer.Rate}, registered: {catalog.Names.Contains(PlainSteps.Name)}");

public sealed class PlainSteps(double rate) : Optimizer(rate), ISaved<PlainSteps>
{
    public static string Name => "plainSteps";

    public static PlainSteps Rebuild(JsonElement settings, Rebuilding rebuilding) => new(rebuilding.Number(settings, "rate"));

    public void WriteSettings(Utf8JsonWriter writer) => writer.WriteNumber("rate", Rate);

    protected override SlotMemory? MemoryOf(Parameter parameter) => null;              // nothing is remembered between steps

    protected override void Recall(Parameter parameter, SlotMemory memory) => RequireNothingMore(parameter, memory);

    protected override Tensor Moved(Parameter parameter, Tensor gradient, double rate, ITensorBackend backend) =>
        backend.Subtract(parameter.Value, backend.Scale(gradient, Scalar(backend, rate)));
}

Convolutions along a series and through a volume.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var series = new Sequential()                                                   // 64 steps of 3 channels
    .Conv1D(16, new Window1D(5) { PaddingMode = PaddingMode.Causal })          // no place sees what follows it
    .Relu().GlobalMaxPool1D().Dense(1)
    .Lower(new Shape(64, 3), new RandomStream(7));

var volume = new Sequential()                                                   // planes by rows by columns by channels
    .Conv3D(4, new Window3D(3, 3, 3) { PaddingMode = PaddingMode.Same })
    .Relu().GlobalAvgPool3D().Dense(1)
    .Lower(new Shape(8, 16, 16, 1), new RandomStream(7));

foreach (var named in series.Slots().Concat(volume.Slots()))
{
    Console.WriteLine($"{named.Path}: {named.Slot.Value.Shape}");               // one kernel matrix for every axis
}

Poolings.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var images = new Sequential()
    .Conv2D(8, new Window(3, 3) { Padding = 1 })
    .Relu()
    .MaxPool2D(2)                                                                    // runs of two that do not overlap, as Keras's pooling does
    .AvgPool2D(new Window(3, 3) { Stride = 2, Padding = 1 }, countsPadding: true)    // the border counts in the average, as PyTorch's count_include_pad
    .Flatten()
    .Dense(10)
    .Lower(new Shape(28, 28, 1), new RandomStream(7));

Console.WriteLine(string.Join(" > ", images.Layers.Select(layer => layer.GetType().Name)));

Global poolings and the dropout of whole channels.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var images = new Sequential()
    .Conv2D(8, new Window(3, 3) { PaddingMode = PaddingMode.Same })
    .Relu()
    .SpatialDropout2D(0.2)                  // whole channels left out while training, in every place at once
    .GlobalAvgPool2D(keepsAxes: true)       // one value for every channel, the axes kept as axes of one place: Keras's keepdims
    .Flatten()
    .Dense(10)
    .Lower(new Shape(28, 28, 1), new RandomStream(7));

Console.WriteLine($"{images.Layers.Count} layers, {images.Slots().Count()} slots");

The three readers read them.

using DeepSharp.Import.Keras;
using DeepSharp.Import.Onnx;
using DeepSharp.Import.PyTorch;
using DeepSharp.Networks;
using DeepSharp.Tensors;

using var keras = File.OpenRead("series.keras");                       // Conv1D('causal'), MaxPooling1D, SpatialDropout1D, GlobalMaxPooling1D
var fromKeras = new KerasFile().Read(keras);

using var graph = File.OpenRead("images.onnx");                        // Conv, MaxPool, GlobalAveragePool
var fromOnnx = new OnnxFile(new BinaryCrossEntropy()).Read(graph);     // a graph names no loss, so it is handed one

var volume = new Sequential().Conv3D(4, new Window3D(3, 3, 3)).Relu().GlobalAvgPool3D().Dense(1).Lower(new Shape(8, 16, 16, 1), new RandomStream(7));
using var state = File.OpenRead("volume.safetensors");                 // a state PyTorch saved holds the numbers, not the network
var fromTorch = new SafetensorsFile(volume, new MeanSquaredError()).Read(state);

Console.WriteLine($"{fromKeras.Network.Slots().Count()} slots from Keras, {fromOnnx.Network.Slots().Count()} from ONNX, {fromTorch.Network.Slots().Count()} from PyTorch");

A distribution in an order.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

string[] bands = [.. Enumerable.Range(0, 70).Select(band => $"w{500 + (50 * band)}")];   // 500 g to 4000 g

var trained = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Number("age", "chicks").Number(bands))
    .SplitAtRandom(0.70, 0.15)
    .Distribution(bands, scaleBy: "chicks", ordered: true)          // the bands have an order: one band off is nearly right
    .NormaliseRow(Norm.L1, bands)
    .Normalise("age", "chicks")
    .Report(report => report.Measure(Metric.Emd, Metric.Rps, Metric.Kl).On(Part.Validation, Part.Test).As(Shown.Numbers))
    .WithTorch(network => network.Dense(32).Relu().Dense(bands.Length).Adam(0.001).EarthMoversDistance().Run(seed: 1, epochs: 30))
    .Build()
    .Train();

Console.WriteLine($"{trained.Measures!.Parts[^1].Values[0].Value:0.000} bands off, on average, on the test rows");

What a flock that arrived short leaves.

using DeepSharp.Pipelines;

string[] bands = [.. Enumerable.Range(0, 70).Select(band => $"w{500 + (50 * band)}")];

var flocks = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Number("age", "planned").Number(bands))          // the bands hold counts: fewer birds may arrive than were planned
    .SplitAtRandom(0.70, 0.15)
    .Distribution(bands, scaleBy: "planned", ordered: true, remainder: "other")  // one more answer after the bands: what is left of the planned flock
    .Normalise("age")
    .Build()
    .Run();

Console.WriteLine(flocks.Batch(Part.Train).AnswerNames?.Last());

target.numbers.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Number("age", "feed", "weight", "mortality"))
    .SplitAtRandom(0.70, 0.15)
    .Numbers(["weight", "mortality"])              // two free numbers, one answer
    .Normalise("age", "feed")
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", prepared.Batch(Part.Train).AnswerNames ?? []));

read.join.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadJoin("planned.csv", "arrived.csv", ["flock"], Unmatched.Drop)     // each planned row beside the row of its arrival; a planned row with none is dropped
    .Declare(schema => schema.Id("flock", ColumnKind.Integer).Number("planned", "arrived"))
    .SplitAtRandom(0.70, 0.15)
    .Target("arrived")
    .Normalise("planned")
    .Build()
    .Run();

Console.WriteLine($"{prepared.CountIn(Part.Train)} training rows");

An id column.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema.Id("flock", ColumnKind.Integer).Number("age", "feed", "weight"))   // the id travels with its row and is no feature
    .SplitAtRandom(0.70, 0.15)
    .Target("weight")
    .Normalise("age", "feed")
    .WithTorch(network => network.Dense(8).Relu().Dense(1).Adam(0.01).MeanSquaredError().Run(seed: 1, epochs: 20))
    .Build()
    .Train();

var predictions = trained.Predict(new CsvRowSource("tomorrow.csv"));

for (var row = 0; row < predictions.Answers.Count; row++)
{
    Console.WriteLine($"flock {predictions.Ids?[row]}: {predictions.Answers[row][0]:0.0}");     // an answer finds its flock by the id
}

feature.cycle.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("bikes.csv")
    .Declare(schema => schema.Timestamp("dteday").Number("day", "phase", "cnt"))
    .Cycle("day", 7)                                                     // a number on a circle of a length you declare
    .Cycle(cycle => cycle.Every(30).Of("phase"))                         // or several columns at once
    .Cyclical(cyclical => cyclical.DayOfYear("dteday").Season("dteday")) // the two periods a moment gained
    .SplitByTime("dteday", train: 0.70, validation: 0.15)
    .Target("cnt")
    .Drop("dteday")
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", prepared.Batch(Part.Train).FeatureNames));

testSeed on split.atRandom.

using DeepSharp.Pipelines;

Pipeline Deal(int seed) => Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitAtRandom(0.70, 0.15, seed: seed, testSeed: 7)         // the test rows are dealt by testSeed alone
    .Target("survived")
    .Normalise("fare", "sibsp", "parch")
    .Build();

var first = Deal(1).Run();                                      // another deal of train and validation...
var second = Deal(2).Run();                                     // ...and the same test rows

Console.WriteLine($"{first.CountIn(Part.Test)} and {second.CountIn(Part.Test)} test rows");

Study and Comparison.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

Pipeline PipelineFor(int deal) => Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitAtRandom(0.70, 0.15, seed: 100 + deal, testSeed: 7)       // another deal of the rows each time, the same test rows
    .Target("survived")
    .Normalise("fare", "sibsp", "parch")
    .Build();

var space = new SearchSpace([new LogRange("rate", 1e-3, 1e-1), new WholeRange("units", 4, 32)]);
var study = new Study(space, (candidate, network) => network
    .Dense(candidate.Whole("units")).Relu().Dense(1).Adam(candidate.Number("rate")).BinaryCrossEntropy().Run(seed: 1, epochs: 10))
{ Trials = 12, Deals = 3, Sampler = new RandomSampler(11) };

var found = study.Run(PipelineFor);                                  // chosen by the validation rows; the test rows are the winner's alone
var again = PipelineFor(0).Run().Train(found.Best.Declared);         // a step trains on rows prepared once

Console.WriteLine($"trial {found.Best.Number} did best ({found.Best.Score:0.000} ± {found.Best.Spread:0.000}), trained again for {again.History!.Epochs.Count} epochs");

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

Pipeline Titanic(int deal, bool stratified)
{
    var declared = Pdd.Create().ReadCsv("titanic.csv").Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"));
    var split = stratified ? declared.SplitStratified("survived", 0.70, 0.15, seed: 100 + deal) : declared.SplitAtRandom(0.70, 0.15, seed: 100 + deal);

    return split.Target("survived").Normalise("fare", "sibsp", "parch").Build();
}

var comparison = new Comparison(network => network.Dense(8).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 1, epochs: 10)) { Deals = 10 };
var effect = comparison.Run(deal => Titanic(deal, stratified: false), deal => Titanic(deal, stratified: true));

Console.WriteLine($"{effect.Mean:0.0000} ± {effect.Spread:0.0000} over {effect.Deals} deals");   // a change read against its own spread

A correlation worked out once.

using DeepSharp.Charts;
using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Optional("age", ColumnKind.Number).Number("fare"))
    .Correlation(["age", "fare", "sibsp"], Shown.Numbers, Coefficient.Spearman)
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build()
    .Run();

var matrix = prepared.Evidence.Values.OfType<CorrelationInput>().Single().Correlate();   // worked out once: Pearson's and Spearman's

Console.WriteLine($"age and fare: {matrix.Pearson["age", "fare"]:0.000} (Pearson), {matrix.Spearman["age", "fare"]:0.000} (Spearman)");
File.WriteAllText("correlation.svg", matrix.Heatmap(Coefficient.Spearman));

ITensorBackend has one more member, FirstLargest.

using DeepSharp.Tensors;

var backend = new CpuBackend();
var scores = Tensor.From(new Shape(2, 3), [1f, 5f, 5f, 7f, 2f, 7f]);

var picked = backend.FirstLargest(scores);                      // a one at the first largest value of each row, noughts elsewhere

Console.WriteLine(string.Join(", ", picked.Values.ToArray()));

0.8.0

New

In the core, byte[].Fingerprint() and three ways to write a file whole.

using System.Text;
using DeepSharp.Pipelines;

byte[] bytes = Encoding.UTF8.GetBytes("open,close\n41800,42000\n");

Console.WriteLine(bytes.Fingerprint());                          // the SHA-256 in lower-case hexadecimal, the same on every machine

bytes.WriteWhole("prices.csv");                                  // whoever looks finds the old file or the whole of the new one
await bytes.WriteWholeAsync("prices-async.csv");                 // the same, without holding a thread
await bytes.WriteWholeOnceAsync("prices-once.csv");              // never replaces what stands; the same bytes standing are the same file

0.7.1

New

PipelineCourse, the steps of a prepared pipeline as a value.

using System.Text.Json.Nodes;
using DeepSharp.Pipelines;

var catalog = StepCatalog.BuiltIn();
var course = PipelineCourse.Table      // PipelineCourse.SeriesInTime is the one for rows that follow each other in time
    .Without("settle.gaps", "feature.add", "scale.given", "drop.columns", "fill.missing", "normalise", "learn.network")
    .Say("read.csv", new JsonObject { ["path"] = "titanic.csv" });

Console.WriteLine(string.Join('\n', course.Waiting(catalog)));      // Step 2, 'declare': waits for what only you can say: 'columns'. ...

course = course
    .Say("declare", JsonNode.Parse("""{"columns": [{"name": "survived", "kind": "integer", "optional": false}]}""")!.AsObject())
    .Say("split.stratified", new JsonObject { ["column"] = "survived" })
    .Say("target", new JsonObject { ["column"] = "survived" });

var prepared = Pdd.From(course, catalog).Run();      // refused, naming every step that still waits, until nothing does
Console.WriteLine(prepared.Table.RowCount);

What waits is what only you know.

using DeepSharp.Pipelines;

var catalog = StepCatalog.BuiltIn();
var step = CourseStep.Of("feature.add");

Console.WriteLine(step.Skeleton(catalog));                      // the step as a block starts, with null where only you can say
Console.WriteLine(string.Join(", ", step.Waiting(catalog)));    // column, left, right

Two courses, because the rules say one cannot teach both.

using DeepSharp.Pipelines;

foreach (var course in PipelineCourse.Named)
{
    Console.WriteLine(string.Join(" > ", course.Steps.Select(step => step.Verb)));
}

var split = PipelineCourse.SeriesInTime.Steps.Single(step => step.Verb == "split.byTime");

Console.WriteLine(split.Said);      // {"gap":1}: the series' course says the gap, which the table's leaves out

A course as a file of its own.

using System.Text.Json.Nodes;
using DeepSharp.Pipelines;

var catalog = StepCatalog.BuiltIn();
var course = PipelineCourse.SeriesInTime
    .Without("learn.network")
    .Say("read.csv", new JsonObject { ["path"] = "prices.csv" });

File.WriteAllText("prices.course.json", course.ToJson());

var read = PipelineCourse.FromJson(File.ReadAllText("prices.course.json"), catalog);

Console.WriteLine(read == course);      // True

0.7.0

New

learn.ml, and .WithML(…) beside .WithTorch(…) and .WithTensorflow(…).

using DeepSharp.Learners.ML;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing(fill => fill.Median("age"))
    .Normalise("age", "fare")                                // declared, so a network can be trained from the same lines
    .WithML(trainer => trainer.FastTree(trees: 50))          // the learner these rows are prepared for
    .Build();

var forATree = pipeline.RunFor(Needs.NoScale);               // what pipeline.TrainWithML() runs: the scalings are left out

Console.WriteLine(string.Join(", ", forATree.Skipped));      // and the run says which, by the place of each step

Boosted trees and a forest, and no other trainer.

using DeepSharp.Learners.ML;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .WithML(trainer => trainer.FastForest(trees: 200, leastRows: 5), seed: 7)   // the seed is written into the declaration
    .Build();

Console.WriteLine(pipeline.Declaration.ToJson());            // run this text again and the same forest is trained

A model and its pipeline as one file.

using DeepSharp.Learners.ML;

var file = MLModelFile.FromJson(File.ReadAllText("titanic.mlmodel.json"));   // reading a model needs no ML.NET

Console.WriteLine($"{file.Trainer.Kind} on {file.Rows} rows, written by ML.NET {file.Library} on {file.Processor}");
Console.WriteLine($"bound to the pipeline text with digest {file.Carried.Digest}");

scale.given, scaling from bounds you give, above the split.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Number("fare"))
    .ScaleGiven("fare", 0, 512)                              // bounds you know, above the split: nothing is learned
    .ScaleGiven(scale => scale.Between("sibsp", 0, 8))       // or one line for many columns
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Build()
    .Run();

Console.WriteLine(prepared.Table.NumbersOf("fare").Min());   // a fare of nothing lands on the low end of the range you gave

shuffle, before the rows are divided.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .Shuffle(seed: 7)                                        // an order drawn from the seed, before the rows are divided
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", prepared.Table.NumbersOf("fare").Take(3)));   // no longer the file's first three rows

0.6.2

New

settle.gaps, and .SettleGaps beside .DropGaps.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Optional("age", ColumnKind.Number).Optional("fare", ColumnKind.Number))
    .SettleGaps(gaps => gaps.Zero("age").Constant(-1, "fare"))   // Zero, Constant, Refuse: no row decides them
    .AddFeature("older", "age", Arithmetic.Plus, "sibsp")        // worked out from a settled column
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("age", "older", "fare")
    .Build()
    .Run();

Console.WriteLine(prepared.Table.NumbersOf("age_was_missing").Count(value => value == 1));   // the gaps stay on record

What a fit decided is read back as the choice it was.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median, refuseAbove: 0.1)
    .Build()
    .Run();

Console.WriteLine(prepared.Fitted[4].Choice("filled"));      // 0: more than a tenth of the ages were gaps, so none is filled

0.6.1

New

A scaling that names no kind now lands its rows between minus one and one.

using DeepSharp.Pipelines;

var declaration = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("sibsp")                                      // names no kind: NormaliseStep.DefaultScale
    .Normalise("fare", Scale.Standard)                       // the old meaning, named
    .Declaration;

Console.WriteLine($"{NormaliseStep.DefaultScale}: {declaration.ToJson()}");

One line does one thing to many columns, and the kind is the method.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Optional("age", ColumnKind.Number).Number("fare"))
    .Reshape(shape => shape.Log1P("fare"))                   // above the split: one line, one shape
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing(fill => fill.Median("age"))                 // below it: one line, one way to fill
    .Normalise(scale => scale
        .MidRange("age", "fare")                             // one kind for a group of columns
        .MaxAbs("sibsp"))
    .Build()
    .Run();

Console.WriteLine(prepared.Table.NumbersOf("fare").Max());

Where the features land is said once: .DefaultFeatures(Form).

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number).Number("fare"))
    .DefaultFeatures(Form.Unit)                              // said once: the features land between nothing and one
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing(fill => fill.Median("age"))
    .Normalise("age", "fare")                                // names no kind, so it is written as min-max
    .Build()
    .Run();

Console.WriteLine(prepared.Table.NumbersOf("fare").Min());

0.6.0

New

A pipeline names the network it prepares its rows for: .WithTorch(…) and .WithTensorflow(…).

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .WithTorch(network => network                     // PyTorch's words; .WithTensorflow(…) says the same in Keras's
        .Dense(16).Relu().Dense(1)
        .Adam(0.01)
        .BinaryCrossEntropy()
        .Run(seed: 20260929, epochs: 100)
        .StoppingAfter(patience: 10))
    .Build();

File.WriteAllText("titanic.declaration.json", pipeline.Declaration.ToJson());   // the file says which network these rows train

pipeline.Train() trains what the declaration names.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .WithTensorflow(network => network.Dense(16).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20260929, epochs: 100))
    .Build();

var trained = pipeline.Train();                  // runs the pipeline for the network it names, trains it and measures it as its report declares
var again = pipeline.Run().Train();              // the same, for a run already in hand

Console.WriteLine(trained.ToJson() == again.ToJson());   // the same network, number for number

The engine is named in the declaration, and the application says what that name stands for.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;
using DeepSharp.Tensors;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .WithTorch(network => network.Dense(16).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20260929, epochs: 100).On("mine"))
    .Build();

// The declaration names an engine; the application says what the name stands for (TorchBackend.OnCpu(), where libtorch is installed).
var trained = pipeline.Train(new Engines().Use("mine", new CpuBackend()));

Console.WriteLine(trained.History!.BestEpoch);

learn.network, the verb a pipeline file writes it as.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

// A pipeline file that names its network is read by a catalog taught the verb that package brings.
var declaration = PipelineDeclaration.FromJson(File.ReadAllText("titanic.declaration.json"), StepCatalog.BuiltIn().WithNetworks());

Console.WriteLine(declaration.Learner?.Verb);         // learn.network

A declaration may name the learner it is written for: INamesTheLearner.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("fare")
    .WithTensorflow(network => network.Dense(8).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 7, epochs: 20))
    .Build();

if (pipeline.Declaration.Learner is { } learner)
{
    Console.WriteLine($"{learner.Verb} needs {learner.Needs}");        // learn.network needs OneScale
}

PartsParameter, the kind of parameter that holds the parts of a model.

using System.Text.Json;
using DeepSharp.Pipelines;

var layers = new PartsParameter(
    "layers",
    "The layers, first to last.",
    [new PartDeclaration("dense", [new PartSetting("units", PartValue.Of(16))])],
    [
        new PartKind("dense", "A layer that answers with so many numbers.", [new WholeNumberParameter("units", "How many numbers.", 16)]),
        new PartKind("relu", "What is below nought left at nought.", []),
    ]);

using var step = JsonDocument.Parse("""{ "layers": [ { "kind": "dense", "units": 8 }, { "kind": "relu" } ] }""");

foreach (var part in layers.Read(step.RootElement))
{
    Console.WriteLine($"{part.Kind}: {string.Join(", ", part.Settings.Select(setting => $"{setting.Key} = {setting.Value.Number}"))}");
}

0.5.0

New

Needs.Numbers is the need of a learner that takes every step as declared; a tree states Needs.NoScale.

using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .Build();

var forATree = pipeline.RunFor(Needs.NoScale);                  // the scalings a tree does without are left out
var train = forATree.Batch(Part.Train, Needs.NoScale);

Console.WriteLine($"{forATree.Skipped.Count} steps left out; {train.RowCount} rows of {train.Width} numbers");

Test a need by what it says.

using DeepSharp.Pipelines;

foreach (var need in Enum.GetValues<Needs>())
{
    Console.WriteLine($"{need}: one scale {need.NeedsOneScale()}, without scaling {need.DoesWithoutScaling()}, takes categories {need.TakesCategories()}");
}

A trained network serves on the engine you hand it.

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;
using DeepSharp.Tensors;

var trained = TrainedNetwork.FromJson(File.ReadAllText("titanic.network.json"), NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn().WithNetworks());
var passengers = new InMemoryRowSource(["pclass", "sex", "age", "sibsp", "parch", "fare"], [["3", "male", "22", "1", "0", "7.25"]]);

// The engine is the caller's, for this call alone: a CpuBackend here, TorchBackend.OnGpu(0) where a card is.
var chances = trained.Predict(passengers, new CpuBackend());

Console.WriteLine($"{chances.AnswerNames[0]}: {chances.Answers[0][0]:0.000}");

An engine of your own keeps a tensor's values where it keeps them.

using DeepSharp.Tensors;

// An engine that keeps its values somewhere of its own hands back what it makes as a tensor standing on that storage.
var tensor = Tensor.On(new Shape(2, 2), new KeptElsewhere([1f, 2f, 3f, 4f]));

Console.WriteLine(tensor.Storage is KeptElsewhere);
Console.WriteLine(tensor.Values.Length);              // copied into this machine's memory the first time they are read, once

sealed class KeptElsewhere(float[] kept) : TensorStorage
{
    public override int Count => kept.Length;

    public override void CopyTo(Span<float> destination) => kept.CopyTo(destination);
}

What the seam refuses is public, and every engine says it first.

using DeepSharp.Tensors;

var left = Tensor.From(new Shape(2, 2), [1f, 2f, 3f, 4f]);
var right = Tensor.From(new Shape(3), [1f, 2f, 3f]);

try
{
    left.RequireSameShape(right, "Add");              // what the light engine asks first of Add; an engine of your own asks it too
}
catch (ArgumentException refused)
{
    Console.WriteLine(refused.Message);                // in the light engine's words, naming the same argument
}

An engine names its version and the device it works on: INamesItsVersionAndDevice.

using DeepSharp.Tensors;

ITensorBackend engine = new CpuBackend();

if (engine is INamesItsVersionAndDevice named)
{
    Console.WriteLine($"{engine.Name} {named.Version} on {named.Device}");   // what a checkpoint records of the engine its run was on
}

A served row that moves a feature every training row held at one value is named, and the report counts such rows.

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var trained = TrainedNetwork.FromJson(File.ReadAllText("titanic.network.json"), NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn().WithNetworks());

foreach (var (feature, value) in trained.TrainedOn.Unvaried ?? new Dictionary<string, double>())
{
    Console.WriteLine($"{feature} held {value} on every training row: nothing was learned about it");
}

var passengers = new InMemoryRowSource(
    ["pclass", "sex", "age", "sibsp", "parch", "fare"],
    [["3", "male", "22", "1", "0", "7.25"], ["3", "Male", "22", "1", "0", "7.25"]]);

if (trained.Predict(passengers).Unfamiliar is { } unfamiliar)
{
    Console.WriteLine(string.Join(" | ", unfamiliar.Select(names => string.Join(", ", names))));   // nothing for the first, sex_other for the second
}

A notebook's report block draws the measures of a model a C# cell trained.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .Report(report => report.Measure(Metric.Accuracy).On(Part.Test).As(Shown.Numbers))
    .WithTensorflow(network => network.Dense(16).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20260929, epochs: 100))
    .Build();

var text = pipeline.Train().Measures!.PredictionsToJson();     // what the report measured, as text that can cross to another process
var measured = pipeline.Run().MeasureAgain(text);               // measured again, by the report, on another run of the same pipeline

Console.WriteLine(measured.Parts[^1].Values[0].Value);

PipelineText, in DeepSharp.Pipelines.

using DeepSharp.Pipelines;

var before = PipelineText.Of(File.ReadAllBytes("titanic.pipeline.json"));
var after = PipelineText.Of(File.ReadAllBytes("titanic.fitted-again.pipeline.json"));

Console.WriteLine($"{before.Digest} (version {before.Version}, comparable: {PipelineText.IsComparable(before.Version)})");
Console.WriteLine(before.IsTheFitOf(after) ? "one fit" : "another fit: other steps, or other numbers learned");

The report, rendered once.

using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")
    .Report(report => report.Measure(Metric.Accuracy, Metric.ConfusionMatrix).On(Part.Train, Part.Test).As(Shown.Numbers, Shown.Drawn))
    .WithTensorflow(network => network.Dense(16).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 20260929, epochs: 100))
    .Build()
    .Train();

File.WriteAllText("titanic-report.html", trained.Measures!.Report().ToHtml());   // the report as its declaration says it is shown
File.WriteAllText("titanic-loss.svg", trained.History!.LossCurve());               // a cell that ends with a chart shows the picture

A pipeline is run for the learner that learns from it, leaving out the steps that learner does without and saying which.

using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")      // a network takes them between -1 and 1, which is the default
    .Build();

var forATree = pipeline.RunFor(Needs.Categories);
var left = forATree.Skipped;                              // the places of encode.categories and the four normalise steps
var train = forATree.Batch(Part.Train, Needs.Categories); // nine numbers a passenger, with their categories as places

Console.WriteLine($"{left.Count} steps left out; {train.RowCount} rows of {train.Width} numbers");

Parquet files, read.

using DeepSharp.Pipelines;

PipelineBuilder[] doors =
[
    Pdd.Create().ReadCsv("titanic.csv"),
    Pdd.Create().ReadParquet("titanic.parquet"),
    Pdd.Create().ReadExcel("titanic.xlsx", "titanic"),     // the first sheet unless one is named; .xls and .xlsb too
    Pdd.Create().ReadJson("titanic.json"),                  // an array of records, one object a row
];

foreach (var door in doors)
{
    var prepared = door
        .Declare(schema => schema.Integer("survived").Number("fare"))
        .SplitStratified("survived", train: 0.70, validation: 0.15)
        .Build()
        .Run();

    Console.WriteLine(prepared.Parts.Count(part => part == Part.Train));      // the same split through every door
}

A notebook reads a Parquet file, an Excel workbook and a JSON file.

using DeepSharp.Pipelines;

// A cell that reads a pipeline the notebook's blocks hand over teaches its catalog every verb they use.
var catalog = StepCatalog.BuiltIn().WithIndicators().WithParquet().WithExcel().WithJson();
var prepared = PreparedData.FromJson(File.ReadAllText("passengers.pipeline.json"), catalog);

Console.WriteLine(prepared.Declaration.Steps.Count);

One evaluation for every door, in the core: network.Predict(features, loss, engine).

using DeepSharp.Networks;
using DeepSharp.Tensors;

var network = new Sequential().Dense(16).Relu().Dense(1).Lower(new Shape(14), new RandomStream(7));
var passengers = Tensor.From(new Shape(2, 14), new float[28]);

var chances = network.Predict(passengers, new BinaryCrossEntropy(), new CpuBackend());   // one evaluation pass, through the loss's sigmoid

Console.WriteLine(chances.Shape);

Numbers trained somewhere else go into a network by the path of each slot, and IImporter is the seam that reads them.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var stream = new RandomStream(42);
var network = new LayerStack(new Dense(2, 2, stream.Draw("initialise:0", 0, 0)), new BatchNorm(2));

network.Load(
[
    new SlotEntry("0.weight", Tensor.From(new Shape(2, 2), [1f, 0.5f, -0.5f, 2f]), "line 1"),
    new SlotEntry("0.bias", Tensor.From(new Shape(2), [0.25f, -0.25f]), "line 2"),
    new SlotEntry("1.weight", Tensor.From(new Shape(2), [1.5f, 0.5f]), "line 3"),
    new SlotEntry("1.bias", Tensor.From(new Shape(2), [0.125f, -0.125f]), "line 4"),
    new SlotEntry("1.running_mean", Tensor.From(new Shape(2), [0.5f, -1f]), "line 5"),
    new SlotEntry("1.running_var", Tensor.From(new Shape(2), [4f, 0.25f]), "line 6"),
]);

Models Keras saved, read: DeepSharp.Import.Keras.

using DeepSharp.Import.Keras;

using var file = File.OpenRead("titanic.keras");
var saved = new KerasFile().Read(file);

Console.WriteLine($"{saved.Network.Slots().Count()} slots hold Keras's numbers, answering through {saved.Loss.GetType().Name}");

Networks exported to ONNX, read: DeepSharp.Import.Onnx.

using DeepSharp.Import.Onnx;
using DeepSharp.Networks;

using var graph = File.OpenRead("titanic.onnx");
var saved = new OnnxFile(new BinaryCrossEntropy()).Read(graph);    // a graph names no loss, so it is handed one

Console.WriteLine($"{saved.Network.Slots().Count()} slots hold the graph's numbers");

A window pads as TensorFlow's 'same' does.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var same = new Window(3, 3) { Stride = 2, PaddingMode = PaddingMode.Same };
var border = same.BordersOver(28, 28);

Console.WriteLine($"{border.Top} above, {border.Bottom} below, {border.Left} before, {border.Right} after");   // none before, one after
Console.WriteLine($"{same.RowsOver(28)} by {same.ColumnsOver(28)} places");                                    // 14 by 14

Keras's words take Keras's settings for the normalisations.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var description = new Sequential()
    .Input(new Shape(28, 28, 1))
    .Conv2D(8, new Window(3, 3) { Stride = 2, PaddingMode = PaddingMode.Same })   // padded as TensorFlow's 'same'
    .BatchNorm(momentum: 0.99, epsilon: 0.001)                                    // Keras's momentum, kept as its complement
    .Relu()
    .Flatten()
    .LayerNorm(epsilon: 0.001)
    .Dense(1);

var network = description.Lower(new Shape(28, 28, 1), new RandomStream(7));

Console.WriteLine(string.Join(", ", network.Slots().Select(named => named.Path)));

A network PyTorch trained runs here: DeepSharp.Import.PyTorch reads a safetensors file into the same network.

using DeepSharp.Import.PyTorch;
using DeepSharp.Networks;
using DeepSharp.Tensors;

var titanic = new Sequential().Dense(16).Relu().Dense(1).Lower(new Shape(14), new RandomStream(7));

using var safetensors = File.OpenRead("titanic.safetensors");
var saved = new SafetensorsFile(titanic, new BinaryCrossEntropy()).Read(safetensors);

Console.WriteLine($"{saved.Network.Slots().Count()} slots hold PyTorch's numbers");

…and the file torch.save(model.state_dict(), file) writes: new TorchSaveFile(network, loss).Read(stream).

using DeepSharp.Import.PyTorch;
using DeepSharp.Networks;
using DeepSharp.Tensors;

var images = new Sequential().Conv2D(4, new Window(3, 3)).Relu().Flatten().Dense(1)
    .Lower(new Shape(28, 28, 1), new RandomStream(7));

using var state = File.OpenRead("images.pt");                       // torch.save(model.state_dict(), "images.pt")
var read = new TorchSaveFile(images, new MeanSquaredError()) { Example = new Shape(28, 28, 1) }.Read(state);

Console.WriteLine($"{read.Network.Slots().Count()} slots hold PyTorch's numbers");

A file's text is shown in a message as words and nothing else: text.Quoted().

using DeepSharp.Networks;

Console.WriteLine("0.weight\nline two‮of a name".Quoted());     // one line, nothing hidden: the line break and the override are written as their escapes

A layer lists the layers it holds: layer.HeldLayers().

using DeepSharp.Networks;
using DeepSharp.Tensors;

var network = new Sequential().Dense(16).Relu().Dense(1).Lower(new Shape(14), new RandomStream(7));

foreach (var (path, layer) in network.HeldLayers())
{
    Console.WriteLine($"{path}: {layer.GetType().Name}");
}

A checkpoint read from one reading of its file: NetworkDocument.ReadCheckpoint(json, network, training, catalog).

using DeepSharp.Networks;

// A file of your own that keeps the network under "model" and what the run needs to go on under "run".
var saved = NetworkDocument.ReadCheckpoint(File.ReadAllText("my-run.json"), "model", "run", NetworkCatalog.BuiltIn());

var options = new FitOptions(seed: 7) { Epochs = 100, ResumeFrom = saved.Run.Checkpoint };   // to go on with saved.Run.Compiled.Fit(…)
Console.WriteLine($"{saved.Network.Network.Slots().Count()} slots, going on to epoch {options.Epochs}");

A model trained somewhere else is trained further behind a pipeline, and kept with it as the one file.

using DeepSharp.Import.Keras;
using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Category("pclass", "sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "sibsp", "parch")      // a network takes them between -1 and 1, which is the default
    .Build()
    .Run();

using var file = File.OpenRead("titanic.keras");
var saved = new KerasFile().Read(file);

var trained = saved.Network.Compile(new Adam(0.001), saved.Loss).Fit(prepared, new FitOptions(seed: 7) { Epochs = 1 });
File.WriteAllText("titanic.network.json", trained.ToJson());

0.4.0

New

A batch says which part it came from and which row each is.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Build()
    .Run();

var batch = prepared.Batch(Part.Validation);

Console.WriteLine($"{batch.Part}: {batch.RowCount} rows, the first is {batch.Keys![0]}");   // which part, and which row each is

A step that writes a column says where its values land.

using DeepSharp.Pipelines;

// What a step's After(before) hands back: the columns it leaves, and where the ones it writes land their values.
var before = ColumnState.None.With("age", ColumnKind.Number);                 // lands nowhere said
var after = before
    .With("age_unit", ColumnKind.Number, Form.Unit)                           // lands between nothing and one
    .WithFamily("route_", Form.Signed);                                       // a family an encoder makes once it has seen the rows

Console.WriteLine($"{after.LandsOf("age_unit")} {after.LandsOf("route_c")} {after.LandsOf("age") is null}");

A file read through the data frame keeps its own text.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsvFrame("titanic.csv")          // read through the data frame's reader, every column as the file writes it
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Build()
    .Run();

Console.WriteLine(prepared.Table.RowCount);   // the rows ReadCsv reads of the same file, known by the same keys

Layers.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var stream = new RandomStream(42);
var network = new LayerStack(
    new Conv2D(1, 4, new Window(3, 3), stream.Draw("initialise:0", 0, 0)),   // a window over each image, channels last
    new Relu(),
    new Flatten(),
    new BatchNorm(64),
    new Dropout(0.2),
    new Dense(64, 1, stream.Draw("initialise:5", 0, 0)));

foreach (var named in network.Slots())
{
    Console.WriteLine($"{named.Path}: {named.Slot.GetType().Name}");   // 0.weight: Parameter ... 3.running_mean: RunningStatistic
}

var images = new CpuBackend().Fill(new Shape(2, 6, 6, 1), 0.5f);
var answers = network.Forward(images, Pass.Evaluation(new CpuBackend()));   // a pass says whether it trains, so no layer carries a mode

Console.WriteLine(answers);

Losses that say what a network's numbers mean.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var maths = new CpuBackend();
var logits = Tensor.From(new Shape(2, 1), [0f, 2f]);               // the network's raw numbers
var answers = Tensor.From(new Shape(2, 1), [1f, 0f]);

var loss = new BinaryCrossEntropy();
var chances = loss.Predictions(logits, maths);                     // through the sigmoid the loss applies itself
var mean = loss.Of(logits, answers, maths);                        // exact at a logit of nought
var refused = new CrossEntropy().Refusal([0.7, 0.7]);              // shares that do not add up to one: refused, with the reason

Console.WriteLine($"{chances.Values[0]} {mean.Values[0]} {refused}");

Optimizers and learning-rate schedules.

using DeepSharp.Networks;

var description = new Sequential().Dense(16).Relu().Dense(1);

var plain = description.Compile(new Sgd(0.05) { Momentum = 0.9 }, new MeanSquaredError());                       // momentum, as PyTorch's SGD has it
var decaying = description.Compile(new Adam(0.01), new MeanSquaredError(), new CosineDecay(epochs: 50));         // the rate worked out from the epoch

LearningRateSchedule[] schedules = [new ConstantRate(), new StepDecay(every: 10, factor: 0.5), new ExponentialDecay(0.95), new CosineDecay(epochs: 50)];

foreach (var schedule in schedules)
{
    Console.WriteLine($"{schedule.GetType().Name}: {schedule.RateAt(epoch: 25, initial: 0.01)}");   // asked once an epoch
}

Console.WriteLine($"{plain.Optimizer.Rate} {decaying.Schedule.GetType().Name}");

A training loop.

using DeepSharp.Networks;
using DeepSharp.Tensors;

static TrainingData Line(int from, int count)
{
    float[] x = [.. Enumerable.Range(from, count).Select(row => row / 10f)];

    return new TrainingData(Tensor.From(new Shape(count, 1), x), Tensor.From(new Shape(count, 1), [.. x.Select(value => (2 * value) + 1)]));
}

var compiled = new Sequential().Dense(8).Relu().Dense(1).Compile(new Adam(0.01), new MeanSquaredError());
var early = new EarlyStopping { Patience = 10, RestoreBest = true };
Checkpoint? last = null;

var history = compiled.Fit(Line(0, 40), Line(40, 10), new FitOptions(seed: 42)
{
    Epochs = 20,
    EarlyStopping = early,
    Checkpoints = new Checkpoints(checkpoint => last = checkpoint),    // taken at the end of every epoch
});

var more = compiled.Fit(Line(0, 40), Line(40, 10), new FitOptions(seed: 42) { Epochs = 100, EarlyStopping = early, ResumeFrom = last });   // goes on bit for bit

Console.WriteLine($"{history.Epochs.Count} epochs, then {more.Epochs.Count}: {more.Stopped}, best was {more.BestEpoch}");

Every draw counted from one seed.

using DeepSharp.Networks;

var stream = new RandomStream(42);

var first = stream.Draw("initialise:0", epoch: 0, step: 0);
var again = stream.Draw("initialise:0", epoch: 0, step: 0);              // the same purpose, the same numbers: nothing was carried forward
var order = stream.Draw("shuffle", epoch: 3, step: 0).Permutation(8);    // the order of epoch 3's rows, worked out afresh

Console.WriteLine($"{first.NextSingle() == again.NextSingle()}: {string.Join(' ', order)}");

A network in Keras's words.

using DeepSharp.Networks;
using DeepSharp.Tensors;

var description = new Sequential().Input(new Shape(14)).Dense(16).Relu().Dense(1);   // every width is worked out from the example's shape

var byHand = description.Lower(new Shape(14), new RandomStream(42));                  // the stack code would write, number for number
var compiled = description.Compile(new Adam(0.01), new BinaryCrossEntropy());         // or built at its first Fit, from the rows and the seed

Console.WriteLine($"{byHand.Slots().Count()} slots, {compiled.Loss.GetType().Name}");

A network written down.

using System.Text;
using System.Text.Json;
using DeepSharp.Networks;
using DeepSharp.Tensors;

var network = new Sequential().Dense(16).Relu().Dense(1).Lower(new Shape(14), new RandomStream(42));

using var buffer = new MemoryStream();
using (var writer = new Utf8JsonWriter(buffer, new JsonWriterOptions { Indented = true }))
{
    writer.WriteStartObject();
    writer.WritePropertyName("network");
    NetworkDocument.WriteNetwork(writer, network, new BinaryCrossEntropy());   // its kinds, every slot's numbers, its loss
    writer.WriteEndObject();
}

var saved = NetworkDocument.ReadNetwork(Encoding.UTF8.GetString(buffer.ToArray()), "network", NetworkCatalog.BuiltIn());

Console.WriteLine($"{saved.Network.Slots().Count()} slots read back to the last bit, loss {saved.Loss.GetType().Name}");

A network trained behind a pipeline.

using DeepSharp.Learners.Networks;
using DeepSharp.Networks;
using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("sibsp", "parch", "fare")
    .Build()
    .Run();

var compiled = new Sequential().Dense(8).Relu().Dense(1).Compile(new Adam(0.01), new BinaryCrossEntropy());
var trained = compiled.Fit(prepared, new FitOptions(seed: 7) { Epochs = 30 });   // judged by the validation rows, never shown the test rows

var served = new InMemoryRowSource(["sibsp", "parch", "fare"], [["1", "0", "7.25"]]);
var chance = trained.Predict(served).Answers[0][0];                               // in the answer's own units

File.WriteAllText("titanic.network.json", trained.ToJson());                      // the network and its pipeline, one file
var again = TrainedNetwork.FromJson(File.ReadAllText("titanic.network.json"), NetworkCatalog.BuiltIn(), StepCatalog.BuiltIn());

Console.WriteLine($"{chance:0.00} {again.Predict(served).Answers[0][0]:0.00}");

A report of what a trained model is measured by.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("sibsp", "parch", "fare")
    .Report(report => report.Measure(Metric.Accuracy, Metric.Recall).On(Part.Validation, Part.Test).As(Shown.Numbers, Shown.Drawn))   // declared before anything learns
    .WithTensorflow(network => network.Dense(8).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 7, epochs: 30))
    .Build()
    .Train();

foreach (var measured in trained.Measures!.Parts[^1].Values)      // the test rows' measures, last
{
    Console.WriteLine($"{measured.Metric}: {measured.Value:0.000}, predicting the training rows' average gives {measured.Baseline:0.000}");
}

Charts.

using DeepSharp.Charts;
using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

var trained = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("sibsp", "fare")
    .Report(report => report.Measure(Metric.Accuracy, Metric.ConfusionMatrix).On(Part.Test).As(Shown.Drawn))
    .WithTensorflow(network => network.Dense(8).Relu().Dense(1).Adam(0.01).BinaryCrossEntropy().Run(seed: 7, epochs: 30))
    .Build()
    .Train();

var history = trained.History!;
var measures = trained.Measures!;

File.WriteAllText("loss.svg", history.LossCurve());                 // the training loss of every epoch, the validation loss beside it
File.WriteAllText("rate.svg", history.LearningRates());
File.WriteAllText("bars.svg", measures.Bars());                      // every measure, each part's beside the average's
File.WriteAllText("confusion.svg", measures.ConfusionMatrices());   // a heatmap of counts for each confusion matrix

A pipeline read from inside a larger file.

using DeepSharp.Learners.Networks;
using DeepSharp.Pipelines;

// A trained network's file holds the pipeline it was trained behind under its "pipeline" key.
var prepared = PreparedData.FromJson(File.ReadAllText("titanic.network.json"), StepCatalog.BuiltIn().WithNetworks(), "pipeline");

Console.WriteLine(prepared.Declaration.Steps.Count);

The word a file writes a choice as.

using DeepSharp.Pipelines;

var written = new[] { Part.Validation.Word(), Metric.ConfusionMatrix.Word(), Shown.Drawn.Word() };

Console.WriteLine(string.Join(' ', written));   // as a pipeline's file writes them: validation confusionmatrix drawn

The operations a backward pass needs.

using DeepSharp.Tensors;

var maths = new CpuBackend();

var rows = Tensor.From(new Shape(2, 3), [1f, 2f, 3f, 4f, 5f, 6f]);
var weights = Tensor.From(new Shape(3, 1), [0.5f, -0.5f, 0.25f]);
var bias = Tensor.From(new Shape(1), [0.1f]);

var outputs = maths.AddRow(maths.MatMul(rows, weights), bias);        // a bias reaches every example of a batch
var loss = maths.Mean(maths.Subtract(outputs, maths.Fill(outputs.Shape, 1f)));   // one value with no axes, as a loss is
var halved = maths.Scale(rows, maths.Fill(new Shape(), 0.5f));        // one value times every value
var totals = maths.SumRows(maths.Transpose(rows));                    // turned round, then added up: 6 and 15

var images = maths.Fill(new Shape(1, 4, 4, 1), 1f);                    // image by row by column by channel
var patches = maths.Unfold(images, new Window(2, 2) { Stride = 2 });   // a row for every place the window stands

Console.WriteLine($"{outputs} {loss.Values[0]} {halved} {totals} {patches}");

Gradients, worked out automatically.

using DeepSharp.Tensors;

var maths = new CpuBackend();
var pass = new RecordingBackend(maths);

var rows = Tensor.From(new Shape(2, 2), [0.80f, -0.57f, 0.79f, -0.67f]);
var weights = Tensor.From(new Shape(2, 1), [0.1f, -0.2f]);
var bias = Tensor.From(new Shape(1), [0f]);
var wanted = Tensor.From(new Shape(2, 1), [0f, 0.01f]);

var misses = pass.Subtract(pass.AddRow(pass.MatMul(rows, weights), bias), wanted);
var loss = pass.Mean(pass.Multiply(misses, misses));

var gradients = pass.GradientsOf(loss, [weights, bias]);
Console.WriteLine(gradients[weights]);   // Tensor 2x1: how much each weight moves the loss

A timestamp column says how its moments are written.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("flocks.csv")
    .Declare(schema => schema
        .Column(new ColumnDeclaration("when", ColumnKind.Timestamp, Optional: false) { Format = "dd/MM/yyyy" })   // 02/03/2015 is the second of March
        .Number("weight"))
    .SplitAtRandom(0.70, 0.15)
    .Target("weight")
    .Build()
    .Run();

File.WriteAllText("flocks.pipeline.json", prepared.ToJson());   // the format is in the file: "format": "dd/MM/yyyy"

A learner that takes every feature on one scale says so where they are handed over.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "sibsp", "parch").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Normalise("sibsp", "parch")                  // fare is left on its own scale
    .Build()
    .Run();

try
{
    prepared.Batch(Part.Train, Needs.OneScale);   // a network takes every feature between -1 and 1, and says so
}
catch (InvalidOperationException refused)
{
    Console.WriteLine(refused.Message);           // names fare, which is declared to land nowhere
}

A quantile bound at nothing is the training extremes.

using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .ClipOutliers("AAPL.Close", Bounds.Quantile, at: 0, outlier: Outlier.Refuse)   // at nothing: the smallest and largest close the training rows hold
    .Build();

try
{
    pipeline.Run();
}
catch (InvalidOperationException beyond)
{
    Console.WriteLine(beyond.Message);   // a later row beyond anything the fit saw stops the run, and says how far out it was
}

Features between minus one and one.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close", "AAPL.Volume"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Normalise(scale => scale
        .MidRange(OutOfRange.Clip, "AAPL.Close")   // the training rows land between -1 and 1; a later close is held at the edge
        .Robust("AAPL.Volume"))
    .Build()
    .Run();

Console.WriteLine($"{Scale.MidRange.Lands()} {prepared.Table.RowCount}");   // Signed: between minus one and one

What should not be there, found by the profile.

using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass").Category("alive").Number("fare"))
    .Profile()
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .Build();

var profile = pipeline.Run().Evidence.Values.OfType<DataProfile>().Single();

foreach (var alert in profile.Alerts)
{
    var answered = alert.Answer.AppliedTo(pipeline.Declaration);   // the steps with the column left out, or the gap said on the schema
    Console.WriteLine($"{string.Join(" ~ ", alert.Columns)}: {alert.Answer.Action}, {answered.Count} steps");
}

A value that stands for a gap, said on the column.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema
        .Integer("survived")
        .Column(new ColumnDeclaration("fare", ColumnKind.Number, Optional: false) { Missing = "0" }))   // a fare of 0 is a fare nobody wrote down
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing(fill => fill.Median("fare"))                                                          // counts the zeros as the gaps they are
    .Build()
    .Run();

File.WriteAllText("titanic.pipeline.json", prepared.ToJson());

Each column's kind, proposed from every cell.

using DeepSharp.Pipelines;

var proposal = KindProposal.Of(new CsvRowSource("titanic.csv"));   // every cell of every row read

foreach (var column in proposal.Columns)
{
    Console.WriteLine($"{column.Name}: {column.Kind} ({column.Gaps} gaps, {column.Distinct} different values)");
}

Console.WriteLine(proposal["pclass"].Offered);   // Category, offered beside the whole numbers it proposes: a class and a count look alike

A source can say what its columns hold.

using DeepSharp.Pipelines;

var proposal = KindProposal.Of(new Coded());

Console.WriteLine(proposal["zip"].Kind);   // Text: what the source states outranks what the cells look like

sealed class Coded : IStatesKinds
{
    public IReadOnlyList<string> ColumnNames { get; } = ["zip"];

    public IEnumerable<IReadOnlyList<string?>> Rows { get; } = [["10115"], ["80331"], ["50667"]];

    public IReadOnlyDictionary<string, ColumnKind> StatedKinds { get; } = new Dictionary<string, ColumnKind> { ["zip"] = ColumnKind.Text };
}

0.3.0

New

A file is read with a catalog.

using DeepSharp.Pipelines;

var catalog = StepCatalog.BuiltIn().WithIndicators();   // the verbs this reader knows; indicators are taught with the call
var text = File.ReadAllText("btceur.pipeline.json");

var declared = PipelineDeclaration.FromJson(text, catalog);   // the person's half alone
var prepared = PreparedData.FromJson(text, catalog);          // both halves, anywhere, no data needed

Console.WriteLine(prepared.Declaration == declared);

The file names its version, and what was fitted names the steps it was fitted behind.

using DeepSharp.Pipelines;

static PipelineDeclaration Written(FillStrategy fill) => Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", fill)
    .Normalise("fare")
    .Declaration;

var median = Written(With.Median);
var mean = Written(With.Mean);

Console.WriteLine($"written as version {PipelineDeclaration.Version}");
Console.WriteLine(median.KeyAt(2) == mean.KeyAt(2));   // True: nothing above the split differs
Console.WriteLine(median.KeyAt(4) == mean.KeyAt(4));   // False: the scaling was fitted behind another fill, so its fit is not reused

DeclareStep.Columns is every column the schema names; DeclareStep.Taking is the ones that take part.

using DeepSharp.Pipelines;

var schema = new DeclareStep(
[
    new ColumnDeclaration("survived", ColumnKind.Integer, Optional: false),
    new ColumnDeclaration("cabin", ColumnKind.Text, Optional: false) { Excluded = true },   // named, with its kind, and left out
]);

Console.WriteLine($"{schema.Columns.Count} columns named, {schema.Taking.Count} take part");   // read Taking, not Columns

The rules hold wherever a step comes from.

using DeepSharp.Pipelines;

try
{
    Pdd.Create()
        .ReadCsv("titanic.csv")
        .OrderBy("Date");                               // a step above the schema: the columns are declared first
}
catch (DeclarationException refused)
{
    foreach (var fault in refused.Faults)               // every fault at once, each with its step
    {
        Console.WriteLine($"step {fault.At + 1}, '{fault.Verb}': {fault.Message}");
    }
}

What a model is asked to predict comes in kinds.

using DeepSharp.Pipelines;

var shares = Pdd.Create().ReadCsv("flocks.csv")
    .Declare(schema => schema.Number("chicks", "light", "heavy"))
    .SplitAtRandom(0.70, 0.15)
    .Distribution(["light", "heavy"], scaleBy: "chicks");     // a whole divided among columns, back as chicks in each

var labels = Pdd.Create().ReadCsv("hens.csv")
    .Declare(schema => schema.Integer("sick", "lame").Number("age"))
    .SplitAtRandom(0.70, 0.15)
    .Labels(["sick", "lame"]);                                // columns that are each nought or one

var ahead = Pdd.Create().ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15, gap: 5)
    .Ahead("AAPL.Close", 5, AheadAs.Return);                  // the return five days on

Console.WriteLine(string.Join(" / ", new object[] { shares.Declaration, labels.Declaration, ahead.Declaration }));

Every answer is handed over.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15, gap: 5)
    .Ahead("AAPL.Close", 5, AheadAs.Return)
    .Drop("Date")
    .Build()
    .Run();

var train = prices.Batch(Part.Train);

Console.WriteLine(string.Join(", ", train.AnswerNames ?? []));   // the answers the output names, in its order
Console.WriteLine(train.Answers?[0].Length);                       // that many numbers a row
Console.WriteLine(train.Labels?[0]);                               // an output of one answer is handed over as labels too

A split in time can keep a gap.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15, gap: 5)   // five days kept apart before each line and at the end
    .Ahead("AAPL.Close", 5, AheadAs.Return)
    .Drop("Date")
    .Build()
    .Run();

Console.WriteLine($"{prices.CountIn(Part.Gap)} rows kept apart: fitted on by nothing, handed to nothing");

The way back in your own units, for a part and for served rows.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("pclass").Number("fare"))
    .SplitAtRandom(0.70, 0.15)
    .Target("fare")
    .Normalise("pclass", "fare")
    .Build()
    .Run();

IReadOnlyList<double[]> said = [.. prepared.Batch(Part.Test).Features.Select(_ => new[] { 0.0 })];   // what a model predicted
var inFares = prepared.BackToOriginal(said, Part.Test);                        // back in the units the fares were read in

var rows = new InMemoryRowSource(["pclass"], [["1"], ["3"]]);                  // served rows come without the answer
var served = prepared.Served(rows);
var forServed = prepared.BackToOriginal([[0.1], [-0.2]], served, rows);        // found again by place, checked by key
Console.WriteLine($"{inFares[0][0]} {forServed[1][0]} {served.Keys?.Count}");

A column can be left out and brought back as it was.

using DeepSharp.Pipelines;

var schema = new DeclareStep(
[
    new ColumnDeclaration("survived", ColumnKind.Integer, Optional: false),
    new ColumnDeclaration("cabin", ColumnKind.Text, Optional: false),
]);

var left = schema.WithColumnExcluded("cabin");                                    // still named, with its kind; nothing reads it
var back = left.WithColumn("cabin", ColumnKind.Text, ["survived", "cabin"]);      // taking it in again is clearing the word
var category = back.WithColumnKind("cabin", ColumnKind.Category);                 // a category remembers the kind it was

Console.WriteLine($"{left.Taking.Count} / {back.Taking.Count} / {category.Columns[1].Was}");

What can be done to a column is said once.

using DeepSharp.Pipelines;

var declaration = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Normalise("fare")
    .Declaration;

var left = new PipelineDeclaration(declaration.Excluding("pclass"));                         // excluded, its kind kept
var back = left.Including("pclass", ColumnKind.Text, CsvRowSource.HeaderOf("titanic.csv"));  // back as it was
var rows = declaration.ChoicesFor(["survived", "pclass", "fare"]).Rows;                      // how each stands

Console.WriteLine($"{back.Count} steps; pclass is {rows[1].Standing}, offered {rows[1].Offers}");

An output is made under its verb in one place.

using System.Text.Json.Nodes;
using DeepSharp.Pipelines;

var declaration = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Declaration;

var output = (INamesTheAnswer)StepCatalog.BuiltIn().Make("target", new JsonObject { ["column"] = "survived" }, carrying: null);
var placed = new PipelineDeclaration(declaration.WithOutput(output));      // an output placed where outputs stand

Console.WriteLine(placed);

What a pipeline decided about its columns can be saved on its own.

using DeepSharp.Pipelines;

var header = CsvRowSource.HeaderOf("titanic.csv");
var decided = Pdd.Create().ReadCsv("titanic.csv").Declare(schema => schema.Integer("survived", "pclass").Number("fare")).Declaration;

File.WriteAllText("titanic.columns.json", PipelinePreset.Of(decided, header).ToJson());   // the decisions alone: no fit, no rows

var saved = PipelinePreset.FromJson(File.ReadAllText("titanic.columns.json"), StepCatalog.BuiltIn());
var again = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(saved, out var declared)                  // the saved schema, saying which columns it decides
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Output(saved, out var taken)                      // the saved drops and output, saying what they change
    .Build();

Console.WriteLine($"{again.Declaration}: {declared.Changes.Count} decided, {taken.DropsNotMade.Count} drops not made, {saved.NewColumns(header).Count} new");

OrderBy, order.by.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")                                    // oldest first; two rows with equal keys, or a gap in one, are refused
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Build()
    .Run();

Console.WriteLine(prices.Table.RowCount);

Drop, drop.columns.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close", "AAPL.Volume"))
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Drop("AAPL.Volume")
    .Drop("Date")                                       // a column the model never sees
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", prices.Batch(Part.Train).FeatureNames));

DropGaps, drop.gaps.

using DeepSharp.Pipelines;

var passengers = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .DropGaps("age")                                    // above the split, so every part loses the same kind of row
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build()
    .Run();

Console.WriteLine($"{passengers.CountIn(Part.Train)} + {passengers.CountIn(Part.Validation)} + {passengers.CountIn(Part.Test)} rows with an age");

FillMissing(column, strategy, refuseAbove: 0.5).

using DeepSharp.Pipelines;

var filled = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median, refuseAbove: 0.5)   // above half the training rows missing, filling is invention
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", filled.Batch(Part.Train).FeatureNames));   // the filled age, and the column that says where the gaps were

Profile and Correlation, evidence.profile and evidence.correlation.

using DeepSharp.Pipelines;

var looked = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass").Optional("age", ColumnKind.Number).Number("fare"))
    .Profile()
    .Correlation(["survived", "pclass", "age", "fare"])
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build()
    .Run();

var profile = (DataProfile)looked.Evidence[2];                 // by the step's place; never in the pipeline's file
var correlation = (CorrelationInput)looked.Evidence[3];

Console.WriteLine($"{profile.Rows} training rows profiled, {profile.Alerts.Count} things found");
Console.WriteLine($"{correlation.Kept} of {correlation.Total} rows are complete");

Pipeline.ViewAt(steps).

using DeepSharp.Pipelines;

var pipeline = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Category("sex").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median)
    .Build();

var view = pipeline.ViewAt(3);                                  // the data after the first three steps, with where each row stands
Console.WriteLine($"{view.Table.RowCount} rows, {view.Standings.Count(standing => standing == Standing.Train)} of them training");
Console.WriteLine(string.Join(", ", view.MeasuredCategories("sex")));   // what the training rows hold, as an encoder learns it

Faults at their line and column.

using DeepSharp.Pipelines;

try
{
    PreparedData.FromJson(File.ReadAllText("btceur.pipeline.json"), StepCatalog.BuiltIn());
}
catch (PipelineFileException refused)
{
    foreach (var fault in refused.Faults)                       // every fault at once, each at its line and column
    {
        Console.WriteLine($"({fault.Line},{fault.Column}): {fault.Message}");
    }
}

A description every door is derived from.

using DeepSharp.Pipelines;

var catalog = StepCatalog.BuiltIn();

File.WriteAllText("pipeline.schema.json", catalog.JsonSchema());   // for an editor: offers the verbs and their keys as you type
File.WriteAllText("VERBS.md", catalog.VerbReference());            // for a person: every verb as a page

var description = catalog.Describe("fill.missing");
Console.WriteLine(description.Purpose);
Console.WriteLine(description.Template);                          // the step a new block starts from

PreparedData.Served(rows).

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Category("sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare")
    .Build()
    .Run();

var arrived = new InMemoryRowSource(["sex", "age", "fare"], [["male", "22", "7.25"], ["female", "", "71.28"]]);   // no answer: it is the question
var served = prepared.Served(arrived);

Console.WriteLine($"{served.RowCount} rows; the first is handed-in row {served.HandedInAt[0]}, key {served.Keys?[0]}");

SourceFolder.

using DeepSharp.Pipelines;

var file = "experiments/titanic.pipeline.json";
var folder = SourceFolder.OfDocument(file);               // a relative path is read from the folder the file sits in

var declaration = PipelineDeclaration.FromJson(File.ReadAllText(file), StepCatalog.BuiltIn());
var prepared = new Pipeline(declaration, rows: null, folder).Run();   // moved together with its data, it still finds it

Console.WriteLine(prepared.CountIn(Part.Train));

Table.Duplicates(parts).

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass", "sibsp", "parch").Number("fare"))
    .SplitAtRandom(0.70, 0.15)
    .Build()
    .Run();

var duplicates = prepared.Table.Duplicates(prepared.Parts);    // the rows that are there more than once
Console.WriteLine($"{duplicates.Groups} repeated rows, {duplicates.GroupsAcrossParts} of them landed in different parts");

0.2.0

New

DeepSharp.Pipelines — a pipeline you can write down.

using DeepSharp.Pipelines;

var declaration = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median)
    .Declaration;                                                    // nothing has run: every verb only records

var text = declaration.ToJson();                                     // save it, hand it to somebody
var again = PipelineDeclaration.FromJson(text, StepCatalog.BuiltIn());

Console.WriteLine(again == declaration);                             // True: both ways reach equally far

The split is a line you cannot step over.

using DeepSharp.Pipelines;

PipelineBuilder above = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number));

// above.FillMissing(...) does not compile: nothing that learns from the data is offered above the line.
FittingBuilder below = above.SplitStratified("survived", train: 0.70, validation: 0.15);

below.FillMissing("age", With.Median);                  // the steps that learn exist only on this builder
Console.WriteLine(below.Declaration);

A gap is filled by a name you can check.

using DeepSharp.Pipelines;

var filled = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number).Optional("fare", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median)           // With.Mean, With.Median, With.Zero, With.Previous, With.Constant(0)
    .FillMissing("fare", With.Constant(0))     // a strategy that takes a number is written with one
    .Build()
    .Run();

Console.WriteLine(filled.ToJson());            // what each fill learned is written beside the declaration

It reads real files.

using DeepSharp.Pipelines;

var strict = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(
        schema => schema.Integer("survived").Optional("age", ColumnKind.Number).Number("fare"),
        Remainder.Refuse)                      // Drop is the default; Keep carries the rest along as text
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build();

Console.WriteLine(strict.Declaration);

Three ways to divide the rows, and none of them a default.

using DeepSharp.Pipelines;

var byTime = Pdd.Create().ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .SplitByTime("Date", train: 0.70, validation: 0.15).Build();

var atRandom = Pdd.Create().ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitAtRandom(0.70, 0.15, seed: 7).Build();                   // the seed is written into the declaration

var stratified = Pdd.Create().ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15).Build();   // a rare answer survives into validation

Console.WriteLine(string.Join(" / ", new object[] { byTime.Declaration, atRandom.Declaration, stratified.Declaration }));

A row belongs to a Part, not to a "split".

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build()
    .Run();

foreach (var part in new[] { Part.Train, Part.Validation, Part.Test })
{
    Console.WriteLine($"{part}: {prepared.CountIn(part)} rows");   // what each row landed in
}

The share you are measured on is never written down.

using DeepSharp.Pipelines;

var byTime = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .SplitByTime("Date", train: 80, validation: 10)          // test: 10, whatever is left
    .Build()
    .Run();

var twoWays = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitAtRandom(0.80)                                      // 0.80 and 0.20, no validation
    .Build()
    .Run();

Console.WriteLine($"{byTime.CountIn(Part.Test)} rows to be measured on, {twoWays.CountIn(Part.Validation)} held for validation");

Predict(10) holds a part back that takes no part in anything.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .Predict(10)                                        // held out of everything
    .SplitByTime("Date", train: 65, validation: 15)     // test: 10 — the four make a whole
    .Drop("Date")                                       // a moment is not a number a model takes
    .Normalise("AAPL.Close", Scale.Robust)
    .Build()
    .Run();

var later = prices.Batch(Part.Predict);                 // handed over like any other part
Console.WriteLine(later.RowCount);

Features, before the split, because they learn nothing.

using DeepSharp.Pipelines;

var rides = Pdd.Create()
    .ReadCsv("rides.csv")
    .Declare(schema => schema.Timestamp("start").Number("distance", "fare"))
    .AddFeature("fare_per_km", "fare", Arithmetic.DividedBy, "distance")   // a column worked out from two others
    .Cyclical("start", Period.HourOfDay, Form.Signed)                      // 23:00 and 00:00 are neighbours
    .SplitByTime("start", train: 0.70, validation: 0.15)
    .Build();

Console.WriteLine(rides.Declaration);

Everything that learns, after it.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Category("sex").Optional("age", ColumnKind.Number).Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median)                           // each is fitted on the training rows alone...
    .Encode("sex", As.OneHot, Unseen.Reserve)                  // ...and a category nobody has seen gets a slot
    .Normalise("fare", Scale.MinMax, OutOfRange.Clip)          // a new high is held at the edge of the learned range
    .Build()
    .Run();

Console.WriteLine(prepared.ToJson());                          // what was learned, replayed unchanged on the rest

A handover, and a way back in.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")                                        // the answer, handed over apart from the numbers
    .Normalise("fare")
    .Build()
    .Run();

var train = prepared.Batch(Part.Train);
Console.WriteLine($"{train.RowCount} rows of {string.Join(", ", train.FeatureNames)}");

var tomorrow = new InMemoryRowSource(["fare"], [["7.25"]]);    // rows nobody had seen: nothing is fitted again
Console.WriteLine(prepared.Replay(tomorrow).RowCount);

var served = PreparedData.FromJson(prepared.ToJson(), StepCatalog.BuiltIn());   // from its own file, no data needed
Console.WriteLine(served.Replay(tomorrow).RowCount);

A value that is not a number gets its own verb.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("trades.csv")
    .Declare(schema => schema.Integer("won").Number("ratio", "volume", "range"))
    .SplitStratified("won", train: 0.70, validation: 0.15)
    .FillNaN("ratio")                                  // stops the run by default: a not-a-number is a broken division
    .Normalise("volume", Scale.Quantile)               // by rank: keeps the whole shape of the training distribution
    .Normalise("range", Scale.Power)                   // towards a bell curve, for a column that leans one way
    .Build()
    .Run();

Console.WriteLine(prepared.CountIn(Part.Train));

DeepSharp.Pipelines.DataFrame — one reader for the long tail.

using DeepSharp.Pipelines;
using Microsoft.Data.Analysis;

var frame = new DataFrame(
    new Int32DataFrameColumn("survived", [0, 1, 1, 0, 1, 0]),
    new DoubleDataFrameColumn("fare", [7.25, 71.28, 7.92, 8.05, 53.1, 8.46]));   // from a file, a query, rows you already had

var prepared = Pdd.Create()
    .ReadDataFrame(frame)                     // ReadCsvFrame(path) and ReadDbAsync(reader) arrive through the same door
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitAtRandom(0.50)
    .Build()
    .Run();

Console.WriteLine(prepared.CountIn(Part.Train));

DeepSharp.Pipelines.Indicators — a fourth package, and not a line of indicator arithmetic.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.High", "AAPL.Low", "AAPL.Close"))
    .OrderBy("Date")                                                        // an indicator reads the rows in their order
    .Add(new AddIndicatorStep("rsi", Indicator.Rsi, ["AAPL.Close"], 14))    // borrowed from MatPlotLibNet, above the split
    .Add(new AddIndicatorStep("atr", Indicator.Atr, ["AAPL.High", "AAPL.Low", "AAPL.Close"], 14))
    .DropWarmUp()
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Build();

Console.WriteLine(prices.Declaration);

With indicators on the data, the row count is the rows minus the longest warm-up.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")
    .Add(new AddIndicatorStep("sma20", Indicator.Sma, ["AAPL.Close"], 20))
    .DropWarmUp()                                      // the rows at the start that no column can speak for yet
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Build()
    .Run();

Console.WriteLine(prices.Table.RowCount);              // the rows minus the longest warm-up

A prediction comes back in the units it was read in.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .OrderBy("Date")
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Target("AAPL.Close")
    .Drop("Date")
    .Normalise("AAPL.Close", Scale.Standard)           // scale what is predicted, and the predictions come back scaled
    .Build()
    .Run();                                            // the run checks the way back before it hands anything over

Console.WriteLine(prepared.BackToOriginal(0.5));       // a prediction in the units the prices were read in

The variance-stabilising shapes, and the tail.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close", "AAPL.Volume"))
    .Reshape("AAPL.Volume", Maths.Log)                                    // above the split: a logarithm needs no other number
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .ClipOutliers("AAPL.Close", Bounds.Iqr, outlier: Outlier.Clip)        // bounds learned from the training rows alone
    .Build()
    .Run();

Console.WriteLine(prices.CountIn(Part.Train));

A moment in time comes apart into the pieces people reason with.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .TimeParts("Date", TimePart.Season, TimePart.Quarter)      // categories: March is not three of anything
    .TimePartsAsNumbers("Date", TimePart.Year)                 // the one piece where the order really is the point
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Drop("Date")
    .EncodeCategories()
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", prices.Batch(Part.Train).FeatureNames));

Which columns are categories is said where the data is declared.

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Category("sex", "embarked").Number("fare"))   // said where the data is declared
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Target("survived")
    .EncodeCategories()                                                                         // takes them by name, as one step
    .Build()
    .Run();

Console.WriteLine(string.Join(", ", prepared.Batch(Part.Train).FeatureNames));

It fits an application that has a host.

using DeepSharp.Pipelines;
using Microsoft.Extensions.DependencyInjection;

var provider = new ServiceCollection()
    .AddDeepSharpPipelines()                           // the pipeline factory and the catalog of verbs
    .BuildServiceProvider();

var pipeline = provider.GetRequiredService<IPipelineFactory>().Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build();

Console.WriteLine($"{pipeline.Declaration} / {provider.GetRequiredService<StepCatalog>().Knows("fill.missing")}");

0.1.0

New

Shape — how big a tensor is.

using DeepSharp.Tensors;

var rows = new Shape(2, 3);
Console.WriteLine(rows);                        // 2x3
Console.WriteLine(rows == new Shape(2, 3));     // True: the same sizes are the same shape

var loss = new Shape();                         // no sizes at all: a single number
Console.WriteLine($"{loss} holds {loss.Count}");

Tensor — the numbers themselves.

using DeepSharp.Tensors;

float[] mine = [1f, 2f, 3f, 4f];
var tensor = Tensor.From(new Shape(2, 2), mine);

mine[0] = 99f;                                  // reusing your own array afterwards...
Console.WriteLine(tensor.Values[0]);            // ...cannot alter the tensor: still 1

CpuBackend — the arithmetic.

using DeepSharp.Tensors;

var maths = new CpuBackend();
var a = Tensor.From(new Shape(2, 2), [1f, 2f, 3f, 4f]);
var b = Tensor.From(new Shape(2, 2), [10f, 20f, 30f, 40f]);

Console.WriteLine(string.Join(", ", maths.Add(a, b).Values.ToArray()));        // 11, 22, 33, 44
Console.WriteLine(string.Join(", ", maths.Multiply(a, b).Values.ToArray()));   // 10, 40, 90, 160

ITensorBackend — the plug it all goes through.

using DeepSharp.Tensors;

static Tensor Twice(ITensorBackend maths, Tensor values) => maths.Add(values, values);

ITensorBackend maths = new CpuBackend();        // handed in; another engine would be handed in the same way
var doubled = Twice(maths, Tensor.From(new Shape(3), [1f, 2f, 3f]));

Console.WriteLine($"{maths.Name}: {doubled}");