Pipeline

The pipeline

This page is the vocabulary: every verb, in the order you write it. All of it runs today — reading a file, declaring its columns, putting the rows in order, working out features and indicators, dividing the rows three ways, filling gaps, encoding categories, six scales, the evidence a run produces, and the handover to whatever learns — each learning step fitted on the training rows alone and replayed unchanged everywhere after, with what it learned written into the file beside the declaration. What is still design is the part after the handover: the measures of a trained model, and the learners themselves. Those sections say so where they begin.

Why it is built this way at all is PDD. The same pipeline written block by block, with the data under any block, is the Notebook. And the reference of every verb exactly as a file holds it — VERBS.md — is written by the steps themselves, so it cannot drift from them.

The verbs live in DeepSharp.Pipelines, which is its own package: it knows nothing about tensors, and a service that only prepares data never carries an engine it does not call. The indicators and the data-frame reader are packages of their own too.

The whole thing in one look

using DeepSharp.Pipelines;

var passengers = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema
        .Integer("survived", "pclass", "sibsp", "parch")
        .Number("fare")
        .Optional("age", ColumnKind.Number)
        .Category("sex", "embarked"))
    .AddFeature("family", "sibsp", Arithmetic.Plus, "parch")
    .SplitStratified("survived", train: 0.70, validation: 0.15)   // test is the rest
    // ---- nothing above this line is allowed to learn from the data ----
    .FillMissing("age", With.Median)
    .EncodeCategories()
    .Normalise("age", "fare", "family")
    .Target("survived")
    .Build()
    .Run();

var train = passengers.Batch(Part.Train);   // rows of numbers, their names, and the answers apart

Everything is reached through a factory and built as a chain, and the chain is not decoration: the stages are different types, so the steps that learn from the data are not offered until you have said how to split.

The chain is where you meet that rule; it is not where the rule lives. A step that learns says so in its own type, and the check is made on the finished declaration — so it holds whether the step came from the chain, from a verb another package added, out of a file somebody edited by hand, or from a notebook.

The rules every pipeline keeps

Whichever way a pipeline was written, the declaration refuses it unless:

• it has at most one source, one schema, one split, one order and one target — a second one used to be ignored, so a file said one thing and the numbers came from another;
• the source is the first step, and the schema comes directly after it, because everything else works on columns;
• nothing that learns from the data stands above the split;
• rows are dropped and put in order above the split, never below it, because the split divides the rows it is given once;
• a step that reads the rows in their order stands below the step that says what that order is;
• every column a step reads is there where it reads it, of a kind it can work on;
• every step does something the run acts on.

A refusal names every fault at once, each with the step it is at — DeclarationException.Faults — and PipelineDeclaration.FaultsIn(steps) gives the same list without refusing, for something that writes a pipeline a piece at a time.

Reading the data

One verb per source, each an extension method living in the package that brings what it needs:

.ReadCsv(path)	DeepSharp.Pipelines	A comma-separated file.
.Read(rows, description)	DeepSharp.Pipelines	Rows handed in: anything that can be turned into an IRowSource. The file records what they were, rather than pretending it can open them again.
.ReadDataFrame(frame)	DeepSharp.Pipelines.DataFrame	A data frame you already hold.
.ReadCsvFrame(path)	DeepSharp.Pipelines.DataFrame	A CSV read through the frame's own reader.
.ReadDbAsync(reader)	DeepSharp.Pipelines.DataFrame	The rows of a database query, through any ADO.NET provider.

Reading is a solved problem with a long tail — pandas exposes nineteen readers — and reproducing that tail is a library in itself. Anything else is one small IRowSource away.

A relative path is read from the folder the pipeline sits in: the folder of the file it was read from, or of the notebook it was written in, and the working directory for a pipeline written in code. So a pipeline file moved together with its data still finds it. Hand the folder over with new Pipeline(declaration, rows, folder) and SourceFolder.OfDocument(path).

A live source is fetched, then read — design, not yet built. An endpoint answers differently every time it is called, so a pipeline that read from one during training would train on different numbers tomorrow while claiming to be the same pipeline. A fetch will land the result as a file, recording what it asked for, and the pipeline reads that file like any other.

Declaring the columns

.Declare(...) comes directly after the source. It names the columns that take part and what each holds — Text, Number, Integer, Boolean, Timestamp, Category — marks the ones that may be absent with Optional, and says what becomes of the rest: dropped by default, because a column nobody declared is a column nobody checked. Remainder.Keep carries them along as text, for a pipeline still finding out what is in the file, and Remainder.Refuse stops the run when the file holds one.

Everything below is checked against it. The columns are followed from the schema down, each step saying what it leaves behind, so a step that reads a column the schema left out, or one a step above took away, is refused where it stands — not a whole run later, as a column nobody could find. When the file is opened, the declared columns are checked against the ones really there before the first step runs.

Values are read the same way on every machine: numbers and dates under the invariant culture, true and false in the spellings files actually use — True and False included. A cell that cannot be read as its column's kind is refused with its row, its column and its value, and a number too large to hold, such as 1e999, is refused rather than read as infinity.

Putting the rows in order

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.High", "AAPL.Low", "AAPL.Close"))
    .OrderBy("Date")                                          // oldest first
    .AddIndicator("rsi", Indicator.Rsi, ["AAPL.Close"], 14)
    .DropWarmUp()
    .SplitByTime("Date", train: 0.70, validation: 0.15)
    .Build()
    .Run();

A file does not promise an order. An export or a query without an ordering hands rows over in whatever order it happened to hold them, and a step that reads the rows before a row then reads the wrong ones: the same prices listed newest first gave a five-day average of 98.352 where the right one is 99.74. So a step that reads the order — an indicator, DropWarmUp, a fill that carries the previous value forward — needs an OrderBy above it, and is refused without one.

OrderBy takes one or more columns and puts the smallest first, comparing the values in their own kind. It refuses a gap in a key, and two rows whose keys are equal: nothing then says which came first, so add a second key column.

Features

A feature is worked out from a single row — or, for an indicator, from the rows that came before it — and learns nothing from the data as a whole, so features stand above the split.

.AddFeature(name, left, Arithmetic.Plus, right)	A column from two others: Plus, Minus, Times, DividedBy.
.Cyclical(column, Period.HourOfDay, form)	A moment as a place on a circle: HourOfDay, DayOfWeek, DayOfMonth, MonthOfYear.
.TimeParts(column, TimePart.Month, …)	The pieces people reason with — Minute, Hour, DayOfWeek, DayOfMonth, Month, Quarter, Season, Year — as categories.
.TimePartsAsNumbers(column, …)	The same pieces as numbers, for the one where the order really is the point.
.Reshape(column, Maths.Log)	Log, Log1P, Reciprocal, Sqrt, Square, ArcSin, Abs, Sign.
.AddIndicator(name, Indicator.Rsi, columns, period)	An indicator, from DeepSharp.Pipelines.Indicators.

Cyclical values. An hour of day is not a number between 0 and 23: 23:00 and 00:00 are neighbours, and a model given the plain number is told they are as far apart as possible. The angle is written as a sine and a cosine, which puts the wrap where it belongs, in one of three forms:

Form.Signed	one column each, -1..1	the plain value
Form.Unit	one column each, (x+1)/2 into 0..1	everything on one range; zero is now 0.5
Form.SplitSign	two columns each, 0..1	pos = max(x,0), neg = max(-x,0)

Signed and Unit are the same feature up to a shift, so a layer with a bias absorbs the difference. SplitSign adds something: each half gets its own weight, so a rise and a fall can be answered differently. It has to be the last form a value takes — normalising one half by its own extremes would give one quantity two slopes — so a scale on a half is refused.

The parts of a moment are categories, because a month is not a quantity: March is not three of anything.

Indicators — Sma, Ema, Rsi, Atr, Adx, Cci, WilliamsR, Obv, Macd, BollingerBands, Stochastic, Vwap — are borrowed from MatPlotLibNet rather than written again. Two things about them are measured rather than promised: each is held to an impulse test — change one row, and no earlier row may move — and the warm-up arrives as a gap, because an indicator of period N has nothing to say about the first N rows. DropWarmUp() drops the rows at the start that no column can speak for yet: 506 Apple rows with a twenty-day band on them are 487 rows of data.

Taking something away

.Drop(columns)	Leaves columns out from here on — above the split or below it.
.DropGaps(columns)	Drops every row with a gap in one of the named columns, above the split.
.DropWarmUp()	Drops the rows at the start that an indicator cannot yet speak for, above the split.

Rows are dropped before they are divided, never after: dropping rows below the split would quietly change how many each part holds, which is what the split promised.

The split

.SplitByTime(...), .SplitAtRandom(...) and .SplitStratified(...) are the line in the chain. Above it nothing may learn from the data; below it the steps that do become available. Which one to use is not the library's decision — by time when you predict forward, at random for rows that do not depend on one another, stratified when an answer is rare — so none of them is the default.

A split divides rows by what they say, not by where they stand. Each row is ranked by a digest of its own contents and the seed, so the same rows are dealt the same way whatever order the file lists them in, and every copy of a repeated row lands where its first copy does. A split in time never divides a moment: every row of one moment lands on the side of the line its first row does.

A row does not belong to a split, it belongs to a part: you split the rows, and Part.Train, Part.Validation, Part.Test and Part.Predict are what they land in.

The number you do not write

You say what training takes and what validation takes. The share you are measured on is whatever is left — it is not a parameter, so nothing can add up to more than there is.

using DeepSharp.Pipelines;

var byTime = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .SplitByTime("Date", train: 80, validation: 10)          // test: 10
    .Build();

var twoWays = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Number("fare"))
    .SplitAtRandom(0.80)                                      // 0.80 and 0.20, no validation
    .Build();

Percentages and fractions say the same thing — 80, 10 and 0.80, 0.10 — and mixing them in one call is refused rather than read as eight-tenths of a percent. Three numbers that have to add to one is a rule that breaks quietly: 0.70, 0.15, 0.10 leaves a twentieth of the rows in no part at all.

A part that takes no part

Predict(10) holds a tenth out of everything. Nothing is fitted on it and nothing is measured on it, so running a trained network over those rows is the closest thing to running it tomorrow; with a split in time they are the newest days in the file.

using DeepSharp.Pipelines;

var prices = Pdd.Create()
    .ReadCsv("apple.csv")
    .Declare(schema => schema.Timestamp("Date").Number("AAPL.Close"))
    .Predict(10)                                        // held out of everything
    .SplitByTime("Date", train: 65, validation: 15)     // test: 10 — the four make a whole
    .Normalise("AAPL.Close", Scale.Robust)
    .Drop("Date")                                       // a moment is not a number a model takes
    .Build()
    .Run();

var later = prices.Batch(Part.Predict);                 // handed over like any other part

It is written once, before the split, and the split is what sets it aside — so a pipeline that holds rows back and then never divides anything is refused where it is built.

Gaps

Two things usually thrown onto one heap are kept apart. A value is missing when it was never there — an empty field, a NULL — and that is data. A value is not a number when arithmetic produced none, which is almost always a fault further upstream.

using DeepSharp.Pipelines;

var filled = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median, refuseAbove: 0.5)   // With.Mean, With.Zero, With.Constant(0),
    .Build()                                             // With.Previous, With.Refuse
    .Run();

The strategy is a named value rather than a flag, because Fill("trades", true, false) tells the next reader nothing. Whatever fills the gap is learned, from the training rows alone, and so is what the fill counts: the Titanic ages have 177 gaps in the file and 133 among the rows a stratified split trains on.

A column is written alongside, always, saying where the gaps were — age_was_missing. Filling destroys the difference between "absent" and "the value happened to be that", and it destroys it for good; if you do not want the column, you drop it.

A fill has a point beyond which it is invention. deck is empty in 688 of the 891 rows, and filling a column like it would manufacture three quarters of it. refuseAbove is the share of the training rows that may be gaps and still be filled. Above it the column is not filled, and the column that marks the gaps speaks for it. It has no default, because a number nobody chose would be the same quiet decision somewhere else.

.FillNaN(column) deals with a value that is not a finite number, and refusing it is the default: a not-a-number or an infinity is somebody's broken division, and carrying it into a model as if it were a measurement is the one thing a pipeline should not do quietly. Every fit refuses one among its training values, naming fill.nan, and the handover refuses one too.

Categories

.Encode(column, As.OneHot) learns which categories exist — from the training rows, like everything else below the line. As.Ordinal writes one column of places instead. An unfamiliar category turns up in production sooner or later, so what happens then is part of the declaration: Unseen.Reserve keeps a slot for it, Unseen.Refuse says plainly that the data is outside what the model was trained on.

Which columns are categories is said where the data is declared — Category("sex", "embarked") in the schema — and .EncodeCategories() then takes them all by name, as one step. A category that never became numbers is refused at the handover rather than dropped in silence.

Normalising

Each kind learns something different, which is why the kind is declared and what it learned is stored apart:

	what it does	what it learns
Scale.Standard	(x - mean) / spread	mean and spread
Scale.MinMax	into 0..1	the smallest and largest value
Scale.MaxAbs	divide by the largest magnitude	that magnitude
Scale.Robust	median and the middle half of the data	median and quartiles
Scale.Quantile	rank, then spread evenly	the shape of the distribution
Scale.Power	reshape towards a bell curve	one parameter per column

Standard and min-max are both pulled around by a single extreme value, so one spike can squeeze everything else into a sliver of the range. On prices and volumes that is the normal case, which is what Robust and Quantile are for.

Two choices are made here rather than discovered later:

• Outside the learned range, while the model is running: OutOfRange.Pass lets the value through, OutOfRange.Clip holds it at the edge, OutOfRange.Refuse says the data has moved.
• The target. Normalise what you predict and the way back is part of the saved pipeline: prepared.BackToOriginal(prediction) puts a prediction back in the units it was read in, walking every step that touched the target backwards. The run checks that the way back really leads back before handing anything over.

.ClipOutliers(column, Bounds.Iqr) holds the extremes to bounds learned from the training rows — by quantile, by spread or by the middle half — and what happens beyond them is your choice: Outlier.Clip, Outlier.Blank or Outlier.Refuse.

.NormaliseRow(Norm.L2, columns) is a different animal: it works across a row rather than down a column and learns nothing, which is what you want when the direction of a row matters and its size does not. A column is named once in it.

Evidence

using DeepSharp.Pipelines;

var looked = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived", "pclass").Optional("age", ColumnKind.Number).Number("fare"))
    .Profile()
    .Correlation(["survived", "pclass", "age", "fare"])
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .Build()
    .Run();

var profile = (DataProfile)looked.Evidence[2];

Evidence is declared with the pipeline, before the numbers exist, so every run produces the same proof and a report cannot shrink to whatever happened to look good. Both kinds are measured on the rows the split trains on — the split below them as much as one above — because a profile over every row lets the rows a model is measured on shape what it is shown.

A profile gives, per column, the rows, the gaps, the values that are not numbers and the distinct values, and for numbers the smallest, the largest, the mean and the median. For each thing it finds it names the step that answers it: a gap with fill.missing, a value that is not a number with fill.nan, a column that never changes with drop.columns, words with an encoder. It also counts the rows that are there more than once.

A correlation says which training rows it may be drawn from — those with a number in every column named — how many that is, and the rule that left the others out, because a correlation over the rows that happened to be complete is a different number from one over all of them. Whatever draws it works the correlation out: the Notebook draws a heatmap.

What they produce is in PreparedData.Evidence, by the step's place, and never in the pipeline's file: evidence is output, not the pipeline. A replay produces none.

The data after any step. pipeline.ViewAt(steps) gives the table as it stands after any number of steps, with where every row stands — in the part the split puts it in, dropped before it gets there, or undivided when nothing divides it. It is what the grid under a notebook block shows.

The handover

Batch(part) is where the pipeline stops: rows of numbers, their column names in a fixed order, and the answer handed over separately when Target named one. A network built here, a trainer from another .NET library and something you wrote all take the same handover, which is the only reason two of them can honestly be compared. A column still holding words, a gap, and a value that is not a finite number are refused there rather than passed on.

Replay(rows) runs the same declaration over rows nobody had seen, with the numbers the training rows produced and nothing fitted again. It walks the steps exactly as the run did — it puts the rows in order and drops the warm-up rows too. Served(rows) hands the result over the way Batch does, without an answer — a served row is the question — and with which of the handed-in rows each served row is, so a prediction finds its way back.

The pipeline as a file

using DeepSharp.Pipelines;

var prepared = Pdd.Create()
    .ReadCsv("titanic.csv")
    .Declare(schema => schema.Integer("survived").Optional("age", ColumnKind.Number))
    .SplitStratified("survived", train: 0.70, validation: 0.15)
    .FillMissing("age", With.Median)
    .Build()
    .Run();

var text = prepared.ToJson();                                               // both halves
var again = PreparedData.FromJson(text, StepCatalog.BuiltIn());             // anywhere, no data needed
var declared = PipelineDeclaration.FromJson(text, StepCatalog.BuiltIn());   // the person's half alone

A built pipeline is saved as one file with two halves, because they have two different authors: the declaration, which is what you wrote, and the fitted half, which is what the fit learned.

{
  "version": 2,
  "declaration": [
    { "step": "read.csv", "path": "titanic.csv" },
    { "step": "declare", "remainder": "drop", "columns": [
        { "name": "survived", "kind": "integer", "optional": false },
        { "name": "age", "kind": "number", "optional": true } ] },
    { "step": "split.stratified", "column": "survived", "train": 0.7, "validation": 0.15, "test": 0.15, "predict": 0, "seed": 20260923 },
    { "step": "fill.missing", "column": "age", "with": "median" }
  ],
  "fitted": [
    { "step": "split.stratified", "prefix": "…",
      "learned": { "rows.predict": 0, "rows.test": 135, "rows.train": 623, "rows.validation": 133, "digest": "…" } },
    { "step": "fill.missing", "prefix": "…", "learned": { "gaps": 133, "value": 29 } }
  ]
}

Keeping the halves apart is what lets you re-fit the same declaration on fresh data, compare two runs by their declarations alone, and run a trained model in a service that never knew the builder existed. Every fitted entry carries a key made from its own step and every step above it, so a fit is never used under steps that changed after it was learned. The split writes an entry of its own: how many rows went to each part, and a digest of the rows it divided.

A file is read with a catalog — the list of verbs the reader knows. StepCatalog.BuiltIn() knows this package's own; add .WithIndicators() for a file that holds indicators. A step nobody registered is refused rather than skipped, and so is a key a step does not take.

Everything wrong with a file is said at once, each fault at its line and column:

btceur.pipeline.json(9,5): Step 4: 'feature.indicator' is a step from DeepSharp.Pipelines.Indicators, which is not registered here. Reference the package and register its steps with the catalog that reads this file.
btceur.pipeline.json(10,5): Step 5: The step 'split.byTime' cannot be read: The shares add up to 1.3 and a split has to use every row.
btceur.pipeline.json(11,5): Step 6: 'normalize' is not a step anything here knows. The nearest one it knows is 'normalise'.

"I do not know this step" and "I know it, but it is not installed" are two different problems for the person reading, so they are two different messages.

The file names its version, and every verb the version from which it means what it says now. A file from 0.2 names none and is read as the first version: its declaration loads, except for the verbs whose meaning has changed since — the three splits and drop.warmup — which are refused by name rather than run the new way. A fitted half from 0.2 is refused: fit again. A file newer than the library is refused whole.

The schema and the template come from the same description the chain is built from. The JSON Schema of the file — pipeline.schema.json — the template a new step starts from, the verb reference and the notebook's form are all generated from each step's own list of parameters, so a file cannot be valid in a way the code is not.

What the run must prove about a model — design, not yet built

.Report(r => r.Measure(Metric.Rmse, Metric.Mae, Metric.R2)
              .On(Part.Train, Part.Validation, Part.Test)
              .As(Show.Grid | Show.Chart))

The profile and the correlation are the evidence about the data, and they run today. The measures of a trained model join them when there is a model: named before the numbers exist, on the three parts side by side — a low error on training beside a high one on test is overfitting, seen rather than deduced.

Choosing what learns — design, not yet built

Everything up to here is the same whatever is going to learn from the result, so this is where the pipeline stops. What learns is your choice — a network built here, a trainer from an established .NET machine-learning library, or one you wrote — and they all plug in at the same handover.

That is what makes a comparison mean anything. Two learners measured on the same prepared data with the same declared measures can be put beside each other; two learners each fed by their own preparation cannot. The same data is not the same representation for everyone, though: a network wants everything numeric, expanded and on one scale, while a tree does not care about scale and is hurt by a wide one-hot column. So a learner will say what it needs, and the steps it does not need are skipped by declaration and written down as skipped — never dropped quietly, because the artefact has to keep saying what each run actually saw.