DeepSharp 0.2.0

The data half of the library: everything a set of rows goes through before a model ever sees it, declared
once as a file you can save and replay. The tensors of the first release are untouched.

Added

• DeepSharp.Pipelines — a pipeline you can write down. A second package, for the path your data takes
  before a model ever sees it. Pdd.Create() starts a pipeline and every verb after it records what is to
  be done rather than doing it, so what you wrote can be saved as a file, handed to somebody, and replayed
  later. ToJson writes the declaration out and FromJson reads it back as the same declaration — a
  property with a test on it, because a promise that both ways reach equally far decays silently otherwise.

• The split is a line you cannot step over. SplitByTime hands back a different kind of builder, and the
  steps that learn from the data — filling a gap, and everything that follows it — exist only on that one.
  The rule holds wherever a step arrives from: through the chain, through the extension point another
  package uses, or out of a file somebody edited by hand, because it is checked on the declaration itself.
  A step that learns says so in its type, and one that divides the rows says so too. A file naming a step
  nothing has registered is refused rather than read with the step left out, and a message says which verb
  and whether it is unknown or merely not installed.

• A pipeline you split is finished. The builder you started with cannot be written to afterwards, so a
  feature cannot arrive on the far side of the line, and two arrangements of the same data are two
  pipelines rather than one declaration containing both.

• A gap is filled by a name you can check. With.Mean, With.Median, With.Zero, With.Previous and
  With.Constant(0) — a word a file did not define is refused rather than carried to whatever a default
  branch would have done with it, and a strategy that takes a number is written with one.

• It reads real files. Declare names the columns and their kinds and says what becomes of the rest —
  dropped by default, because a column nobody declared is a column nobody checked. An empty cell is a gap
  and stays one; a cell that cannot be read as what it was declared to be is refused with its row, its
  column and its value. True and false are accepted in the spellings files actually use, and numbers and
  dates are read the same way on every machine.

• Three ways to divide the rows, and none of them a default. By time, which refuses a row that has no
  time rather than guessing where it belongs; at random with a seed written into the declaration; and
  stratified, which deals each group out separately so a rare answer survives into validation.

• A row belongs to a Part, not to a "split". You split the rows; what each row lands in is a part
  of the data — Part.Train, Part.Validation, Part.Test, Part.Predict — which is what Batch,
  CountIn and PreparedData.Parts speak in. The verbs keep the word for the act: SplitByTime,
  SplitAtRandom, SplitStratified.

• The share you are measured on is never written down. You say what training takes and what validation
  takes; test is whatever is left. Three numbers that must add to one is a rule that breaks quietly — 0.70,
  0.15 and 0.10 leaves a twentieth of the rows in no split at all and every number afterwards is computed
  over less data than you think — so the third number is not a parameter at all. 80, 10 and 0.80, 0.10
  say the same thing, and mixing percentages and fractions in one call is refused rather than read.
  SplitAtRandom(0.80) is the two-way division, with nothing held for validation.

• Predict(10) holds a part back that takes no part in anything. Nothing is fitted on it and nothing
  is measured on it, so running a trained network over those rows is the closest thing to running it
  tomorrow; with a split in time they are the newest rows in the file. Training, validation and test are
  then the ninety that remain, and Batch(Part.Predict) hands them over like any other part. A pipeline
  that holds rows back and then never splits is refused where it is built.

• Features, before the split, because they learn nothing. A column worked out from two others, and a
  moment in time written as a place on a circle so that eleven at night and midnight are neighbours — in
  whichever of three forms you ask for: the plain value, the same value between nothing and one, or two
  columns saying how far up and how far down.

• Everything that learns, after it. Filling a gap, encoding a category, and six ways of bringing a
  column onto a comparable scale. Each is fitted on the training rows alone and then replayed unchanged
  over validation, test and anything that arrives later; what each one learned is written into the file
  beside the declaration. What happens to a value outside the learned range is your choice, said out loud —
  a price meets a new high the first week it is in production.

• A handover, and a way back in. Target names the column being predicted and it is handed over apart
  from the numbers, never among them. Batch(part) gives rows of numbers with their names in a fixed
  order, and refuses a column that still holds words, a gap nobody filled, or a target that no longer
  exists. Replay runs the same declaration over rows nobody had seen, with the numbers the training rows
  produced and nothing fitted again — which is what serving is — and a saved pipeline can be loaded back
  from its own file to do exactly that.

• A value that is not a number gets its own verb. FillNaN stops the run by default, because a
  not-a-number is somebody's broken division and carrying it into a model as if it were a measurement is
  the one thing a pipeline should not do quietly. Two more scales complete the set: by rank, which keeps
  the whole shape of the training distribution, and a reshaping towards a bell curve for a column that
  leans heavily one way.

• DeepSharp.Pipelines.DataFrame — one reader for the long tail. A third package, reading a pipeline's
  rows out of a Microsoft.Data.Analysis data frame, and so out of anything that can fill one: a file, a
  database query, rows you already had. ReadDataFrame, ReadCsvFrame and ReadDbAsync all arrive through
  the same door the pipeline already had for rows handed in, which is why the pipeline itself still needs no
  reader beyond its own. An absent value stays absent rather than becoming a not-a-number.

• DeepSharp.Pipelines.Indicators — a fourth package, and not a line of indicator arithmetic. Moving
  averages, RSI, ATR, ADX, MACD, Bollinger bands, the stochastic and VWAP, borrowed from MatPlotLibNet's
  published package. They stand above the split because they learn nothing, and two things about them are
  measured rather than promised: every one is held to an impulse test — change one row and no earlier row
  may move — and the warm-up arrives as an absence rather than as a number nobody took.

• With indicators on the data, the row count is the rows minus the longest warm-up. DropWarmUp() cuts
  the rows at the start that no column can speak for yet, before the split, because a row nobody can use
  should never land in one. An indicator of period N does not say "nothing happened" about the first N
  rows, it says "not enough history yet" — and filling that with a number learned from training would
  invent a measurement nobody took. 506 rows with a twenty-period average on them are 487 rows of data.

• A prediction comes back in the units it was read in. Scale what a model is asked to predict and its
  predictions come back scaled, so BackToOriginal walks the steps that touched the target backwards and
  undoes each — every scaling, and a logarithm, a root or a reciprocal. The run checks it before handing
  anything over: transform, undo, compare against what was read, and stop if it does not lead back. A step
  that threw information away says so rather than returning a number in units nobody can name, and undoing
  a logarithm gives the middle value rather than the average one, which is said where it matters.

• The variance-stabilising shapes, and the tail. Reshape(column, Maths.Log) — and Log1P, Reciprocal,
  Sqrt, Square, ArcSin, Abs, Sign — above the split, because the logarithm of a number does not depend on
  any other number. ClipOutliers holds the extremes to bounds learned from the training rows, by
  quantile, by spread, or by the middle half, and what happens outside them is your choice: held at the
  edge, made a gap, or refused.

• A moment in time comes apart into the pieces people reason with. TimeParts("Date", TimePart.Season, TimePart.Quarter) — and the minute, the hour, the day of the week, the day of the month, the month and
  the year. They arrive as categories, because a month is not a quantity: March is not three of
  anything and December is not twelve times January. TimePartsAsNumbers is there for the piece where the
  order really is the point, and where time wraps round a circle says it better than either.

• Which columns are categories is said where the data is declared. Category("sex", "embarked") in the
  schema, and EncodeCategories() takes them by name afterwards, so adding one to the schema does not mean
  remembering a second line further down. A category that never became numbers is refused at the handover
  rather than dropped in silence.

• It fits an application that has a host. services.AddDeepSharpPipelines() registers the pipeline
  factory and the catalog of verbs, so a pipeline is resolved the way everything else in a .NET application
  is. A package that brings verbs of its own registers them as a contribution, and every contribution is
  applied whichever order the registrations happen to be written in. None of it is required: a console
  program that writes Pdd.Create() with no container anywhere works exactly the same, and a test holds
  that door open. Samples/DeepSharp.Sample.Pipelines shows both.