Repository navigation
Live sources
A source that answers differently every time it is asked — an exchange's prices, say — cannot be read while a model trains. The pipeline would train on other numbers tomorrow and still claim to be the same pipeline, and that is the one thing PDD exists to prevent. So a live source is fetched once, over a closed window, and landed as a file; the pipeline then reads that file like any other, and the same pipeline gives the same numbers tomorrow.
DeepSharp.Pipelines.Binance does that for the candles of a market on Binance. It brings the HTTP, the retries and the
pacing, and nothing else does: the pipeline's file names a file and nothing of Binance, so a project that only serves a
trained model never carries any of it.
dotnet add package DeepSharp.Pipelines.Binance
A window is a symbol, one of the lengths Binance documents for a candle, and two moments:
using DeepSharp.Pipelines;
var from = new DateTime(2024, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var candles = new BinanceCandles("BTCEUR", "1d", from, from.AddYears(1));
var landed = await candles.LandAsync(SourceFolder.WorkingDirectory); // asks Binance once
Console.WriteLine(landed); // BTCEUR-1d-20240101T000000Z-20250101T000000Z.csvLandAsync asks Binance, writes the candles as a file in the folder it is handed and a record of the asking beside it,
and gives the name of the file, relative to that folder. Asking for the same window again asks Binance nothing: the file
is held to the fingerprint its record names and reused.
A pipeline written in code reads a relative path from the working directory, so the file is read like any other:
.ReadCsv(landed). The door does both in one line, landing the window in the working directory where it is not landed yet
and adding that read — awaited, since landing a window is waiting:
using DeepSharp.Pipelines;
var from = new DateTime(2024, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var candles = new BinanceCandles("BTCEUR", "1d", from, from.AddYears(1));
var pipeline = (await Pdd.Create().ReadBinanceAsync(candles))
.Declare(schema => schema
.Timestamp("timestamp")
.Number("close")
.Optional("trades", ColumnKind.Number))
.OrderBy("timestamp")
.SplitByTime("timestamp", train: 0.70, validation: 0.15, gap: 1)
.Ahead("close", 1, AheadAs.Return)
.Drop("timestamp")
.Normalise(scale => scale.Columns("close").MaxAbs("trades"));What the pipeline's file says of it is a plain read.csv step with the landing's name — the file stays at the version it
was and no verb was added. A program that serves the trained model reads that file with StepCatalog.BuiltIn() and
DeepSharp.Pipelines alone, with no HTTP, no retry and nothing of Binance anywhere in it. To land somewhere else — beside a
notebook, say — hand LandAsync that folder; a pipeline reads a relative path from the folder it sits in, as
Pipeline says.
The window is half open: the candles that open at or after from and before to. Each end must be where a candle of
that length opens, since Binance moves a start that is not forward to the next candle and a rounded window would stop
naming what it holds. An end that is not such a moment is refused, with the two nearest named.
| Interval | A candle opens at |
|---|---|
1s, 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 8h, 12h
|
a multiple of its length counted from midnight |
1d |
midnight |
3d |
midnight on every third day counted from the first of January 1970 |
1w |
midnight on a Monday |
1M |
midnight on the first of a month |
A moment that says nothing of its zone is universal, as a pipeline's own are; one that says it is local is the instant it stands for. A window ends where Binance has closed its last candle at least a minute ago, by Binance's own clock — a candle still open changes with every request — or it is refused before anything is spent, saying where the latest window of that length that can be landed now ends. A window that would take more than two thousand pages of a thousand candles is refused where it is written, naming how many pages it would take; land it in pieces, each ending where the next begins.
The file is named by the window, with no colon in it and a month spelled 1mo, because 1m and 1M are one name to a
file system that does not tell the cases apart. Beside it stands the record, ….manifest.json.
| Column | What it holds |
|---|---|
timestamp |
when the candle opens, in universal time to the second: 2024-01-01T00:00:00Z
|
open, high, low, close
|
the prices, exactly as Binance wrote them |
volume |
what was traded in the candle, in the base asset |
quoteVolume |
what was traded in it, in the quote asset |
trades |
how many trades there were |
takerBuyVolume, takerBuyQuoteVolume
|
how much of the volume was bought by takers, in each asset |
Every number is as Binance spelled it, because a row is known by what it says and a number written again is a different row. The bytes are fixed — UTF-8 with no mark, line feeds, one line feed at the end, the invariant culture — and a test compares them, since pipelines are declared against these columns. When a candle closes is the next one's opening less a millisecond, so it is not a column. A candle Binance does not have is counted in the record and left out; nothing is made up.
The record is one JSON object of version 1:
| Key | What it holds |
|---|---|
version, venue, host, endpoint
|
what wrote it, and where the pages came from |
terms |
where Binance's terms for its public data are written |
asked |
the symbol, the interval and the two ends of the window |
askedAt, serverTime, settleAfterCloseSeconds
|
when it was asked by this machine's clock, what Binance's clock said, and how long a closed candle is left to settle |
pagesEstimated, pages
|
how many pages the window was expected to take, and each page as it came: the moment it was asked from, its rows, the SHA-256 of its body and the Date and x-mbx-uuid it came with |
rows, gaps, first, last
|
what the file holds, and how many candles of the window Binance did not answer with |
file, fingerprint
|
the file's name and the SHA-256 of its bytes |
What changes from one fetch to the next — when, Binance's clock, the receipts — lives in the record and never in the file the pipeline reads, so the file's fingerprint names the candles alone. The record says nothing of being complete: the move of the file into place is what completes a landing, and the record is written after it. Nothing is written until the whole window has been read, so a landing that was cancelled or stopped leaves nothing behind.
| In the folder | What happens |
|---|---|
| nothing | the window is landed |
| the file and its record | the file's bytes are held to the fingerprint the record names and reused; Binance is not asked |
| the file's bytes are not the record's | refused, and left as they are: a pipeline fitted behind the first landing would have learned from other numbers |
| a record without its file | refused: nothing is fetched to take the file's place |
| a record of a newer version, one that cannot be read, or one of another window | refused, touching nothing |
| a file without its record | landed again, as when a program stopped between the two writes; the record is written only if Binance said the same, and other numbers are refused with the first landing left in place |
A landing is never replaced, and no option asks it to be. Of two landings of one window at once, one stands and the other finds the same bytes or is refused: within a process on every system, and between processes where the system refuses a move onto a taken place in one step, as Windows does. On Linux and macOS .NET looks at the place and then renames, so a landing replaced all the same is caught by its record's fingerprint, and the next landing refuses it.
Binance counts what an address spends by the minute, and every other program on the address spends from the same count. A landing is a one-off with no deadline, so it yields by construction:
- no two requests are closer than half a second, every try included;
- the weight the address has used is read from every answer, and the landing waits for the next minute once that is half of what the minute allows — whoever used it. That wait is no part of a try: the ten seconds a try may take begin when its request is sent, so a minute spent waiting uses up none of the five tries;
- a busy answer or a fault of Binance's, a try that does not answer in ten seconds and an answer that is not a page are tried again behind a wait that doubles from a second, as long as Binance asks when it asks and never past two minutes; a unit of work is tried five times and fails once;
- a firewall (403), a ban (418) or a region Binance does not answer from (451) stops the landing at once, naming how long when Binance says; a wait longer than two minutes is a stop too;
- a refusal in Binance's own words is carried in the exception and the request is not asked again.
| Thrown | When |
|---|---|
ArgumentException |
a window that cannot be: a symbol or an interval Binance does not know, an end off the grid, a window that does not end after it begins, or one of more than two thousand pages |
DirectoryNotFoundException |
the folder is not there; nothing is asked of Binance until it is |
InvalidOperationException |
the window ends where Binance has not closed its last candle, or Binance holds no candle of it |
BinanceException |
Binance stopped the landing, asked for a longer wait than a landing makes, refused it in its own words, or could not be made to answer; Status and Code carry what it said |
InvalidDataException |
what stands in the folder cannot be reused, or Binance now answers otherwise than the file that stands there says it did |
Through, On and At choose how a window is sent and never what it names; each gives another window of the same
candles. Through(handler) is the handler the requests go through — a proxy's, or one a test hands in — and it is yours:
a landing never disposes it. On(clock) is the TimeProvider every wait, every timeout and every stamp runs on, so a
test moves a clock and waits for nothing. At(address) is another address than Binance's public market data, an http
or https address of a host alone — a stand-in on this machine, say. There is no credential to give it, and none is
taken: the package asks for public data and nothing else.
Nothing of this. The assemblies a model server references — the pipeline library, the tensors, the network's learner, the
declaration of the ML.NET one — name neither HTTP nor Polly nor Binance, and a test holds them to it. A check installs the
packages as a person would and runs an application twice: once landing a window from a stand-in venue on the machine, and
once reading the pipeline that landing wrote with DeepSharp.Pipelines alone, where neither the package nor Polly is
loaded. Neither reaches Binance, and nothing in the tests does: the suites that run documents send every address but this
machine's to a proxy nothing listens at, and a program in a document that a suite runs may not name the landing.
What lands is Binance's data, under Binance's terms, which the record points to and which are for whoever lands the data to read before using it or passing it on. This repository holds none of it — a test or a sample that wants candles generates them — and the package is not affiliated with Binance.
- An account. No credential, no signed request, no order: a key would make it a client of somebody's account, which is another library.
- Another venue. Binance's wire format lives in one internal class and nothing is shaped for a second; the second source, when there is one, shows what the two share, and it will be a package of its own.
- A stream. A landing is a one-off over a closed window; live names the source, not a connection kept open.
-
The archives. Binance also publishes whole files of history, a day or a month at a time, at
data.binance.vision, which is the better way to years of one-second candles; this pages the public market-data interface and refuses a window that would take more than two thousand pages.
Pipeline has the readers beside this door, and Roadmap says where it stands among the rest.