Skip to content

v2.0.0

v2.0.0: 2.0.0: read transfer patterns from a file, in node or the browser
@linusnorton linusnorton tagged this 07 Sep 20:36 
Replaces the MySQL transfer pattern table with the file raptor writes,
which is one line per pattern, front coded against the line before it
and brotli compressed. A national feed is 33MB for 34 million patterns,
where the database it replaces was the one thing standing between this
and running anywhere.

Loading:
 - loadTransferPatterns takes whatever the environment can give bytes
   from, the same sources the feed loader takes: a Response, a stream,
   a Blob, the bytes, or a node stream
 - loadTransferPatternsFromUrl fetches and reads as it downloads, so a
   feed and its patterns can be fetched at once and each is parsed while
   the other is still arriving
 - decompression is DecompressionStream("brotli"), which node and the
   browsers both have, so there is one path for each and nothing in it
   names a node module. A host that sends Content-Encoding: br has the
   body decoded before this sees it, which is noticed from the header
   rather than decompressed twice
 - createQuery is the whole of the wiring between having a feed and its
   patterns and being able to plan

The index holds the stations between a pattern's two ends packed into
one string rather than an array, since at 34 million patterns an array
each costs more than the stations in it.

Breaking:
 - Container moves to transfer-pattern-planner/node. It reads paths off
   the file system, and a bundler resolves what it is pointed at whether
   or not the import turns out to be reachable, so naming node:fs in the
   package root made every browser bundle configure around it. The root
   is now the same code in either place
 - DatabaseTransferPatternRepository, the mysql2 dependency, the
   DATABASE_* variables and docker-compose.yml are gone. The patterns
   are named by TRANSFER_PATTERNS
 - InMemoryTransferPatternRepository takes the packed index
 - the exports map nests types inside each condition, so a CommonJS
   consumer resolves dist/cjs declarations rather than the ESM ones and
   no longer gets TS1479

Also fixes TransferPatternFactory throwing on an origin and destination
with no pattern between them, which a database usually had something for
and a file routinely does not, and stops it appending the destination to
the caller's array.

Claude-Session: https://claude.ai/code/session_011JPQTHyL1kY4FgZkX2uU1y
Assets 2
Loading