Skip to content

v3.0.0

  • v3.0.0
  • 32226e0
  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
  • Choose a tag to compare

  • v3.0.0
  • 32226e0
  • Choose a tag to compare

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
@linusnorton linusnorton tagged this 08 Sep 18:18
* Read a station's patterns when a query asks for them

The whole feed is 34 million patterns: half a gigabyte held and ten
seconds to read, before a single query. A planner answering a few of them
need not hold any of it.

npm run pattern-files writes a file per station, holding every pattern that
touches it. A pattern goes into both of the stations it runs between, the
second one turned round, so every line of a station's file begins with it
and a query only needs the stations it departs from. Filing it under one
end instead would have made the alphabet decide: a pattern is stored under
whichever end sorts first, so an origin sorting before its destinations
needs one file and one sorting after needs one per destination. Measured
over the London terminals, that is one file for AAP and eighteen for ZWT.
Both ends costs three times the storage, which is the thing there is most
of and the thing least read.

LazyTransferTreeRepository answers the same one method the tree does, and
gets a station's bytes from a PatternProvider: from a directory, or fetched
from a URL named for the station, which a browser and a cache can each make
sense of without knowing the format.

getPatterns cannot fetch, since nothing awaits it, so a repository that
does not hold everything is told the origins first and plan is a promise
again. I made it synchronous when nothing in the chain was asynchronous;
that stopped being true here. The alternative was leaving the caller to
prepare and returning no patterns when they forgot, which comes back as a
journey that does not exist rather than as a fault.

TransferTree is TransferTreeRepository, since there are two of them now.
PatternReader is back: it was deleted for having no callers and now has
one, in the station file writer.

On the GB feed, the same journeys either way:

  startup   10.6s -> 3.2s
  ZWT to the London terminals, 137 journeys, from one file

* Fix what the review found in reading a station at a time

prepare left the stations it already held where they were, so the map's
insertion order no longer said which had been asked for longest ago, and
forget dropped from the front. A query preparing four origins where three
were held dropped the three it was about to use, and getPatterns threw part
way through planning. It also meant a station asked for over and over was
still dropped on schedule, which is a cache in name only. Touching every
origin, held or not, fixes both, and the limit is a floor now rather than a
ceiling: a group larger than it is worth keeping for the length of a query
rather than failing.

A station being read is waited on rather than read again, so two queries
starting together fetch it once. Two queries still share what is held,
which is the point of holding it, and the class says so.

The station files took out nothing, so a directory holding the release
before it kept serving a station the feed had since dropped, as current.
This run's files stay and the rest go.

Neither provider looked at what it was given. A station is what names the
file, so one containing ../ read outside the directory and one that parsed
as a URL fetched another host. checkCodeWidths already says what a station
may be. A base without a trailing slash silently lost its last segment, so
it is refused.

The generator is published since #30, and the writer of the station files
was not in its index, which is the checkout it was meant to remove.

Also: the merge and the station files had copied the compression level and
the working file naming between them, comment left on one; both are in
PatternFormat now. An unused import got through because biome reports that
as a warning and npm test does not fail on warnings, so it is an error.
TransferPatternTreeNode had been caught by three renames in a row, none of
which meant it, and is PatternTreeNode. plan returns a promise and
TransferTree is gone from the exports, so 3.0.0.

Same journeys on the GB feed, 123 tests.

* Keep the file system out of the package root

DirectoryPatternProvider was exported from the root, which put node:fs and
node:path in the import graph of an entry that had none. Container says
why that matters and this made it untrue: a bundler resolves a node module
whether or not the import turns out to be reachable, and sideEffects: false
only helps the ESM path.

There is a src/node.ts now holding what reads a file system, which is the
container and this provider, and the ./node subpath points at it rather
than straight at the container. The root exports UrlPatternProvider,
LazyTransferTreeRepository and PatternProvider as before, which is the
whole of what a browser needs.

Traced from src/index.ts, the root reaches no node builtin again, as it did
before this branch.

* Say which two things read a file system, not one

* Forget a read of a station that failed

The promise for a station being read was taken out after it resolved, so
one that rejected stayed. Every later prepare then awaited it and rethrew
the first error without asking the provider again, and a station lost to a
503 was lost until the process went. Chaining the removal onto the read
clears it however it settles, and before the prepare waiting on it resumes.

It also leaves alone a read started while an earlier one was resolving,
which taking it out after the await did not.

This came in with the dedupe of reads in flight two commits ago.
Assets 2
Loading