Repository navigation
v0.15.4
Faster bundle imports. A backfill spent nearly all its time sleeping: it paces at half
the lower of the namespace and principal rates (25 items/s at default limits), 40 s per
1,000-item batch, against about half a second of work.
- Under an allowance (§6.6) the importer now paces at the allowance's full rate, since its
bucket holds up no other writer, and splits batches by itsitemsPerBatch/batchSize. It
finds its own allowance as the core does (the grant's rootsubandkid, or the author
with authentication disabled), counts every draw (dry runs, uploads, copies) so it doesn't
run into 429s, and from a minute before the allowance'suntilgoes back to the
namespace's limits. With an allowance of 1,000 items/s, 2,000 documents wait 3.5 s instead
of 80 s; 145,000 would take minutes instead of over an hour. Without one, pacing is as
before, less the time each batch took. Progress lines say what the import is paced by. - Planning reads target heads from
/headspages instead of oneGETper document
(twice in snapshot mode): 0.005–0.014 requests per document instead of 1–2. A small import
into a large namespace keeps per-document lookups. BenchmarkImport(internal/bundle) measures an import's work, requests per document and
the time it would sleep.
The client's requests use a connection pool of their own, a clone of
http.DefaultTransport, so code that closes the default one's idle connections doesn't
break them.
Full Changelog: v0.15.3...v0.15.4