Skip to content

v0.0.2

Choose a tag to compare

@GolyBidoof GolyBidoof released this 25 Sep 10:18
· 12 commits to main since this release

v0.0.2: early, potentially buggy, and not ready for large-scale use.

This release is mostly about speed, and two of the fixes are the same bug wearing different hats: an option that parses, shows up in --help, passes its own unit test, and never actually reaches the engine. Both survived because the tests covered parsing and never the wiring, which is a mistake I have now written a test for.

The big one was in the vendored bridge client. It put a 60 ms floor between consecutive request starts, and since that floor was global its cost scaled with the page count instead of with request latency. Page uploads were capped at 16.2 per second for the whole process, which meant 15 seconds of dead time before OCR could even begin on a 244-page volume, and around four minutes across a 16-volume batch. It does 178 pages per second now, about 11x faster, and that number was measured with a 1768 KB page, which is bigger than a real page, so it is the conservative end. The floor only ever applied to page uploads, so it is gone rather than made optional. The retry backoff and the bridge's own admission control are untouched.

Fixed:

  • --eb-concurrency did nothing. The driver handed all three stores the CMOA value, so ebookjapan quietly ran at 12 pages at a time instead of the 32 the help text promised. Each store now has its own key and each adapter reads its own.
  • CMOA rendered an entire volume on a single thread. A missing --jobs was passed through as undefined, which fell into a jobs = 1 default inside the renderer, so a 244-page volume spent about 44 seconds rendering instead of about 9. The pool is now sized from core count and divided across the volumes actually running at once, so a 16-volume batch does not start 16 separate pools.
  • Zip CRC-32 was computed one bit at a time on the main thread after every page had finished downloading. It uses zlib.crc32 now, with a table-driven fallback, which takes 110 MiB/s to about 16 GiB/s.

Changed:

  • BookWalker's encrypted pages are unscrambled and re-encoded on a small worker pool instead of on the event loop. The output is byte for byte identical, the pool is lazy so a plaintext volume never starts a thread, and it measured 2.7x faster on four workers.
  • The vendored BookWalker sampler accepts an optional normalizePage hook. Nothing changes if you do not pass one.

Still true, and worth repeating: this is a 0.0.x. It is verified against a small number of volumes, error handling on unusual input is thin, and the three stores can change their viewers without warning at any time. Please do not point it at a large queue yet. Start with one volume, check the output, and work up.

The download side is now much faster than it was, which mostly means the OCR step is even more obviously the bottleneck, so that is where the next round of work goes.

160 checks across 6 files, plus 11 opt-in live checks and 6 vendored engine tests.