Repository navigation
Releases
v1.1.0
DMMブックス is the sixth store. dokuha now downloads its free full volumes and 試し読み samplers through the same browserless viewer protocol as the other five stores, walks a DMM series from its product pages, and keeps the resume behaviour the rest of the CLI has. DMM pages arrive as a 64×64 tile shuffle, so sharp rebuilds them and --no-descramble is the explicit way to keep the shuffled bytes. Paid and library titles have no cookie path here, so this release covers the free volumes and the samplers.
Added
- DMMブックス: free volumes and 試し読み samplers. A pasted
free_streaming/viewer URL downloads a volume that is free in full for its limited window, and a pastedtachiyomi/URL downloads that one sampler directly, with no--seriesand no listing request. The adapter calls/viewerapi/auth/for the short-lived session, sends the free route's CloudFront values in theCookieheader and the sampler'sauth_infoas signed query parameters (decoded once soURLSearchParamscannot double-encode the signature), reads the plainconfiguration_pack.json, fetches every JPEG, and rebuilds each 64×64 tile shuffle withsharp. A 403 on a page or on the manifest refreshes the auth once and retries, and 429 and 5xx responses retry with the page's own budget. - DMM series and sampler walks.
--serieson a DMM product page,/volumes/listing or/latest/reads thecontents_bookBFF listing, keeps the volumes with afree_streaming_urland stops only when the pager'stotal_countis reached, failing loudly if it sees fewer volumes than the store says exist.--download-samplersadds each remaining volume'stachiyomipreview. A DMM series page pasted without--seriesis refused with a message rather than quietly taken as one volume. --dmm-concurrencysets how many DMM page fetches are in flight, default 16 and maximum 32. It is a per-store flag beside the other stores' and--concurrencystill overrides it.- DMM page and manifest requests go over HTTP/2. Node's global
fetchspeaks HTTP/1.1 only, and the DMM CDNs serve HTTP/2. The newsrc/download/h2-fetch.jsmultiplexes DMM page and manifest requests over one HTTP/2 session per CDN origin, falls back to the globalfetchwhen an origin negotiates HTTP/1.1 or an h2 request fails, and keepsAccept-Encoding: identitybecause it does not decompress. The gain is latency rather than bandwidth: on a link with about 300 ms round-trip time, eight concurrent HTTP/1.1 page requests reached about 68% of a 10.6 MB/s link and HTTP/2 reached about 99%, taking a 232-page volume from about 27.5 s to about 19 s. Below about 100 ms RTT the difference disappears, so this is a high-latency win and not a general speedup. - Resume keeps the descramble state honest. A DMM folder marker records whether its pages were descrambled and which shuffle version rebuilt them. When a run's requested mode disagrees with the marker, the pages are downloaded again instead of being reused as if they were the requested kind; a completed folder with the matching mode still resumes without a page request.
Fixed
- A DMM sampler folder no longer gets two
(試し読み)suffixes. The display title and the folder helper both added the suffix, which produced a path such asタイトル(試し読み)(試し読み); the folder helper now adds it once. - A 403 on
configuration_pack.jsonwas a hard failure. It is the same signature rollover as a page 403, so it now refreshes the auth once and reads the manifest again; the refresh is shared by the manifest and every page worker, so a rollover costs one/viewerapi/auth/call rather than one per page. - DMM page attempts now follow
--retries. The adapter used a fixed three attempts, so--retrieswas ignored for this store. It now drives the per-page budget, and the single 403 refresh stays outside that budget because the retry runs against the replacement session. - A short DMM series walk now fails instead of looking complete. The listing is read until
pager.total_countis reached; a walk that sees fewer volumes than the store reports is an error naming the count, rather than a series silently truncated to its first page.
Changed
- DMMブックス is the sixth store in the docs, the help text and the bench.
--help, the store matrix, the URL and session tables, the series walker, the naming rules, the phase notes indocs/performance.mdand thebench:phasesstore list now name DMMブックス.tools/bench-phases.mjscarries a DMM entry whose viewer URL has to be pasted in first because thecidexpires. - The stores and troubleshooting pages were corrected where they described an earlier draft. A missing
sharpis documented as a hard error rather than a shuffled page (--no-descrambleis the opt-in), the 403 refresh is documented for both a page andconfiguration_pack.json,--dmm-concurrencyis separated from the--parallelvolume count, and a pastedtachiyomi/URL is documented as a single sampler that needs no flags. - The discovered suite is now 33 files, 32 of them offline.
tests/test_dmm.mjspins 119 checks on its own: the URL routes, CloudFront signing, the manifest, the 64×64 geometry, descrambling, the sampler, the 403 refresh,--retries, the descramble marker and page order.tests/test_h2_fetch.mjsadds 70 checks for the HTTP/2 transport and its HTTP/1.1 fallback. The one live network check still needsDOKUHA_TEST_NETWORK=1.
v1.0.0
The first stable release. dokuha-cli is the browserless batch downloader of the dokuha project: five stores through their own viewer protocols, series downloads that find the free volumes themselves, a live progress block, and an optional hand-off to mokuro-bridge for OCR and delivery. This release collects everything since 0.1.1 and settles the interfaces that scripts can rely on: the --json summary, the CLI options, the exit codes and the bridge client contract.
Added
- The live block can be paused so its text can be selected and copied. The block repaints every 120 ms, moving the cursor and rewriting the same rows, which is why a drag-selection jumps while a download is running. Pressing
pfreezes every paint; the run keeps going, progress updates keep accumulating, and notes that arrive meanwhile are held. Pressingpagain paints the current state, prints the held notes, and resumes. A run that ends while frozen prints the held notes and its final frame rather than leaving the terminal frozen. The footer advertisesp: pauseandp: resumebesidec: compact, and--no-progressremains the fully static alternative when stdin is not a terminal. - The live block shows cumulative sizes. A done row already keeps the manga's downloaded size; the footer now totals it across every volume (
149.4 MiB manga), and a bridge run adds the size of webp, cbz, mokuro and series.json as the upload stream reports them, on a third line that appears only when a bridge produces output. Sizes are file totals, not deltas, and the per-seriesseries.jsonis counted once rather than once per volume. The third line is reserved in the row budget so a full-height volume list does not push the block off the screen.
Fixed
- Remote destinations lost the artifact sizes. The terminal
done.uploads[]the bridge builds for a remote destination carriesbytesandtotal_bytes, while the local one carriessize; reading onlysizemade every remote entry non-finite, so the footer's third line (webp, cbz, mokuro and series.json) disappeared on exactly the runs that upload, andseries.jsonhad no other event to recover it from. Both spellings are accepted now, a local frame keeps itssize, andmokuro.outputFilesexposes one normalizedsizeper file in--json. --forcesent a collision policy the bridge does not understand. Finalize takesfail(the default),skiporoverwrite; the CLI sent the booleantrue, which providers treat asfail, so a forced re-run uploaded the artifacts and then died with "destination already exists" after the OCR work had already been spent.--forcenow sendsoverwrite=overwrite, and it also bypasses the local completion receipt, which previously made a forced run skip the hand-off it was asked to repeat.- A terminal
partial_uploadwas reported as a success. That frame means the destination received the volume files but notseries.json, so the reader cannot build the series; the bridge's own client treats it as a failure, and the CLI now does too: the volume is counted as failed, the run exits 1, the bridge's reason is printed with a--forceretry hint, andmokuro.status,mokuro.messageandmokuro.partialare in--json. - A finished bridge session could fail a volume needlessly. A persisted session record that reached its terminal state answers
GET /session/{id}/statuswith 400 "Session already finalized" (a 404 is the normal "finished and cleaned up"), and only the 404 was handled, so a re-run could fail on work that was already done. Both are a skip now, and so is a finalize that answers 400 because another client finalized first;status()errors carry the bridge's response body so the two cases can be told apart. --destrejected historical spellings after an account was renamed. Resolution matched the exact id or the derivedprovider-name, but the bridge derives that alias from the storage name only until an account is renamed, and it turns underscores into hyphens. Resolution now also matches theproviderandaccountpair/upload-methodsreports, somega:workstill finds a renamed account andmega:my_workfindsmega-my-work. A bare provider id is resolved as an alias first, which is what the bridge does.--dest-folderwas still resolved by the bridge on the resume-only path. The--mokuropath resolved a relative folder on this machine, but the--no-mokurosession-resume path sent it raw, so the bridge resolved it against its own working directory.resolveDestinationnow resolves it once for both callers.- A finished early cover push painted the footer as uploading. The bridge keeps an
uploadobject on the session after the early cover upload withactive: falseuntil the real transfer starts; the status monitor counted the object instead ofactive, so the footer's "1 uploading" and the per-volume "bridge uploading" outlived the cover for the whole OCR phase. - A failed volume could still make the run read 100 percent. A volume that failed after 20 of 200 pages was weighted as a whole finished volume, so the footer could show 100 percent and a full bar beside a red
failedrow and0/1 done. A failure now carries only the fraction it actually reached, matching the meter the row already drew. - The footer percentage and its bar could disagree for a whole run. A plain download was still weighted with the mokuro stages (download 30, OCR 45, upload 25), so a run that fetched all 297 pages read 56 percent with a full blue bar, and a resumed 5-volume run showed 40 percent after six seconds.
pipelineExpectednow tells the block whether a bridge follows; without one, the pages are the whole measure and the bar fill and the percentage share one basis. A bridge handoff that starts on a resumed session turns the staged weights on at the moment it happens. - ebookjapan and BookWalker's local finalize looked like a bridge phase. Their local
finalizingstep (zipping pages after the download) switched the block to the mokuro weights, which made an ebookjapan download-only run report 88 percent beside a 98 percent bar, and stall at 51 percent while the bar ran to 89. Onlyuploadingandocrcount as bridge phases now. - The footer percentage could step backwards. A queued volume's page total grows the denominator before its numerator, so the number dipped by up to 2.4 points. The display keeps a running high-water mark, and never shows 100 percent until every row has settled (a fully fetched run reads 99 percent until the last volume reports done).
- A local destination never showed the assemble band, and a done volume without upload bytes stayed cyan. The cyan band required a remote destination, so a local bridge run showed a full blue bar beside rows assembling the archive, and a skipped or already-present volume left the whole bar cyan at 100 percent. Assembling is a pipeline stage for every bridge run, and a done volume is green.
- A finalizing row could show a full bar while the destination was still confirming the transfer. The bridge's upload percentage can reach 100 while MEGA is still closing the file, and the row then drew a full meter. It now stops one cell short until the volume is actually done, which is what the 0.1.1 changelog already promised.
- A finished mokuro volume left the footer short of OCR'd pages. The last wait_ocr frame can still carry one page pending, so a five-volume run finished with
275 OCR'dfor 297 pages.finish()now normalizes the OCR counters. - BookWalker rows said
waitinguntil the first page landed. Every other store claims the download phase when the page total is known; BookWalker was the only one waiting for the first completed page, so its download bar appeared a second late on a sampler and longer on a small volume. - A BookWalker page failure threw instead of counting. The adapter incremented
progress.get(key).failed, butLiveProgresshad noget, so the error path raised a TypeError instead of reporting the bad page. - Series rows showed store IDs instead of titles during the handshake. A series walk already knows each volume title, but the row label preferred the id, so BookWalker series rows were UUIDs for the first eight seconds of the audit run.
rowLabelnow preferstask.title. - Rows wrapped on terminals narrower than about 48 columns. The fixed columns plus the minimum label width overflowed, and a wrapped row makes the redraw clear one line too few. A row is now clamped to the terminal width.
- The bridge connect failure path called a
progress.stopthat does not exist. It was a guarded no-op, so a failed bridge left the ticker running until the process exited; it now callsstopTicker(). - The duplicate-session recovery threw a
ReferenceError. When a bridge answered a start with "already finalizing" and the follow-up session check also failed, the skip path callednoteResume(ctx, ...)wherectxwas never a parameter, so the volume failed with a nonsense error instead of being skipped as a duplicate. The caller's context is now passed in. - The same volume could be downloaded twice into one folder. A repeated URL, or a series URL plus one of its own volume URLs, created two tasks with the same key; both writers raced on the same page files and completion marker, and the summary counted the volume twice. Tasks are deduplicated by key before dispatch.
- A normal run did not say why an input was skipped. The rejection list was printed after the build promise was awaited, but with the default streaming mode that promise settles after the scheduler starts, so the list was still empty. A typo in a batch was silent and the exit code was 0; the reason now prints when discovery rejects the input.
- The grouped Kindle "loan could not be returned" line could be missing.
kindleReturnWorkwas created on the per-volume child context, so the run's...
dokuha-cli 0.1.1
Kindle series discovery is one request for the whole series, with the access, offer and sampler probes run at once. Progress reporting fixes, and documentation corrected where it described the removed walk.
v0.1.1
Added
- Kindle series discovery is one request, not one per volume. A series list comes from the store's
seriesAsinListendpoint at a page size of 100, so a 72-volume series arrives in a single call, and the per-volume access, Kindle Unlimited offer and 試し読み probes then run at once instead of in a chain where each answer named the next volume. Discovery for a nine-volume series went from roughly twenty seconds to about ten, with the probes themselves at one second. The collection ASIN the endpoint needs is read from the Kindle product page, and when a volume has no such page, which is the case for Kindle Unlimited only titles, it is found through a related volume instead of being treated as a series with no volumes. A discovery that fails outright is now reported rather than falling back to downloading a single volume, which previously hid a nine-volume series behind one book. - BookWalker pages are named after the manifest source they came from. A page written from
OEBPS/text/cover.xhtmlis now0001 cover.jpginstead ofpage-0001.jpg, in the folder, in the filename sent to mokuro-bridge, and in the ZIP entry. The stem is sanitized for every target filesystem and truncated by encoded length so the whole entry fits 255 UTF-8 bytes. The four-digit ordinal stays first, so a plain name sort is still reading order. Ported from the userscript repo's PR #1; the naming policy lives insrc/download/page-files.jsand reaches the engine as the optionaloptions.pageName(index, source, extension), which defaults to the old name. - A newer release is announced, and
--updateinstalls it. After a normal rundokuhaprints one line to stderr when npm has a newerdokuha-cli, at most one registry request a day, cached under~/.cache/dokuha-cli/. The check starts before the download and is read after it with a short bound, so it never delays a run, and every network failure is silent. It stays off for--json,--quietand any non-terminal stderr, where a script may be parsing output;DOKUHA_UPDATE_NOTICE=1opts a non-terminal back in andDOKUHA_NO_UPDATE_CHECK=1disables the notice entirely.dokuha --check-updateasks immediately, bypassing the day-long cache, and exits 0 when current, 1 when a newer release exists and 2 when the check failed.dokuha --updateprints and then runsnpm install -g dokuha-cli@latestfor a global npm install, and refuses with one line when the running copy is a git checkout or a local path install, including annpm link, where that command would upgrade the wrong thing. No new dependency: the check uses the runtimefetchagainsthttps://registry.npmjs.org/dokuha-cli/latest, and the comparison tolerates a leadingvand prerelease tags.
Fixed
--download-samplersdid nothing on k-manga. Its resolver read aconfig.samplersthat is never set, while every other store readsctx.samplers, so the flag was silently ignored for that store alone. It now lists the free volumes and the 試し読み previews.--dest-folderwas resolved against the bridge's directory. A relative path landed next to mokuro-bridge, not next to the command that was run. The CLI now resolves it on this machine, like--out.- A mistyped
@filecookie failed one volume at a time.--bw-cookieand--kindle-cookienow check that an@FILEexists while parsing, so a typo is one message at startup instead of a failure per store. - A Kindle Unlimited return could fail on the first lookup. Amazon's
your-bookslisting lags the borrow, and one lookup on the first miss marked a finished volume as failed and left the loan open against the six-slot ceiling. The lookup retries with a fresh listing. - ebookjapan pages came out as a shuffled mosaic. The viewer WASM emits each page's tile coordinates in the coordinate space of the box asked for by
open_param, and the CLI read those coordinates out of the CDN image, which is served at the page's intrinsic size. Asking for a downscaled box (size: 1200) made every source offset land in the top-left corner of the image, so every rebuilt page was a patchwork of unrelated fragments. Asking for the intrinsic box fixes it, the same volume that took 8 minutes and produced unreadable pages now takes 26 seconds and produces clean, full-resolution pages. Volumes already downloaded this way look complete to the resume gate; re-fetch them with--force. - A resumed volume trusted a truncated page. The whole-volume "already present" gate counted image files and compared the count to the expected pages, so a half-written
page-0002.jpgmade the folder look complete. Every store now validates each page with one shared completeness check (isCompleteImageFile), and a skipped volume reports the bytes actually on disk instead of0. This is what the four failing Kindle resume checks were reporting. - A URL could be handed to the wrong store. Detection tried every store's unanchored pattern in order, so a link to Amazon inside an ebookjapan URL's query string resolved to Kindle. An absolute URL is now claimed by the store that owns its host before any pattern is tried, and an ebookjapan homepage or account page is no longer displayed as a downloadable volume.
- Kindle
--seriesdropped the session. The registry callsseries(det, input, ctx)but the walk expected(det, config, io), so the config was a URL string and--kindle-cookiewas ignored on that path. The descriptor now adapts at the seam. - Skip paths bypassed the result contract. The "already present" and Kindle Unlimited skip paths did not go through
normalizeStoreResult, and the KU path setskippedtotruewhere every other result uses a page count. FORCE_COLOR=0forced colour on, andNO_COLORoverrodeFORCE_COLOR;FORCE_COLORnow wins and0means off.--dry-runexited 0 with nothing to do while the same run without it exited 2. The exit code no longer depends on a dry run.- A missing option value leaked Node's wording (
argument missing); it now names the flag, and a misspelled flag suggests the real one. - Documented ranges were not enforced:
--jpeg-qualityis 1-100 and--km-concurrencyis 1-32. - A fatal error printed a raw stack. A full disk, a permission error, a refused connection, an unknown host and a reset connection now each get one actionable sentence naming the likely cause.
Changed
-
The overall progress bar weights each volume by its page count. Counting every volume equally meant a twelve-page sampler finishing advanced the bar as far as a two-hundred-page volume finishing, so a run could look nearly green while the real work had barely started. A volume completed in an earlier run now also counts as downloaded, which is what keeps a resumed run's bar green rather than dyed cyan from the assemble band alone.
-
The row bar stops one cell short while finalizing. A full bar beside an upload at zero percent read as a finished volume.
-
One shared "already present" gate decides whether a volume is done.
src/download/existing-volume.jsruns ahead of every adapter's page loop. It validates each image with the sharedisCompleteImageFilecheck rather than counting files, sums the bytes actually on disk, and can ask the bridge whether the volume already exists at the destination, so a repeat run skips both the download and the OCR. -
Four dead flags removed.
--title-dir(a no-op alias),--kindle-max-volumes(parsed, read nowhere),--bw-login(always refused),--local-dir(an alias of--dest local --dest-folder DIR). -
--overwritemerged into--force, which already meant "ignore what is already there". -
--descramblewith--no-descrambleis now an error instead of silently letting--descramblewin regardless of order. -
--helpis two-tier: the dozen flags a normal run needs, then an Advanced section for store-specific and tuning options. -
The bridge wait loops are bounded by
--ocr-waitinstead of polling forever, and the completion marker is a size/mtime signature rather than a SHA-256 of every page, so a resumed run no longer reads the whole volume back off disk. -
The summary now separates download time from OCR time, so a long
--mokurorun is not mistaken for slow downloading, and it names how many URLs were not recognised instead of leaving that to the lines above. -
src/detect.jsdeleted;src/platforms/index.jsis the only detection entry point. -
--quietand--no-progressstay separate.--quietsilences the run;--no-progressreplaces the live block with one plain line per volume. They were reviewed as a merge candidate and kept, because merging them would have removed a behaviour rather than a duplicate.
v0.0.9: five stores, a platform registry, and CI
A worker crash could hang the whole run, --km-concurrency never reached the engine, and --no-title-dir let a sampler overwrite the volume it belongs to.
Fixed
- A crashed render worker hung the entire run. Both worker pools handed the next page to a worker that had already died, and
postMessageto a terminated worker is a silent no-op, so that page's promise never settled. Because the pool is closed in afinallythat runs after the downloads are awaited, one crash meant the CLI produced no output and never exited; aftersizecrashes every remaining page did the same. A dead worker is now dropped instead of reused, the pool refills itself, and anexitwithout amessagesettles the in-flight job rather than parking it forever. --km-concurrencydid nothing. It was declared, shown in--help, read by the driver and consumed by the k-manga adapter, butparseOptionsnever copied it into the config, so the documented default of 16 was really a hardcoded 8. The five per-store page counts are now followed from declaration through the parse to the store context in one test, so the next dead flag fails the suite rather than shipping.- A sampler overwrote its own volume under
--no-title-dir. A sampler and the full volume share one content id, so naming the folder after the id sent both to the same directory, where they overwrote each other page for page. CMOA, BookWalker and Kindle were affected; k-manga already carried the sample suffix. OnevolumeFolderhelper now decides this for every store. - A store-supplied title of
.or..wrote outside--out. The ebookjapan sanitiser stripped separators but not trailing dots, so the path resolved to the parent directory. It now uses the same sanitiser as every other store, which also removes a second, looser copy of the rule. - A re-run re-downloaded every descrambled ebookjapan page. The "is this page already on disk" check looked for WebP magic regardless of the format actually written, so with descrambling on -- where the pages are JPEG -- it never matched, and the whole volume was fetched and re-encoded again without
--force. - A page the descrambler could not rebuild was reported as a success. When the WASM glue used a canvas call outside the known set, the composition returned no plan and the raw scrambled bytes were written to a
.jpgpath and counted as downloaded. It is now a failed page. - One flaky probe aborted a whole CMOA scan. Probing deliberately rethrows anything that is not "this volume does not exist", so a single 429 or 500 while scanning killed the run even when every other URL was fine. A thrown probe now counts as a miss.
- A volume whose page uploads mostly failed was reported as a success. Upload failures were recorded but never affected
ok, so a 200-page volume with 199 bad pushes exited 0 with a.mokurobuilt from the one page that arrived. - A finalize stream that ended without a
doneframe counted as a success. The vendored BookWalker client throws on that input and the ebookjapan client did not, so the same truncated upload was judged two ways. --parallel Nran up to2Nvolumes. Each scheduling lane built its own pool ofparallelworkers, which contradicts both the help text and the number the summary reports. Both lanes now draw from one gate.<link rel="next">on a BookWalker series page is now resolved against the page URL; a relative href used to reach the fetcher as-is and fail to parse.- A series page fetch that answers non-200 now releases its socket instead of leaking one per page.
Changed
- The stores are a registry, not a set of branches. Each store owns a descriptor: the URLs it answers to, its progress label, the lane it runs on, how a title URL expands, its series walk, and its download.
run.jsdispatches through that registry, so adding a store is one descriptor rather than a new branch in the driver, a switch in the adapter, and an entry in three tables. - Folder naming, the volume marker and the shared scrapers moved out of the CMOA adapter, which every other adapter had been importing from.
- The ebookjapan viewer's WASM module and glue chunks are no longer vendored. They are ebookjapan's own proprietary build output and redistributing them under this project's MIT licence is not ours to do. They are now fetched from the store on first use and cached under
~/.bwdd-cli/ebookjapan-assets/(or$BWDD_EBOOKJAPAN_ASSETS);vendor/README.mdhas the refresh procedure.
Notes
- The tests that pinned the wiring by grepping the source have been replaced with behavioural ones. A check like "does
run.jscontain the textretries: config.retries" cannot tell a working build from a renamed variable, and it fails on any refactor; the same guarantee is now expressed by running the code.
v0.0.2
v0.0.2: early, potentially buggy, and not ready for large-scale use.
This release is mostly about speed, and two of the fixes are the same bug wearing different hats: an option that parses, shows up in --help, passes its own unit test, and never actually reaches the engine. Both survived because the tests covered parsing and never the wiring, which is a mistake I have now written a test for.
The big one was in the vendored bridge client. It put a 60 ms floor between consecutive request starts, and since that floor was global its cost scaled with the page count instead of with request latency. Page uploads were capped at 16.2 per second for the whole process, which meant 15 seconds of dead time before OCR could even begin on a 244-page volume, and around four minutes across a 16-volume batch. It does 178 pages per second now, about 11x faster, and that number was measured with a 1768 KB page, which is bigger than a real page, so it is the conservative end. The floor only ever applied to page uploads, so it is gone rather than made optional. The retry backoff and the bridge's own admission control are untouched.
Fixed:
--eb-concurrencydid nothing. The driver handed all three stores the CMOA value, so ebookjapan quietly ran at 12 pages at a time instead of the 32 the help text promised. Each store now has its own key and each adapter reads its own.- CMOA rendered an entire volume on a single thread. A missing
--jobswas passed through asundefined, which fell into ajobs = 1default inside the renderer, so a 244-page volume spent about 44 seconds rendering instead of about 9. The pool is now sized from core count and divided across the volumes actually running at once, so a 16-volume batch does not start 16 separate pools. - Zip CRC-32 was computed one bit at a time on the main thread after every page had finished downloading. It uses
zlib.crc32now, with a table-driven fallback, which takes 110 MiB/s to about 16 GiB/s.
Changed:
- BookWalker's encrypted pages are unscrambled and re-encoded on a small worker pool instead of on the event loop. The output is byte for byte identical, the pool is lazy so a plaintext volume never starts a thread, and it measured 2.7x faster on four workers.
- The vendored BookWalker sampler accepts an optional
normalizePagehook. Nothing changes if you do not pass one.
Still true, and worth repeating: this is a 0.0.x. It is verified against a small number of volumes, error handling on unusual input is thin, and the three stores can change their viewers without warning at any time. Please do not point it at a large queue yet. Start with one volume, check the output, and work up.
The download side is now much faster than it was, which mostly means the OCR step is even more obviously the bottleneck, so that is where the next round of work goes.
160 checks across 6 files, plus 11 opt-in live checks and 6 vendored engine tests.