Skip to content

perf(core): optimize async operation & progress events #23

Merged
ido-pluto merged 63 commits intobetafrom
feat/speed-ipull
Apr 22, 2026
Merged

perf(core): optimize async operation & progress events #23
ido-pluto merged 63 commits intobetafrom
feat/speed-ipull

Conversation

@ido-pluto
Copy link
Copy Markdown
Owner

@ido-pluto ido-pluto commented Mar 15, 2026

Description of change

Refactor to some core functionality of ipull that improves download speed and make ipull more light wight.

  • Replacing pretty-bytes & pretty-ms with an optimize version, that lower the cpu load by ~35% (!!!)
  • Optimizing disk writing with queue & debounce to lower the event loop load & context switch ~60% IPull 4.0.3 is heavy on the event loop #19
  • Debouncing event at the network file-engine level ~10%
  • Cacheing engine CLI calculation ~5%
  • --save to directory is hang and saves to current directory #21
  • Other bug fixes
  • Event order and status flags for Existing (skip existing) and Cloned (file cloned). From now on start and finish events will emit for those edge cases.
  • Pause now will close the connections and resume will reopen them.
  • Better TypeScript hints for events.

All the performance benchmarks can be found here

https://ido-pluto.github.io/ipull-speed-test/

https://github.com/ido-pluto/ipull-speed-test

Breaking chage:

  • '-st' shortcut is not supported yet in the cli, use '--style' instead.
  • Some types has been change has I bump deps
  • Auto file name functionality has been change to better capture the file name

Pull-Request Checklist

  • Code is up-to-date with the main branch
  • npm run format to apply eslint formatting
  • npm run test passes with this change
  • This pull request links relevant issues as Fixes #0000
  • There are new or updated unit tests validating the change
  • Documentation has been updated to reflect this change
  • The new commits follow conventions explained
    in CONTRIBUTING.md

ido-pluto and others added 25 commits May 22, 2025 23:51
- Global CLI, you do not need to wrap active downloads with downloadSequence
- Auto increase parallel stream, maximize download speed
- Not reusing redirected URL by default - prevent token expires for long downloads
- Performance & stability improvements
- Remote CLI progress, show download progress in another process easily

BREAKING CHANGE:
- `partsURL` removed in favor of `partURLs`
- Not reusing redirected URL by default
- Different chunk size based on programType
- You can recall `.download()` and it will not throw an error - will return the same promise of downloading (do *not* redownload the file)
- You can change the parallel stream count after download has started
# Conflicts:
#	package-lock.json
#	package.json
#	src/download/download-engine/download-file/download-engine-file.ts
#	src/download/download-engine/download-file/progress-status-file.ts
#	src/download/download-engine/engine/download-engine-multi-download.ts
#	src/download/download-engine/engine/download-engine-nodejs.ts
#	src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts
#	src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts
#	src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts
#	src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts
#	src/download/transfer-visualize/progress-statistics-builder.ts
#	src/download/transfer-visualize/transfer-cli/GlobalCLI.ts
#	src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts
#	src/download/transfer-visualize/transfer-cli/transfer-cli.ts
#	test/browser.test.ts
#	test/download.test.ts
#	test/fetchDownloadInfo.test.ts
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR focuses on performance and dependency reduction in the download progress/CLI visualization and Node.js write path, while updating CI/build config and tests to match the new behavior.

Changes:

  • Replaced pretty-bytes / pretty-ms with new in-repo “fast” formatters and updated CLI/progress rendering to use them.
  • Reworked parts of the download engine runtime behavior (write buffering via WriteQueue, progress throttling, metadata-save hook changes, CLI update debouncing/caching).
  • Updated test setup (timeouts, fixtures) and CI build pipeline (new tsconfig.prod.json, build:prod, Vitest upgrade).

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
vitest.config.ts Switches to a global test timeout instead of forcing single-thread workers.
tsconfig.prod.json Adds a production TypeScript build config (used by CI).
test/utils/files.ts Updates large-file test fixture URL.
test/utils/download.ts Adjusts local test fixture path/metadata fields.
test/fetchDownloadInfo.test.ts Updates header string and snapshot formatting; removes per-suite timeout override.
test/download.test.ts Tweaks chunk size and removes per-suite timeout override.
test/copy-file.test.ts Removes per-suite timeout override.
test/browser.test.ts Updates expected hashes; changes concurrency options; skips one browser-memory test.
src/download/transfer-visualize/utils/prettyMSFast.ts Adds fast compact duration formatting.
src/download/transfer-visualize/utils/prettyBytesFast.ts Adds fast byte formatting with optional locale/format controls.
src/download/transfer-visualize/transfer-statistics.ts Avoids divide-by-zero when total bytes is 0.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts Replaces lodash debounce with manual time-based throttling + SIGINT behavior changes.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts Switches duration formatting to the new formatter.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts Switches byte formatting to the new formatter.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts Adds caching/invalidation for displayed engines; adapts to new updateStatues() API.
src/download/transfer-visualize/progress-statistics-builder.ts Adds common transfer-action aggregation + cached loading status.
src/download/transfer-visualize/format-transfer-status.ts Switches formatting helpers to new fast formatters.
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts Introduces buffered coalescing queue for positional writes.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts Integrates WriteQueue, changes metadata flush behavior, adds finalizer-based fd cleanup.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts Cleans abort listener management; switches timeout formatting; adjusts length detection.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts Cleans abort listener lifecycle; switches timeout formatting; adjusts length detection.
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts Adds cloned-listener cleanup to prevent leaks.
src/download/download-engine/engine/download-engine-nodejs.ts Adapts save-progress hook to new write-stream metadata API.
src/download/download-engine/engine/download-engine-multi-download.ts Ensures close() happens on error path too.
src/download/download-engine/download-file/progress-status-file.ts Replaces ProgressStatusFile class usage with EMPTY_PROGRESS_STATUS constant.
src/download/download-engine/download-file/download-engine-file.ts Reworks progress status creation, adds progress throttling, changes save-progress hook shape.
src/cli/cli.ts Adjusts save-path interpretation and output naming behavior.
README.md Improves browser usage docs around CORS/range + manual download info.
package.json Adds build:prod, upgrades dependencies (Vitest, xmlhttprequest-ssl, semantic-release), removes pretty-* deps.
.github/workflows/test.yml Uses build:prod in CI.
.github/workflows/build.yml Uses build:prod in CI; updates release job environment usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/cli/cli.ts Outdated
Comment thread src/download/download-engine/download-file/download-engine-file.ts
Comment thread src/download/transfer-visualize/utils/prettyBytesFast.ts
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors ipull’s core download/streaming and CLI progress pipeline to reduce event-loop load and CPU usage, while adding a local test server and a broad set of regression tests for range handling, redirects, progress events, and error propagation.

Changes:

  • Replace pretty-bytes/pretty-ms + chalk with faster internal formatters and ansis, and reduce CLI update frequency via custom debouncing/caching.
  • Optimize disk writes by replacing the previous debounce writer with a coalescing WriteQueue flushed via positional writev.
  • Expand test coverage substantially using a new local Express test server with range/no-range/mismatch scenarios.

Reviewed changes

Copilot reviewed 73 out of 75 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
vitest.config.ts Adjust Vitest worker settings; add longer timeout.
tsconfig.prod.json Add dedicated production TS build config for CI/build.
test/utils/local-server.ts New local test server for range/redirect/mismatch scenarios.
test/utils/hash.ts Switch hashing helper to fs/promises.
test/utils/files.ts Swap BIG_FILE test URL to a different remote fixture.
test/utils/fetchRangeDownloadParts.ts Helper to fetch multiple HTTP ranges in parallel.
test/utils/download.ts Update download test helpers for new part shape/options.
test/redirect-handling.test.ts New tests for redirect following and URL reuse behavior.
test/range-validation-error.test.ts New tests for invalid range option validation errors.
test/range-download-parts.test.ts New test for assembling multiple part URLs into one file.
test/progress-events.test.ts New test asserting progress/event emissions occur.
test/pause-resume-abort.test.ts New tests for pause/resume/abort flows not hanging.
test/parallel-streams-edge.test.ts New edge tests for parallel stream counts and auto-increase.
test/mixed-parts.test.ts New test mixing range and non-range parts in a single download.
test/fetchstream-error-propagation.test.ts New test ensuring fetch errors propagate and don’t hang.
test/fetchDownloadInfo.test.ts Rework fetchDownloadInfo tests to use the local server.
test/eventEmitter.test.ts New tests for the custom internal EventEmitter.
test/download.test.ts Update download tests to match new engine options/behavior.
test/daynamic-content-length.test.ts Update dynamic-length tests to fs/promises cleanup calls.
test/copy-file.test.ts Update copy tests to fs/promises cleanup calls.
test/content-length-mismatch.test.ts New tests for invalid/mismatched Content-Length handling.
test/browser.test.ts Update browser tests and snapshots; adjust concurrency/repeats.
test/accept-range-false.test.ts New tests for no-range behavior and content-length=0 cases.
src/utils/fs.ts Add small internal fs helpers (ensureFile/pathExists/readJSON).
src/utils/EventEmitter.ts New internal EventEmitter implementation with typed overloads.
src/download/transfer-visualize/utils/progressBarIcons.ts Replace chalk usage with ansis.
src/download/transfer-visualize/utils/prettyMSFast.ts Add fast compact ms formatter.
src/download/transfer-visualize/utils/prettyBytesFast.ts Add fast pretty-bytes replacement (number/bigint + options).
src/download/transfer-visualize/transfer-statistics.ts Fix divide-by-zero percentage when total is 0.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts Replace lodash debounce with time-gated getter-based update model.
src/download/transfer-visualize/transfer-cli/progress-bars/summary-transfer-cli-progress-bar.ts Swap chalk → ansis.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts Swap chalk/pretty-ms → ansis/prettyMSFast.
src/download/transfer-visualize/transfer-cli/progress-bars/base-transfer-cli-progress-bar.ts Swap chalk types/colors → ansis; tighten spinner typing.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts Swap chalk/pretty-bytes → ansis/prettyBytesFast.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts Cache CLI engine/status collection; adapt to new TransferCli updates.
src/download/transfer-visualize/progress-statistics-builder.ts Switch to internal EventEmitter; optimize transfer-action aggregation/caching.
src/download/transfer-visualize/format-transfer-status.ts Replace pretty-bytes/pretty-ms with fast formatters; adjust percentage formatting.
src/download/download-engine/utils/sleepPromise.ts Add internal sleepPromise to replace sleep-promise.
src/download/download-engine/types.ts Update part shape (remote/download size, range, fetchStream, per-part opts).
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts New buffered coalescing write queue with parallel positional flush.
src/download/download-engine/streams/download-engine-write-stream/utils/BytesWriteDebounce.ts Remove old debounce writer implementation.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts Integrate WriteQueue + metadata flush; remove fs-extra; add fd lifecycle handling.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-browser.ts Add totalLength to writer path; adjust blob creation.
src/download/download-engine/streams/download-engine-write-stream/base-download-engine-write-stream.ts Update abstract write() signature to include totalLength.
src/download/download-engine/streams/download-engine-fetch-stream/utils/smart-chunk-split.ts Track saved length incrementally; pass totalLength to write callback.
src/download/download-engine/streams/download-engine-fetch-stream/utils/content-disposition.ts Trim trailing ; to improve filename parsing resilience.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts Add watchdog-based stream timeout handling; headers timeout abort; content-length validation updates.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-local-file.ts Replace fs-extra ensureFile; adjust locking + async close.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts Add headers timeout abort + watchdog-based throttling; content-length checks; dynamic-length logic adjustments.
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts Replace eventemitter3; add watchdog/timers; add range support; add headers timeout abort utility.
src/download/download-engine/engine/error/missing-event-error.ts Remove unused error type.
src/download/download-engine/engine/error/invalid-content-length-error.ts Remove unused engine-level invalid content length error.
src/download/download-engine/engine/error/download-already-started-error.ts Remove unused error type.
src/download/download-engine/engine/error/RangeOutOfPartLengthError.ts New error for user-specified range exceeding remote size.
src/download/download-engine/engine/download-engine-nodejs.ts Adjust save path handling; replace fs-extra; support per-part internal URL objects.
src/download/download-engine/engine/download-engine-multi-download.ts Switch to internal EventEmitter overload emits; ensure close on error.
src/download/download-engine/engine/download-engine-browser.ts Support internal full-part URL objects; fetch stream selection changes.
src/download/download-engine/engine/base-download-engine.ts Add range/full-part URL model; validate ranges; switch to internal EventEmitter overload emits.
src/download/download-engine/engine/DownloadEngineRemote.ts Switch to internal EventEmitter overload emits.
src/download/download-engine/download-file/progress-status-file.ts Replace ProgressStatusFile class with EMPTY_PROGRESS_STATUS constant.
src/download/download-engine/download-file/downloaderProgramManager.ts Switch from sleep-promise to sleepPromise.
src/download/download-engine/download-file/download-programs/switch-program.ts Pass parallelStreams into program constructors.
src/download/download-engine/download-file/download-programs/download-program-stream.ts Optimize single-stream case; constructor updated.
src/download/download-engine/download-file/download-programs/download-program-chunks.ts Constructor updated for explicit parallelStreams.
src/download/download-engine/download-file/download-programs/base-download-program.ts Store parallelStreams from argument (not saved progress); remove 1-stream early return here.
src/download/download-engine/download-file/download-engine-file.ts Per-part options (streams/program); new throttled progress; range support; internal emitter overload emits.
src/const.ts Replace fs-extra readJSON with internal readJSON helper.
src/cli/utils/find-download-dir.ts Replace fs-extra lstat with fs/promises.
src/cli/cli.ts Remove -st shorthand; implement directory/file save-path detection; rename connections option parsing.
package.json Node >=20; add build:prod; remove several deps; bump toolchain/test deps.
README.md Update browser docs for CORS/range info hints.
.gitignore Ignore additional test artifacts and downloaded fixtures.
.github/workflows/test.yml Use npm run build:prod for TS validity.
.github/workflows/build.yml Use npm run build:prod; adjust release env.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors ipull’s core download pipeline to reduce event-loop overhead and improve throughput, including faster formatting helpers, debounced/throttled progress emission, and a new buffered write queue approach, alongside a large suite of new tests and CI build adjustments.

Changes:

  • Replace several dependencies/utilities (EventEmitter, fs helpers, pretty-bytes/pretty-ms, chalk→ansis) to reduce CPU and bundle weight.
  • Rework download engine internals: per-part options, range support/validation, stream watchdog/timeouts, write buffering/flush behavior.
  • Add extensive vitest coverage using a local express server for range/redirect/mismatch/pause/resume edge cases and update CI to build with tsconfig.prod.json.

Reviewed changes

Copilot reviewed 73 out of 75 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
vitest.config.ts Adjust vitest runtime defaults (timeout; remove forced single-worker config).
tsconfig.prod.json Add production TypeScript build config used by CI.
test/utils/local-server.ts New local express test server with range/redirect/mismatch routes.
test/utils/hash.ts Switch test hashing helper to fs/promises.
test/utils/files.ts Update large test file URL used by tests.
test/utils/fetchRangeDownloadParts.ts Add helper to fetch multiple ranged parts in tests.
test/utils/download.ts Update test download helpers to new engine/part options + internal fs utils.
test/redirect-handling.test.ts Add redirect reuse behavior tests.
test/range-validation-error.test.ts Add range validation error tests for browser API.
test/range-download-parts.test.ts Add multipart ranged download correctness test.
test/progress-events.test.ts Add progress/event emission sanity test.
test/pause-resume-abort.test.ts Add pause/resume/abort behavioral tests.
test/parallel-streams-edge.test.ts Add parallelStreams edge/stress tests.
test/mixed-parts.test.ts Add mixed accept-range true/false parts test.
test/fetchstream-error-propagation.test.ts Add fetchStream error propagation / non-hanging test.
test/fetchDownloadInfo.test.ts Replace external/throttled proxy test with local-server-based test.
test/eventEmitter.test.ts Add tests for new custom EventEmitter.
test/download.test.ts Update download tests to new per-part parallel stream configuration.
test/daynamic-content-length.test.ts Update dynamic content-length tests to fs/promises APIs.
test/copy-file.test.ts Update copy tests to fs/promises APIs.
test/content-length-mismatch.test.ts Add tests for invalid Content-Length handling.
test/browser.test.ts Update browser tests (hash snapshots, repeats config, fs/promises).
test/accept-range-false.test.ts Add tests for accept-range false / content-length 0 scenarios.
src/utils/fs.ts New internal fs helpers (ensureFile/pathExists/readJSON).
src/utils/EventEmitter.ts New internal typed EventEmitter implementation.
src/download/transfer-visualize/utils/progressBarIcons.ts Switch terminal styling from chalk to ansis.
src/download/transfer-visualize/utils/prettyMSFast.ts Add faster compact ms formatter.
src/download/transfer-visualize/utils/prettyBytesFast.ts Add faster bytes formatter with trunc/locale options.
src/download/transfer-visualize/transfer-statistics.ts Fix percentage calculation for total=0.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts Replace lodash debounce with time-based throttling + SIGINT handling tweaks.
src/download/transfer-visualize/transfer-cli/progress-bars/summary-transfer-cli-progress-bar.ts Switch styling from chalk to ansis.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts Switch styling + pretty-ms to new fast prettyMS implementation.
src/download/transfer-visualize/transfer-cli/progress-bars/base-transfer-cli-progress-bar.ts Switch styling from chalk to ansis + SpinnerName typing.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts Use prettyBytesFast + ansis; update spinner typing.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts Cache CLI engine/status calculation; adjust update flow and finished handling.
src/download/transfer-visualize/progress-statistics-builder.ts Migrate to internal EventEmitter; cache loading status; compute common transfer action.
src/download/transfer-visualize/format-transfer-status.ts Use prettyBytesFast/prettyMSFast; adjust percentage formatting.
src/download/download-engine/utils/sleepPromise.ts Add local sleep helper to replace sleep-promise dependency.
src/download/download-engine/types.ts Expand part metadata (range, sizes, fetchStream, per-part options).
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts New coalescing write buffer + parallel positional flush design.
src/download/download-engine/streams/download-engine-write-stream/utils/BytesWriteDebounce.ts Remove previous debounce-based writer implementation.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts Replace fs-extra/debounce writer with WriteQueue + metadata flush integration.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-browser.ts Update write signature to include totalLength; tweak Blob typing.
src/download/download-engine/streams/download-engine-write-stream/base-download-engine-write-stream.ts Update abstract write() signature to include totalLength.
src/download/download-engine/streams/download-engine-fetch-stream/utils/smart-chunk-split.ts Track savedLength incrementally; pass totalLength into write callback.
src/download/download-engine/streams/download-engine-fetch-stream/utils/content-disposition.ts Trim trailing ; before parsing Content-Disposition.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts Rework XHR stream watchdog/timeouts + abort handling + totalLength callback.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-local-file.ts Replace fs-extra ensureFile; async close; lock scope change.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts Rework fetch stream to use watchdog/throttling + new abort controller + totalLength callback.
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts Replace eventemitter3/sleep-promise; add range support, watchdog, headersTimeout; update callback signature.
src/download/download-engine/engine/error/missing-event-error.ts Remove unused/legacy engine error.
src/download/download-engine/engine/error/invalid-content-length-error.ts Remove legacy engine-level invalid-content-length error.
src/download/download-engine/engine/error/download-already-started-error.ts Remove unused/legacy error type.
src/download/download-engine/engine/error/RangeOutOfPartLengthError.ts Add explicit error for user-specified range exceeding remote size.
src/download/download-engine/engine/download-engine-nodejs.ts Refactor Node engine to new part model; fix savePath directory handling; reflink fast-path updates.
src/download/download-engine/engine/download-engine-multi-download.ts Migrate to internal EventEmitter and emit0/emit1 usage; adjust error close behavior.
src/download/download-engine/engine/download-engine-browser.ts Refactor browser engine to new full-part URL model.
src/download/download-engine/engine/base-download-engine.ts Add range support, full part URL model, range validation, internal EventEmitter.
src/download/download-engine/engine/DownloadEngineRemote.ts Migrate to internal EventEmitter and emit0/emit1 usage.
src/download/download-engine/download-file/progress-status-file.ts Replace ProgressStatusFile class with EMPTY_PROGRESS_STATUS object.
src/download/download-engine/download-file/downloaderProgramManager.ts Replace sleep-promise with local sleepPromise.
src/download/download-engine/download-file/download-programs/switch-program.ts Pass parallelStreams explicitly into program constructors.
src/download/download-engine/download-file/download-programs/download-program-stream.ts Add single-stream shortcut in stream program; accept parallelStreams in ctor.
src/download/download-engine/download-file/download-programs/download-program-chunks.ts Accept parallelStreams in ctor.
src/download/download-engine/download-file/download-programs/base-download-program.ts Set parallel stream count from passed value (not from saved progress).
src/download/download-engine/download-file/download-engine-file.ts Major engine refactor: per-part options, new progress model, throttled progress, range size changes.
src/const.ts Replace fs-extra readJSON with internal readJSON helper.
src/cli/utils/find-download-dir.ts Replace fs-extra lstat with fs/promises lstat.
src/cli/cli.ts Fix savePath directory/file detection; remove -st short option; update connections option handling.
package.json Bump Node engine requirement; add build:prod; update deps (ansis, commander, vitest, etc.).
README.md Update browser usage docs with CORS/defaultFetchDownloadInfo guidance.
.gitignore Ignore new/updated test artifacts (big-file.bin, gguf files).
.github/workflows/test.yml Run TypeScript validity via npm run build:prod.
.github/workflows/build.yml Use npm run build:prod; adjust release env usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/download/download-engine/download-file/download-engine-file.ts
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors ipull’s core download pipeline to reduce event-loop load and improve throughput, including faster formatting utilities, a new write-queue implementation, and broader test coverage via a local HTTP test server.

Changes:

  • Replaced/optimized progress formatting and CLI rendering (pretty-bytes/pretty-ms → internal fast versions; chalk → ansis; CLI update debouncing/caching).
  • Refactored download engine internals (custom EventEmitter, per-part metadata/range support, fetch-stream throttling/watchdog, write buffering via WriteQueue).
  • Expanded and stabilized tests by replacing remote fixtures with a local Express test server and adding new edge-case suites.

Reviewed changes

Copilot reviewed 74 out of 76 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
vitest.config.ts Adjusts vitest runtime settings (timeout) for longer-running integration tests.
tsconfig.prod.json Adds a prod-focused TS build config used by CI (build:prod).
test/utils/local-server.ts New local Express server for deterministic range/redirect/content-length scenarios.
test/utils/hash.ts Switches to fs/promises for hashing helpers.
test/utils/files.ts Updates large-file fixture URL used in tests.
test/utils/fetchRangeDownloadParts.ts Adds helper to fetch multiple ranged parts in parallel for tests.
test/utils/download.ts Updates test download helpers to match new per-part fields and engine options.
test/redirect-handling.test.ts Adds redirect reuse vs non-reuse behavior tests.
test/range-validation-error.test.ts Adds validation tests for invalid ranges.
test/range-download-parts.test.ts Adds multi-part ranged download test and byte-for-byte verification.
test/progress-events.test.ts Adds event emission/progress smoke coverage.
test/pause-resume-abort.test.ts Adds pause/resume/abort behavior tests.
test/parallel-streams-edge.test.ts Adds edge-case tests for high/low parallel stream counts.
test/mixed-parts.test.ts Adds mixed accept-range parts test.
test/fetchstream-error-propagation.test.ts Adds test ensuring fetch-stream failures reject instead of hanging.
test/fetchDownloadInfo.test.ts Refactors to local test server and validates redirected downloads deterministically.
test/eventEmitter.test.ts Adds unit tests for the new typed EventEmitter implementation.
test/download.test.ts Updates download tests to new createDownloadFile options and engine behavior.
test/daynamic-content-length.test.ts Updates file ops to fs/promises and test harness assertions.
test/copy-file.test.ts Updates file ops to fs/promises and modern vitest assertions.
test/content-length-mismatch.test.ts Adds mismatch tests for invalid Content-Length handling.
test/browser.test.ts Updates assertions/fixtures and adjusts concurrency options in browser tests.
test/accept-range-false.test.ts Adds tests for non-range servers and content-length=0 cases.
src/utils/fs.ts Introduces lightweight fs helpers replacing fs-extra use cases.
src/utils/EventEmitter.ts Adds a custom typed EventEmitter with optimized emit paths.
src/download/transfer-visualize/utils/progressBarIcons.ts Switches coloring library to ansis.
src/download/transfer-visualize/utils/prettyMSFast.ts Adds optimized compact duration formatting.
src/download/transfer-visualize/utils/prettyBytesFast.ts Adds optimized byte formatting implementation.
src/download/transfer-visualize/transfer-statistics.ts Fixes percentage calc for total=0.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts Refactors CLI update logic to a time-based debounce without lodash.debounce.
src/download/transfer-visualize/transfer-cli/progress-bars/summary-transfer-cli-progress-bar.ts Switches dim/cyan formatting from chalk to ansis.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts Switches chalk→ansis and pretty-ms→prettyMSFast.
src/download/transfer-visualize/transfer-cli/progress-bars/base-transfer-cli-progress-bar.ts Switches chalk→ansis and updates spinner typing.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts Switches chalk→ansis and pretty-bytes→prettyBytesFast.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts Adds caching for CLI engine/status selection and refactors progress update flow.
src/download/transfer-visualize/progress-statistics-builder.ts Switches EventEmitter impl, changes progress emission, and adds “common transfer action” tracking.
src/download/transfer-visualize/format-transfer-status.ts Switches to fast pretty functions and trunc-based percentage formatting.
src/download/download-engine/utils/sleepPromise.ts Adds internal sleep helper to remove sleep-promise dependency.
src/download/download-engine/types.ts Updates per-part schema (remoteFileSize/downloadSize/range/fetchStream).
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts Adds new buffered/merged write queue for fewer syscalls/context switches.
src/download/download-engine/streams/download-engine-write-stream/utils/BytesWriteDebounce.ts Removes old debounce writer implementation.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts Replaces debounce writer with WriteQueue; refactors metadata flush/write path.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-browser.ts Updates write signature (adds totalLength) and Blob creation casting.
src/download/download-engine/streams/download-engine-write-stream/base-download-engine-write-stream.ts Updates abstract write signature to include totalLength.
src/download/download-engine/streams/download-engine-fetch-stream/utils/smart-chunk-split.ts Refactors to avoid repeated reduce and to pass totalLength to writer.
src/download/download-engine/streams/download-engine-fetch-stream/utils/content-disposition.ts Adds trimming for headers ending in ;.
src/download/download-engine/streams/download-engine-fetch-stream/errors/RenewFetchError.ts Adds internal error for renewing fetch on pause.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts Adds watchdog-based stream responsiveness checks and refactors header/length handling.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-local-file.ts Removes fs-extra ensureFile and adjusts close semantics to async.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts Adds watchdog-based reading/throttling, content-length validation, and abort controller refactor.
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts Switches to custom EventEmitter, adds watchdog/timers, range offsets, and abort controller helper.
src/download/download-engine/engine/error/missing-event-error.ts Removes unused error type.
src/download/download-engine/engine/error/invalid-content-length-error.ts Removes unused engine-level error (stream-level remains).
src/download/download-engine/engine/error/download-already-started-error.ts Removes unused error type.
src/download/download-engine/engine/error/RangeOutOfPartLengthError.ts Adds explicit error for invalid range end vs remote size.
src/download/download-engine/engine/download-engine-nodejs.ts Refactors part URL handling (FullPartURL), save-path logic, and reflink clone attempt.
src/download/download-engine/engine/download-engine-multi-download.ts Switches to custom EventEmitter and adjusts event emission APIs.
src/download/download-engine/engine/download-engine-browser.ts Refactors part URL handling to FullPartURL and stream selection.
src/download/download-engine/engine/base-download-engine.ts Adds ranged part support, FullPartURL parsing, and range validation.
src/download/download-engine/engine/DownloadEngineRemote.ts Switches to custom EventEmitter and emitX APIs.
src/download/download-engine/download-file/progress-status-file.ts Removes ProgressStatusFile class; adds EMPTY_PROGRESS_STATUS constant.
src/download/download-engine/download-file/downloaderProgramManager.ts Removes sleep-promise usage in favor of internal sleepPromise.
src/download/download-engine/download-file/download-programs/switch-program.ts Threads parallelStreams into program creation.
src/download/download-engine/download-file/download-programs/download-program-stream.ts Adjusts ctor signature and adds explicit single-stream fast path.
src/download/download-engine/download-file/download-programs/download-program-chunks.ts Adjusts ctor signature to accept parallelStreams.
src/download/download-engine/download-file/download-programs/base-download-program.ts Uses passed parallelStreams instead of reading from saved progress.
src/download/download-engine/download-file/download-engine-file.ts Major refactor: per-part options, progress throttling, range sizing, and new stream lifecycle handling.
src/const.ts Replaces fs-extra JSON read with internal readJSON helper.
src/cli/utils/find-download-dir.ts Removes fs-extra; uses lstat from fs/promises.
src/cli/cli.ts Fixes --save directory handling and removes -st shortcut option.
package.json Updates engine to Node >=20, adds build:prod, updates deps, removes fs-extra/chalk/eventemitter3/etc.
README.md Updates browser example guidance for CORS + defaultFetchDownloadInfo.
.gitignore Ignores new test fixtures and generated files.
.github/workflows/test.yml Switches TS check step to npm run build:prod.
.github/workflows/build.yml Switches build step to npm run build:prod and adjusts release env.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/download/download-engine/engine/download-engine-nodejs.ts
Comment thread src/download/download-engine/download-file/download-engine-file.ts
Comment thread src/download/download-engine/download-file/download-engine-file.ts
Comment thread src/download/download-engine/download-file/download-engine-file.ts
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors core download/CLI internals to reduce event-loop overhead and improve throughput (buffered disk writes, progress/event debouncing), while also replacing several dependencies with lighter in-repo utilities and adding a broad new test suite (mostly via a local HTTP test server) to validate edge cases like redirects, ranges, and content-length mismatches.

Changes:

  • Replace eventemitter3, pretty-bytes, pretty-ms, chalk with in-repo EventEmitter + pretty*Fast utilities and ansis, updating CLI formatting/output accordingly.
  • Rework download pipeline (fetch streams + write streams) to support range-aware offsets, throttled progress emission, and a new high-throughput Node.js WriteQueue.
  • Add local-server-based integration tests for redirects/range validation/content-length mismatch and update CI to use build:prod with a new tsconfig.prod.json.

Reviewed changes

Copilot reviewed 74 out of 76 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
vitest.config.ts Adjusts Vitest runtime configuration (now thread pool + explicit timeout).
tsconfig.prod.json Adds production TS build config used by CI (build:prod).
test/utils/local-server.ts New local Express server to simulate range/no-range/redirect/mismatch behaviors.
test/utils/hash.ts Switches hashing utility to fs/promises.
test/utils/files.ts Changes the large test download URL used by several integration tests.
test/utils/fetchRangeDownloadParts.ts Adds helper to fetch multiple ranged parts via fetch.
test/utils/download.ts Updates test download helpers and creates DownloadFile objects with new part fields.
test/redirect-handling.test.ts Adds redirect reuse behavior tests.
test/range-validation-error.test.ts Adds range validation error tests.
test/range-download-parts.test.ts Adds multi-URL range part download test.
test/progress-events.test.ts Adds event/progress emission coverage for DownloadEngineFile.
test/pause-resume-abort.test.ts Adds pause/resume/abort behavioral tests.
test/parallel-streams-edge.test.ts Adds stress/edge tests for parallelStreams behavior.
test/mixed-parts.test.ts Adds mixed range + non-range parts test.
test/fetchstream-error-propagation.test.ts Adds regression test to ensure fetch-stream errors don’t hang.
test/fetchDownloadInfo.test.ts Replaces external throttled proxy test with local-server-based info tests.
test/eventEmitter.test.ts Adds unit tests for the new in-repo EventEmitter.
test/download.test.ts Updates download tests to new per-part options and chunk sizing.
test/daynamic-content-length.test.ts Updates dynamic content-length tests to fs/promises APIs.
test/copy-file.test.ts Updates copy tests to fs/promises and new helpers.
test/content-length-mismatch.test.ts Adds explicit content-length mismatch tests (incl. expected error).
test/browser.test.ts Updates browser tests (new hash snapshot, concurrency config, fs API changes).
test/accept-range-false.test.ts Adds non-range server behavior tests.
src/utils/fs.ts Introduces small fs helpers (ensureFile, pathExists, readJSON) to replace fs-extra usage.
src/utils/EventEmitter.ts Adds new typed EventEmitter implementation with fast-path emits.
src/download/transfer-visualize/utils/progressBarIcons.ts Switches CLI coloring from chalk to ansis.
src/download/transfer-visualize/utils/prettyMSFast.ts Adds optimized compact pretty-ms replacement.
src/download/transfer-visualize/utils/prettyBytesFast.ts Adds optimized pretty-bytes replacement (incl. formatting helpers).
src/download/transfer-visualize/transfer-statistics.ts Fixes divide-by-zero in percentage calculation when total === 0.
src/download/transfer-visualize/transfer-cli/transfer-cli.ts Reworks CLI update scheduling (debounce/interval logic, SIGINT behavior).
src/download/transfer-visualize/transfer-cli/progress-bars/summary-transfer-cli-progress-bar.ts Updates summary progress bar styling to ansis.
src/download/transfer-visualize/transfer-cli/progress-bars/fancy-transfer-cli-progress-bar.ts Updates fancy progress bar styling + switches to prettyMSFast.
src/download/transfer-visualize/transfer-cli/progress-bars/base-transfer-cli-progress-bar.ts Updates base progress bar styling/types for ansis + spinner typings.
src/download/transfer-visualize/transfer-cli/multiProgressBars/BaseMultiProgressBar.ts Uses prettyBytesFast, ansis, and updated spinner types.
src/download/transfer-visualize/transfer-cli/GlobalCLI.ts Adds caching for CLI engine collection and reworks progress updates via getter.
src/download/transfer-visualize/progress-statistics-builder.ts Switches to in-repo EventEmitter, tweaks aggregation logic, caches loading status.
src/download/transfer-visualize/format-transfer-status.ts Switches to pretty*Fast utilities and trunc formatting logic.
src/download/download-engine/utils/sleepPromise.ts Adds small sleepPromise helper to replace sleep-promise.
src/download/download-engine/types.ts Updates part shape (remoteFileSize, downloadSize, range, fetchStream) and progress types.
src/download/download-engine/streams/download-engine-write-stream/utils/WriteQueue.ts Adds new coalescing, adaptive buffering write queue for Node.js write stream.
src/download/download-engine/streams/download-engine-write-stream/utils/BytesWriteDebounce.ts Removes old debounce-based write aggregator implementation.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-nodejs.ts Switches Node writer to WriteQueue, removes fs-extra, adds fd finalization logic.
src/download/download-engine/streams/download-engine-write-stream/download-engine-write-stream-browser.ts Updates browser writer signature to accept totalLength and adjusts Blob creation typing.
src/download/download-engine/streams/download-engine-write-stream/base-download-engine-write-stream.ts Updates abstract writer API to include totalLength.
src/download/download-engine/streams/download-engine-fetch-stream/utils/smart-chunk-split.ts Updates chunk splitting to use downloadSize and tracks saved length incrementally.
src/download/download-engine/streams/download-engine-fetch-stream/utils/content-disposition.ts Trims trailing ; from header before parsing.
src/download/download-engine/streams/download-engine-fetch-stream/errors/RenewFetchError.ts Adds internal error used to signal a fetch renewal/retry path.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-xhr.ts Refactors XHR stream logic (timeouts/watchdog/content-length validation/progress).
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-local-file.ts Removes fs-extra, uses ensureFile, and makes close async.
src/download/download-engine/streams/download-engine-fetch-stream/download-engine-fetch-stream-fetch.ts Refactors fetch streaming (watchdog loop, throttling, timeout abort, range handling).
src/download/download-engine/streams/download-engine-fetch-stream/base-download-engine-fetch-stream.ts Switches to in-repo EventEmitter, adds range offsets, watchdog, and timeout abort controller.
src/download/download-engine/engine/error/missing-event-error.ts Removes unused engine error.
src/download/download-engine/engine/error/invalid-content-length-error.ts Removes old engine-layer invalid content-length error.
src/download/download-engine/engine/error/download-already-started-error.ts Removes unused engine error.
src/download/download-engine/engine/error/RangeOutOfPartLengthError.ts Adds explicit error for range exceeding remote part size.
src/download/download-engine/engine/download-engine-nodejs.ts Reworks option parsing to “full part URL” objects, updates progress persistence/clone path.
src/download/download-engine/engine/download-engine-multi-download.ts Switches to in-repo EventEmitter and fast-path emits; adjusts error/close behavior.
src/download/download-engine/engine/download-engine-browser.ts Aligns browser engine creation with new “full part URL” internal representation.
src/download/download-engine/engine/base-download-engine.ts Adds range support and part URL normalization; reworks download-file creation.
src/download/download-engine/engine/DownloadEngineRemote.ts Switches to in-repo EventEmitter and fast-path emits for remote progress.
src/download/download-engine/download-file/progress-status-file.ts Replaces class with EMPTY_PROGRESS_STATUS constant; adds Cloned flag.
src/download/download-engine/download-file/downloaderProgramManager.ts Replaces sleep-promise with sleepPromise.
src/download/download-engine/download-file/download-programs/switch-program.ts Updates program factory signature to accept parallelStreams.
src/download/download-engine/download-file/download-programs/download-program-stream.ts Passes parallelStreams into base program; adds fast-path for parallelStreams === 1.
src/download/download-engine/download-file/download-programs/download-program-chunks.ts Passes parallelStreams into base program.
src/download/download-engine/download-file/download-programs/base-download-program.ts Uses passed parallelStreams instead of reading from saved progress.
src/download/download-engine/download-file/download-engine-file.ts Large refactor: per-part options, throttled progress, range offsets, write API change, cloned flag path.
src/const.ts Replaces fs-extra.readJSON with in-repo readJSON.
src/cli/utils/find-download-dir.ts Replaces fs-extra.lstat with fs/promises.
src/cli/cli.ts Updates CLI save-path detection, removes -st shortcut, and adjusts option names.
package.json Node engine bump, dependency swaps (ansis, remove fs-extra/chalk/pretty-*), adds build:prod.
README.md Expands browser usage notes for acceptRangeIsKnown / defaultFetchDownloadInfo.
.gitignore Adds test fixture outputs to ignore list.
.github/workflows/test.yml Switches TS validity step to npm run build:prod.
.github/workflows/build.yml Switches build to build:prod and adjusts release environment variables.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/download/download-engine/engine/DownloadEngineRemote.ts
Comment thread src/download/download-engine/download-file/download-engine-file.ts
Comment thread src/download/download-engine/engine/download-engine-nodejs.ts
Comment thread .github/workflows/build.yml
Comment thread vitest.config.ts
Comment thread test/utils/files.ts
@ido-pluto
Copy link
Copy Markdown
Owner Author

@copilot create me a commit message (do not commit!) for marge, with all the breaking changes - use semantic release syntax

@ido-pluto ido-pluto merged commit b82ae38 into beta Apr 22, 2026
3 checks passed
@github-actions
Copy link
Copy Markdown

🎉 This PR is included in version 5.0.0-beta.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants