Skip to content

Activate DuckDB worker lazily - #14938

Merged
jmcphers merged 4 commits into
mainfrom
bugfix/duckdb-eager-activation
Jul 20, 2026
Merged

Activate DuckDB worker lazily#14938
jmcphers merged 4 commits into
mainfrom
bugfix/duckdb-eager-activation

Conversation

@jmcphers

@jmcphers jmcphers commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

This change reduces Positron's steady state memory usage by 50-100MB, unless you have Data Explorer tabs open against CSV files (or others that use DuckDB).

The positron-duckdb extension activated onStartupFinished and forked its native DuckDB worker (almost 100MB!) in activate(), so the worker was resident in every window even when no data file was ever opened. Here's Positron's Process Explorer on a fresh boot of a recent release build:

Positron Version: 2026.08.0 build 96
Code - OSS Version: 1.124.0
Commit: fd9c70152c2210f0cf30f3c368db960dd5dfe367
Date: 2026-07-14T15:13:40-06:00
Electron: 42.2.0
Chromium: 148.0.7778.97
Node.js: 24.15.0
V8: 14.8.178.14-electron.0
OS: Darwin arm64 25.5.0

Note that the duckdbWorker.js worker is loaded and consuming 98MB:

image

This was especially wasteful in web/server builds where the extensions are a greater fraction of total memory use. This PR makes both the extension and its worker lazy, and reclaims the worker's memory when it's no longer in use.

Summary

  • Lazy activation: the extension now activates on a custom onPositronDataExplorerBackend:positron-duckdb event instead of onStartupFinished. MainThreadDataExplorer fires this event (via activateByEvent) the first time a dataset routed to the provider is accessed, so the extension stays dormant until a data file is actually opened. The existing 30s provider-registration wait in ExtHostDataExplorer.$handleRpc covers the activation window.
  • Lazy worker spawn: DuckDBInstance.create() no longer forks the worker. The child process is spawned lazily on the first query (the path already used for crash recovery), i.e. only when a dataset is opened.
  • Idle self-shutdown: the extension tracks open Data Explorers. When the last one closes, the worker self-terminates after a 2-minute cooldown to release its native memory; the next query transparently respawns it and re-imports on demand. A new disposeBackend lifecycle signal (main thread -> ext host -> the provider's optional closeDataset handler) delivers the close notification without activating a dormant extension.

Release Notes

New Features

  • N/A

Bug Fixes

  • Reduced Positron's memory footprint at startup by only starting the backend for CSV/TSV/Excel files when it is first used

Validation Steps

@:data-explorer @:duck-db @:connections

  1. Start Positron (ideally a web/server build) with no data files open. Confirm no duckdbWorker.js node process is running and the positron-duckdb extension is not activated.
  2. Open a data file (e.g. a .csv, .parquet, or .duckdb) in the Data Explorer. The extension activates and the worker process spawns; the data renders normally.
  3. Close all Data Explorer tabs backed by DuckDB. After ~2 minutes with none open, the worker process exits.
  4. Open another data file. The worker respawns transparently and the data renders normally.

Existing e2e coverage exercises the file-open/connection path that lazy activation must not break; new unit tests cover the worker lifecycle (lazy spawn, idle shutdown, respawn, cancelled shutdown) and the disposeBackend close signal.

@github-actions

Copy link
Copy Markdown

E2E Tests 🚀
This PR will run tests tagged with: @:critical @:data-explorer @:duck-db @:connections

Why these tags?
Tag Source
@:critical Always runs (required)
@:data-explorer PR description
@:duck-db PR description
@:connections PR description

More on automatic tags from changed files.

readme  valid tags

@jmcphers
jmcphers requested a review from softwarenerd July 16, 2026 23:53
@softwarenerd

softwarenerd commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

I verified this.

Before launching a CSV or Parquet file:

image

After launching a CSV or Parquet file:

image

After two minutes, the duckdbWorker was cleaned up:

image

@softwarenerd softwarenerd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks out in testing. Approved.

@jmcphers
jmcphers merged commit d6eac7b into main Jul 20, 2026
26 checks passed
@jmcphers
jmcphers deleted the bugfix/duckdb-eager-activation branch July 20, 2026 15:03
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants