Skip to content

Conversation

jheer
Copy link
Contributor

@jheer jheer commented Jul 3, 2025

This PR adds a new filesystem configuration option: forceFullHTTPReads. If enabled, this flag suppresses the use of HTTP range requests for DuckDB-WASM running in the browser.

The flag is set via the database open method:

const worker = new Worker(config.mainWorker);
const db = new AsyncDuckDB(new ConsoleLogger(), worker);
await db.instantiate(config.mainModule, config.pthreadWorker);
await db.open({
  filesystem: {
    forceFullHTTPReads: true
  }
});

The flag passes in through WASM (C code), and back to a worker thread (TypeScript code) where the actual file loads are performed.

Why add this flag?

  • Firefox has bad interactions with range requests and compression on some servers (including GitHub Pages), causing file loading in DuckDB-WASM to crash rather than gracefully fallback to full reads. Setting this flag sidesteps the error and enables successful file loading.
  • When one knows ahead of time that the full data is needed, full file reads can be more efficient by making a single HTTP call rather than multiple.
  • Full file reads result in fewer operations when connecting to object storage like R2 and S3, potentially reducing bills.

Copy link
Collaborator

@carlopi carlopi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!!

@carlopi carlopi merged commit 1766b6b into duckdb:main Jul 4, 2025
15 checks passed
@jheer jheer deleted the jh/force-full-http-reads branch July 4, 2025 09:06
@Fil
Copy link
Contributor

Fil commented Jul 7, 2025

Thanks! Do we have a recommended pattern in the case where we don't know if we need to enable this flag? A try/catch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants