AsyncDuckDB.instantiate emits error (worker terminated with 1 pending requests) and hangs

[ryan-williams]

What happens?

In some situations, AsyncDuckDB.instantiate emits this error and hangs next build:

worker terminated with 1 pending requests

I've run into this a few times while trying to use duckdb-wasm in Next.js apps. Here's some repro info (copied from runsascoded/duckdb-wasm-issue):

Repro

duckdb-utils/src/duckdb.ts calls AsyncDuckDB.instantiate:

const { worker, bundle } = await nodeWorkerBundle()
console.log("bundle:", bundle)
const logger = { log: () => {}, }
const db = new AsyncDuckDB(logger, worker)
console.log("instantiating db")
await db.instantiate(bundle.mainModule, bundle.pthreadWorker)  // ❌ "worker terminated with 1 pending requests"
console.log("instantiated db")

Github Actions repro

Here's an example of the error in a Github Action:

bundle: {
  mainModule: '/home/runner/work/duckdb-wasm-npm-link/duckdb-wasm-npm-link/node_modules/@duckdb/duckdb-wasm/dist/duckdb-eh.wasm',
  mainWorker: '/home/runner/work/duckdb-wasm-npm-link/duckdb-wasm-npm-link/node_modules/@duckdb/duckdb-wasm/dist/duckdb-node-eh.worker.cjs',
  pthreadWorker: null
}
instantiating db
worker terminated with 1 pending requests
 ⚠ Restarted static page generation for / because it took more than 60 seconds

Dockerfile repro

For some reason, the error only happens in this Dockerfile when the duckdb-utils module is npm linked:

FROM node:19.3.0
COPY . /src

# ✅ Build succeeds with local duckdb-utils install
RUN cd /src/duckdb-utils/dist && npm i
WORKDIR /src
RUN npm i
# This fails in Github Actions, but not in this Docker build, or on the underlying host, on my M1 macbook or an Amazon Linux EC2 instance
RUN npm run build

# ❌ Build fails with `npm link`ed duckdb-utils
RUN cd /src/duckdb-utils/dist && npm link
RUN npm link duckdb-utils
RUN npm run build  # ❌ `worker terminated with 1 pending requests` inside AsyncDuckDB constructor

Clone + docker build

Clone this repo, and build the Dockerfile:

git clone --recurse-submodules https://github.com/runsascoded/duckdb-wasm-issue
cd duckdb-wasm-issue
docker build -t duckdb-wasm-issue .  # ❌ fails
# 7.865 bundle: {
# 7.865   mainModule: '/src/node_modules/@duckdb/duckdb-wasm/dist/duckdb-eh.wasm',
# 7.865   mainWorker: '/src/node_modules/@duckdb/duckdb-wasm/dist/duckdb-node-eh.worker.cjs',
# 7.865   pthreadWorker: null
# 7.865 }
# 7.865 instantiating db
# 7.868 worker terminated with 1 pending requests
# 68.12  ⚠ Restarted static page generation for / because it took more than 60 seconds
# … etc.

I observe the same docker build failure on my M1 macbook and an Amazon Linux instance I tested.

Host repro

I also see the same failure when running directly on each underlying host (macbook / AZLinux), without Docker:

• npm run build succeeds when duckdb-utils is npm installed directly.
• npm run build fails when duckdb-utils is npm linked.

I don't know why Github Actions' ubuntu-latest demonstrates the problem more easily (without the npm link step).

Earlier repro: next.js#57819

I previously filed a repro at next.js#57819.

All the repros I've found involve Next.js, but the fact that the failure occurs within duckdb-wasm, and manifests as a hang instead of a normal Error / exception, makes me feel like it may be primarily a duckdb-wasm problem. I have only tried to use duckdb-wasm within Next.js, so I'm not sure it's specific to Next.

To Reproduce

• Github Actions example
• Dockerfile example
• duckdb-wasm: ok under `pages/`, broken under `app/` vercel/next.js#57819 also contains a repro.

Browser/Environment:

Node v19.3.0

Device:

M1 macbook, Amazon Linux, Github Actions ubuntu-latest

DuckDB-Wasm Version:

1.28.0

DuckDB-Wasm Deployment:

@duckdb/duckdb-wasm/dist/duckdb-eh.wasm

Full Name:

Ryan Williams

Affiliation:

Runs As Coded LLC

[ryan-williams]

Will try to add more details later but 2 things that seem to help:

Skip this npm i in the Dockerfile:

-RUN cd /src/duckdb-utils/dist && npm i
+RUN cd /src/duckdb-utils/dist

Make sure web-worker is in the dependencies array of the outer package (this was already the case in my repro, but I ran into it in another project today).