apache-arrow marked as external dependency

[hamilton]

Hello folks ~

So thrilled to experiment w/ duckdb-wasm! Thank you for all your hard work.

I'm having a bit of trouble bundling duckdb-wasm in a simple case. I'm using Rollup and opting for the simplest possible input file:

import * as duckdb from "@duckdb/duckdb-wasm/dist/duckdb-esm";
const JSDELIVR_BUNDLES = duckdb.getJsDelivrBundles();
console.log("resolving bundles");
async function resolve() {
  // Select a bundle based on browser checks
  const bundle = await duckdb.selectBundle(JSDELIVR_BUNDLES);
  // Instantiate the asynchronus version of DuckDB-Wasm
  const worker = new Worker(bundle.mainWorker);
  const logger = new duckdb.ConsoleLogger();
  const db = new duckdb.AsyncDuckDB(logger, worker);
  return await db.instantiate(bundle.mainModule, bundle.pthreadWorker);
}

(async () => {
  const db = await resolve();
  console.log(db);
})();

But rollup produces this error:

rollup v2.59.0
bundles src/main.js → public/poc.js...
(!) Unresolved dependencies
https://rollupjs.org/guide/en/#warning-treating-module-as-external-dependency
apache-arrow (imported by node_modules/@duckdb/duckdb-wasm/dist/duckdb-esm.js)
(!) Missing global variable name
Use output.globals to specify browser global variable names corresponding to external modules
apache-arrow (guessing 'apacheArrow')

Here is a repository that hopefully reproduces this (including my rollup file, includes instructions): https://github.com/hamilton/duckdb-wasm-issues

Is there something obvious that I'm missing here? (There usually is!)

[ankoh]

Thanks for trying it out!

Bundling duckdb-wasm is less straight-forward than I anticipated due to currently broken bundles of apache-arrow 6.0.0.
A fix is already on the arrow master but we'll have to wait for 6.0.1 until this gets easier.
There are workarounds for webpack and esbuild but I haven't checked rollup yet.
You have to override the dependencies to resolve apache-arrow as @duckdb/duckdb-wasm/dist/duckdb-esm right now.

I'll add examples for the different bundlers as soon as we get arrow 6.0.1.

[hamilton]

Thanks for the quick response @ankoh! Let me know if you'd like any assistance with documentation or examples when 6.0.1 lands.

[llimllib]

@hamilton here's the esbuild build script I'm using to build with a working version of apache-arrow, not sure exactly how to do it in rollup though

[peder1001]

Also looking forward to this.

Looks like 6.0.1 just released? https://github.com/apache/arrow/releases/tag/apache-arrow-6.0.1
Hopefully won't take them too long to update npm.

[peder1001]

Apache Arrow is now at 6.0.1 on NPM.
https://www.npmjs.com/package/apache-arrow

[llimllib]

I still am not able to build against it, I tried removing my hack and it didn’t work

[ankoh]

So you installed the pre-released version 0.1.13-dev8.0 with

npm install @duckdb/duckdb-wasm@0.1.13-dev8.0

and imported DuckDB-Wasm using only:

import * as duckdb from "@duckdb/duckdb-wasm";

?
I tested this with esbuild and webpack, i'll check rollup later
we definitely need test the major bundlers in the ci

[llimllib]

Yes, here's a repro with esbuild. From a fresh directory:

$ npm install @duckdb/duckdb-wasm@0.1.13-dev8.0 esbuild &&   \
    echo 'import * as duckdb from "@duckdb/duckdb-wasm";' >> test.js && \
    node_modules/.bin/esbuild test.js --bundle --outfile=out.js

added 35 packages, and audited 36 packages in 842ms

found 0 vulnerabilities
 > node_modules/apache-arrow/io/node/builder.js:20:25: error: Could not resolve "stream" (use "--platform=node" when building for node)
    20 │ const stream_1 = require("stream");
       ╵                          ~~~~~~~~

 > node_modules/apache-arrow/io/node/reader.js:21:25: error: Could not resolve "stream" (use "--platform=node" when building for node)
    21 │ const stream_1 = require("stream");
       ╵                          ~~~~~~~~

 > node_modules/apache-arrow/io/node/iterable.js:21:25: error: Could not resolve "stream" (use "--platform=node" when building for node)
    21 │ const stream_1 = require("stream");
       ╵                          ~~~~~~~~

 > node_modules/apache-arrow/io/node/writer.js:21:25: error: Could not resolve "stream" (use "--platform=node" when building for node)
    21 │ const stream_1 = require("stream");
       ╵                          ~~~~~~~~

4 errors

how did you get that to build with esbuild?

[llimllib]

I did manage to get rid of Could not resolve "apache-arrow" though, with all upgrades applied, and if I npm install -D stream it builds

[llimllib]

but then actually using the build fails, the browser console gives:

Uncaught TypeError: class heritage stream_1.Readable is not an object or null
    js iterable.ts:37
    __require viewer_duckdb.js:9
    <anonymous> Arrow.node.ts:22

[domoritz]

With es build, you may need apache/arrow#11742. I'll try to get it released soon.

[peder1001]

Not sure if its useful. But I tried to use it in an Angular project now, and get build errors:

(using @duckdb/duckdb-wasm@0.1.13-dev8.0, and import * as duckdb from '@duckdb/duckdb-wasm';

`Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/duckdb_module.d.ts:1:23 - error TS2688: Cannot find type definition file for 'emscripten'.

1 ///
~~~~~~~~~~

Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/duckdb_module.d.ts:6:39 - error TS2304: Cannot find name 'EmscriptenModule'.

6 export interface DuckDBModule extends EmscriptenModule {
~~~~~~~~~~~~~~~~

Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/duckdb_module.d.ts:7:23 - error TS2304: Cannot find name 'stackSave'.

7 stackSave: typeof stackSave;
~~~~~~~~~

Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/duckdb_module.d.ts:8:24 - error TS2304: Cannot find name 'stackAlloc'.

8 stackAlloc: typeof stackAlloc;
~~~~~~~~~~

Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/duckdb_module.d.ts:9:26 - error TS2304: Cannot find name 'stackRestore'.

9 stackRestore: typeof stackRestore;
~~~~~~~~~~~~

Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/duckdb_module.d.ts:10:19 - error TS2304: Cannot find name 'ccall'.

10 ccall: typeof ccall;
~~~~~

Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/runtime.d.ts:1:23 - error TS2688: Cannot find type definition file for 'emscripten'.

1 ///
~~~~~~~~~~

Error: node_modules/@duckdb/duckdb-wasm/dist/types/src/bindings/runtime.d.ts:27:87 - error TS2503: Cannot find namespace 'Emscripten'.

27 export declare function callSRet(mod: DuckDBModule, funcName: string, argTypes: Array<Emscripten.JSType>, args: Array): [number, number, number];

etc.

[ankoh]

Our benchmarks are built against the esm using esbuild but they are on node, so it works there.
Regarding the typing issue, you'll need @types/emscripten. I'll move it to the dependencies.

[ankoh]

@peder1001 in the meantime, you could use @duckdb/duckdb-wasm/dist/duckdb-browser.js directly. It's an iife.