WASM-based parquet #2103

kylebarron · 2022-03-08T01:40:27Z

No description provided.

modules/parquet/src/lib/parse-geoparquet.ts

ibgreen · 2022-03-08T19:14:53Z

modules/parquet/package.json

@@ -42,6 +42,7 @@
    "@loaders.gl/compression": "3.2.0-alpha.1",
    "@loaders.gl/loader-utils": "3.2.0-alpha.1",
    "@loaders.gl/schema": "3.2.0-alpha.1",
+    "apache-arrow": "^7.0.0",


Let's discuss... arrow 7 is a breaking change. I have a PR that updates to arrow 7 but have been holding it off for loaders.gl 4.0.

Or make apache-arrow a peer dependency? The only functions used are arrow.tableFromIPC (for reading) and arrow.tableToIPC (for writing). Arrow 7 isn't necessary; I just did yarn add apache-arrow and it chose the latest version

The problem is that those functions have changed name compared to pre v7 releases. v7 was a major refactor that massively improves tree-shaking of arrow but there are a bunch of breaking changes.

Yes, I dropped the dev dependencies version down to ^4 and had to reimplement both tableToIPC and tableFromIPC using the lower level RecordBatchFileWriter and RecordBatchStreamReader.

loaders.gl/modules/parquet/src/lib/encode-parquet-wasm.ts

Lines 26 to 28 in 26317f7

export function tableToIPC(table: Table): Uint8Array {

return RecordBatchFileWriter.writeAll(table).toUint8Array(true);

}

loaders.gl/modules/parquet/src/lib/parse-parquet-wasm.ts

Lines 21 to 28 in 26317f7

function tableFromIPC(input: ArrayBuffer): Table {

const reader = RecordBatchStreamReader.from(input);

const recordBatches: RecordBatch[] = [];

for (const recordBatch of reader) {

recordBatches.push(recordBatch);

}

return new Table(recordBatches);

}

modules/parquet/test/index.js

kylebarron · 2022-03-08T22:22:18Z

modules/parquet/test/data/files.js

+  {supportedJs: false, supportedWasm: true, title: 'data_index_bloom_encoding_stats', path: 'good/data_index_bloom_encoding_stats.parquet'},
+  {supportedJs: false, supportedWasm: false, title: 'delta_binary_packed', path: 'good/delta_binary_packed.parquet'},
+  {supportedJs: false, supportedWasm: false, title: 'delta_byte_array', path: 'good/delta_byte_array.parquet'},
+  {supportedJs: false, supportedWasm: false, title: 'delta_encoding_optional_column', path: 'good/delta_encoding_optional_column.parquet'},
+  {supportedJs: false, supportedWasm: false, title: 'delta_encoding_required_column', path: 'good/delta_encoding_required_column.parquet'},


These are new files from https://github.com/apache/parquet-testing that didn't previously exist here.

ibgreen · 2022-03-08T22:45:28Z

modules/parquet/src/lib/encode-parquet-wasm.ts

+ * @param type Whether to serialize the Table as a file or a stream.
+ */
+export function tableToIPC(table: Table): Uint8Array {
+  return RecordBatchFileWriter.writeAll(table).toUint8Array(true);


Is writeAll() still present in v7 though? Those kind of static methods were a big part of what was removed in v7...

In this case yes. The above is the definition of tableToIPC in v7:
https://github.com/apache/arrow/blob/e90472e35b40f58b17d408438bb8de1641bfe6ef/js/src/ipc/serialization.ts#L24-L51

kylebarron · 2022-04-12T20:02:30Z

I think the main thing that needs to be fixed in this PR is how to bundle the wasm in a way that works out of the box for node and browser. I tried to use a dynamic import(), though that gives an error when building:

modules/parquet/src/lib/encode-parquet-wasm.ts:10:24 - error TS1323: Dynamic imports are only supported when the '--module' flag is set to 'es2020', 'es2022', 'esnext', 'commonjs', 'amd', 'system', 'umd', 'node12', or 'nodenext'.

10   const module = await import('parquet-wasm/esm/arrow1');
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Previously, it was in an import/require ESM/CJS mess. Node was complaining about the use of require, even when TS was using import, so it's likely the current tsconfig has something to do with that.

I'm not too experienced with these bundling issues so help would be greatly appreciated 🙂

kylebarron · 2022-04-26T17:11:51Z

modules/core/src/iterators/make-stream/make-dom-stream.ts

@@ -34,6 +34,7 @@ export function makeStream<ArrayBuffer>(
            controller.close();
          } else {
            // TODO - ignores controller.desiredSize
+            // @ts-ignore


Should change this into a @ts-expect-error. I got an error about this originally but maybe it's now fixed on master.

modules/parquet/src/lib/load-wasm/load-wasm-browser.ts

felixpalmer

Very excited for this PR! Added a few comments, although I must confess I have no experience bundling WASM modules so I may be totally off-base

modules/parquet/src/lib/parse-parquet-wasm.ts

felixpalmer · 2022-04-27T12:52:22Z

modules/parquet/test/parquet-wasm-loader.spec.ts

+import {WASM_SUPPORTED_FILES} from './data/files';
+
+const PARQUET_DIR = '@loaders.gl/parquet/test/data/apache';
+const wasmUrl = 'http://localhost:5000/node_modules/parquet-wasm/arrow1/esm_bg.wasm';


The whole flow of referencing an external wasm file and using the localhost:5000 override seems close to the flow we have for webworkers. Would it make sense to unify the two? I.e. create a small web worker script that includes the WASM and implements the parse interface? I know this would mean losing the non-worker loader, but how much use is this if need to dynamically load a WASM blob from an external URL?

And yes, I think both WASM and web workers need to be instantiated asynchronously, right? I don't know much about packaging web workers. In the geopackage loader (which relies on the WASM sql.js) we have a similar process where the WASM is loaded from an external source that the user can configure:

loaders.gl/modules/geopackage/src/lib/parse-geopackage.ts

Lines 120 to 131 in 7acfc31

async function loadDatabase(arrayBuffer: ArrayBuffer, sqlJsCDN: string | null): Promise<Database> {

// In Node, `locateFile` must not be passed

let SQL: SqlJsStatic;

if (sqlJsCDN) {

SQL = await initSqlJs({

locateFile: (file) => `${sqlJsCDN}${file}`

});

} else {

SQL = await initSqlJs();

}

return new SQL.Database(new Uint8Array(arrayBuffer));

}

I was more thinking in terms of packaging. A number of loaders have a build-worker target in package.json which then automatically creates a worker file on the npm and provides a way to use a local version of the worker in unit tests. The idea was that you could create a similar build-worker step for parquet-wasm which would abstract away all the WASM loading into that worker

@felixpalmer In principle that is a good idea. Are you good at wasm bundling / loading? I have looked into this briefly in the past but overall this seemed like a messy corner of javascript and I didn't find any approach that I got excited about. If someone can demonstrate a convincing solution that seems reasonably simple, generic, portable and future proof I'd be happy to consider it.

I'm afraid I have no experience and share the view that it seems to be a messy part of JavaScript

modules/parquet/package.json

modules/parquet/src/parquet-wasm-loader.ts

modules/core/src/iterators/make-stream/make-dom-stream.ts

yarn.lock

kylebarron · 2022-04-28T19:18:39Z

Thanks. Should be finally green now.

WIP: wasm-based parquet

98d2e6a

kylebarron commented Mar 8, 2022

View reviewed changes

modules/parquet/src/lib/parse-geoparquet.ts Outdated Show resolved Hide resolved

scaffold

1328c3e

ibgreen reviewed Mar 8, 2022

View reviewed changes

kylebarron added 4 commits March 8, 2022 13:48

WIP Add parquet wasm loader and writer

7feb20d

Minimal test of parquet wasm loader passing

bb92876

Update parquet wasm tests

26317f7

Remove geo-specific code

6942269

kylebarron commented Mar 8, 2022

View reviewed changes

Remove geoparquet test files, uncomment code

b167770

ibgreen approved these changes Mar 8, 2022

View reviewed changes

This was referenced Apr 9, 2022

Parquet Maximum call stack size exceeded error & simple wasm benchmark #2144

Open

Failed to execute 'importScripts' on 'WorkerGlobalScope' #2129

Closed

kylebarron added 3 commits April 12, 2022 11:23

Merge branch 'master' into kyle/parquet-wasm

85f2cc5

Bump parquet-wasm to 0.3.0

2eda63b

Try to use dynamic import

0957585

kylebarron added 2 commits April 25, 2022 21:52

Fix test

aabd4e6

Working parquet bundling!

c177f3d

kylebarron mentioned this pull request Apr 26, 2022

esm2 entrypoint with no import.meta.url kylebarron/parquet-wasm#110

Merged

kylebarron added 4 commits April 26, 2022 10:47

Bump version to 0.3.1

8aba428

lint

15ef76a

Merge branch 'master' into kyle/parquet-wasm

b2e11a9

Remove test.only

7b8c199

kylebarron marked this pull request as ready for review April 26, 2022 17:09

kylebarron commented Apr 26, 2022

View reviewed changes

modules/parquet/src/lib/load-wasm/load-wasm-browser.ts Outdated Show resolved Hide resolved

kylebarron added 2 commits April 26, 2022 11:28

Move into wasm/ folder

a8aac33

Pass through wasmUrl option

a745c40

kylebarron changed the title ~~WIP: wasm-based parquet~~ WASM-based parquet Apr 26, 2022

felixpalmer approved these changes Apr 27, 2022

View reviewed changes

kylebarron commented Apr 27, 2022

View reviewed changes

modules/parquet/src/parquet-wasm-loader.ts Outdated Show resolved Hide resolved

kylebarron commented Apr 27, 2022

View reviewed changes

modules/parquet/src/parquet-wasm-loader.ts Outdated Show resolved Hide resolved

kylebarron added 7 commits April 27, 2022 14:28

Update loader id and set worker to false

1d3ea3e

Fix loading of esm bundle

587ed24

Update list of supported files in wasm

0293a5c

Fix test data url

874a230

Remove ts-ignore

ede990a

Merge branch 'master' into kyle/parquet-wasm

ec5b7ee

Add ts-expect-error

b57cb59

kylebarron commented Apr 28, 2022

View reviewed changes

modules/core/src/iterators/make-stream/make-dom-stream.ts Outdated Show resolved Hide resolved

ibgreen reviewed Apr 28, 2022

View reviewed changes

yarn.lock Outdated Show resolved Hide resolved

Revert unintentional ts upgrade

91f3bc8

kylebarron merged commit 4a35b19 into master Apr 28, 2022

kylebarron deleted the kyle/parquet-wasm branch April 28, 2022 19:46

jorisvandenbossche mentioned this pull request Jun 3, 2022

"GeoParquet" and "GeoArrow" #714

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WASM-based parquet #2103

WASM-based parquet #2103

kylebarron commented Mar 8, 2022

ibgreen Mar 8, 2022

kylebarron Mar 8, 2022 •

edited

ibgreen Mar 8, 2022

kylebarron Mar 8, 2022

kylebarron Mar 8, 2022

ibgreen Mar 8, 2022

kylebarron Mar 8, 2022

kylebarron commented Apr 12, 2022

kylebarron Apr 26, 2022

felixpalmer left a comment

felixpalmer Apr 27, 2022

kylebarron Apr 27, 2022

felixpalmer Apr 28, 2022

ibgreen Apr 28, 2022

felixpalmer Apr 29, 2022

kylebarron commented Apr 28, 2022

	export function tableToIPC(table: Table): Uint8Array {
	return RecordBatchFileWriter.writeAll(table).toUint8Array(true);
	}

	function tableFromIPC(input: ArrayBuffer): Table {
	const reader = RecordBatchStreamReader.from(input);
	const recordBatches: RecordBatch[] = [];
	for (const recordBatch of reader) {
	recordBatches.push(recordBatch);
	}
	return new Table(recordBatches);
	}

	async function loadDatabase(arrayBuffer: ArrayBuffer, sqlJsCDN: string \| null): Promise<Database> {
	// In Node, `locateFile` must not be passed
	let SQL: SqlJsStatic;
	if (sqlJsCDN) {
	SQL = await initSqlJs({
	locateFile: (file) => `${sqlJsCDN}${file}`
	});
	} else {
	SQL = await initSqlJs();
	}
	return new SQL.Database(new Uint8Array(arrayBuffer));
	}

WASM-based parquet #2103

WASM-based parquet #2103

Conversation

kylebarron commented Mar 8, 2022

Choose a reason for hiding this comment

kylebarron Mar 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kylebarron commented Apr 12, 2022

Choose a reason for hiding this comment

felixpalmer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kylebarron commented Apr 28, 2022

kylebarron Mar 8, 2022 •

edited