Node.js: npm install takes c.7mins (Mac M1)

[archiewood]

What happens?

When I try to install duckdb from npm, the process takes 7 minutes on a high spec computer.

We'd like to include DuckDB in our package, but this extends our current install time from around 30s to 7 minutes.

To Reproduce

mkdir test-duckdb
cd test-duckdb
npm init # And press enter to accept all the standard options
npm install duckdb

you can replace the last line with
time npm install duckdb
Which gives you the install time:
real 6m44.266s

OS:

MacOS Ventura - Version 13.0 (22A380)

DuckDB Version:

[Any] / NPM version: 0.5.1

DuckDB Client:

Node.js API

Full Name:

Archie Wood

Affiliation:

Evidence

Have you tried this on the latest master branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree

[Mause]

Which version of NodeJS are you using? And on which architecture?

[archiewood]

$ node -v
v18.10.0
$ npm -v
8.19.2
$ uname -a
Darwin Archies-MacBook-Pro.local 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:15:09 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 arm64

[archiewood]

Adding a bit of colour:

This issue seems to persist on other MacOS version with an M1 chip. MacOS 12.6.1 seems to have the same issue with M1.

However the install time on an older MacOS machine is about 4s.

[hannes]

We may be missing a binary distribution here

[Mause]

I'm not sure if we're actually building NodeJS binaries for M1 at all?

[archiewood]

Very excited for this!

Just tried to test the new prebuilt binaries. It still seemed to kick into building from source?

I was using duckdb@0.5.2-dev1809.0.
Let me know if that's not the correct version, if I should be waiting for a latest release, use a different version of node, etc.

Relevant logs lines:

npm ERR! node-pre-gyp info using node@18.10.0 | darwin | arm64
npm ERR! node-pre-gyp info check checked for "/Users/archie/Projects/test-duckdb/node_modules/duckdb/lib/binding/duckdb.node" (not found)
npm ERR! node-pre-gyp http GET https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for duckdb@0.5.2-dev1809.0 and node@18.10.0 (node-v108 ABI, unknown) (falling back to source compile with node-gyp) 

Full trace:

~/Projects/test-duckdb$ npm i duckdb@0.5.2-dev1809.0
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm ERR! path /Users/archie/Projects/test-duckdb/node_modules/duckdb
npm ERR! command failed
npm ERR! signal SIGINT
npm ERR! command sh -c -- node-pre-gyp install --fallback-to-build
npm ERR! CXX(target) Release/obj.target/duckdb/src/duckdb_node.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/database.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/data_chunk.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/connection.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/statement.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/utils.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/parquet-amalgamation.o
npm ERR!   CXX(target) Release/obj.target/duckdb/src/duckdb.o
npm ERR! node-pre-gyp info it worked if it ends with ok
npm ERR! node-pre-gyp info using node-pre-gyp@1.0.10
npm ERR! node-pre-gyp info using node@18.10.0 | darwin | arm64
npm ERR! node-pre-gyp info check checked for "/Users/archie/Projects/test-duckdb/node_modules/duckdb/lib/binding/duckdb.node" (not found)
npm ERR! node-pre-gyp http GET https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for duckdb@0.5.2-dev1809.0 and node@18.10.0 (node-v108 ABI, unknown) (falling back to source compile with node-gyp) 
npm ERR! node-pre-gyp WARN Hit error response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1809.0-node-v108-darwin-arm64.tar.gz 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@9.1.0
npm ERR! gyp info using node@18.10.0 | darwin | arm64
npm ERR! gyp info ok 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@9.1.0
npm ERR! gyp info using node@18.10.0 | darwin | arm64
npm ERR! gyp info find Python using Python version 3.10.7 found at "/opt/homebrew/opt/python@3.10/bin/python3.10"
npm ERR! gyp info spawn /opt/homebrew/opt/python@3.10/bin/python3.10
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   '/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'make',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/archie/Projects/test-duckdb/node_modules/duckdb/build/config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/archie/Library/Caches/node-gyp/18.10.0/include/node/common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=/Users/archie/Library/Caches/node-gyp/18.10.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=/opt/homebrew/lib/node_modules/npm/node_modules/node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=/Users/archie/Library/Caches/node-gyp/18.10.0/<(target_arch)/node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=/Users/archie/Projects/test-duckdb/node_modules/duckdb',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp info ok 
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@9.1.0
npm ERR! gyp info using node@18.10.0 | darwin | arm64
npm ERR! gyp info spawn make

[Mause]

It looks like the master build failed yeah, I'll look into it today

[archiewood]

Not totally sure and might be off-track here, but I'm not sure that node-gyp supports Python 3.11 yet

[archiewood]

Ah see you have already PRed a fix, ignore

[Mause]

I don't think the GitHub actions runners would have 3.11 by default yet, given it was only released a couple of weeks ago? It looks more like our version of node-gyp was pretty outdated, and I have a PR up (#5221) to fix that, which will hopefully solve these build issues

[whscullin]

It looks like the MacOS node builds are being skipped? I don't see either x64 or arm64 builds for darwin in the CDN at the moment?

[Mytherin]

The OSX node builds on the master seem to be failing at the moment, we are looking into fixing it

[Mause]

The OSX node builds on the master seem to be failing at the moment, we are looking into fixing it

My PR above should fix that issue

[archiewood]

I'm not entirely sure if I have been finding the right action, but the builds still seem to be failing for Node v>15.

When I try npm i duckdb@0.5.2-dev1980.0, it's still falling back to build.

It seems to not be finding a darwin-arm64 binary

npm ERR! node-pre-gyp info using node@18.10.0 | darwin | arm64
npm ERR! node-pre-gyp info check checked for "/Users/archie/Projects/test-duckdb/node_modules/duckdb/lib/binding/duckdb.node" (not found)
npm ERR! node-pre-gyp http GET https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1980.0-node-v108-darwin-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1980.0-node-v108-darwin-arm64.tar.gz 
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for duckdb@0.5.2-dev1980.0 and node@18.10.0 (node-v108 ABI, unknown) (falling back to source compile with node-gyp) 
npm ERR! node-pre-gyp WARN Hit error response status 403 Forbidden on https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev1980.0-node-v108-darwin-arm64.tar.gz 

[Mause]

@archiewood yep, there ended up being another issue I had to fix in #5284, and the bindings are now uploaded - https://duckdb-node.s3.amazonaws.com/duckdb-v0.5.2-dev2131.0-node-v108-darwin-arm64.tar.gz

If you could test this out before the release of 0.6, that would be very much appreciated

[archiewood]

It's working! Installs in 5s!

time npm i duckdb@0.5.2-dev2131.0

npm WARN deprecated @npmcli/move-file@2.0.1: This functionality has been moved to @npmcli/fs

added 109 packages, and audited 110 packages in 5s

6 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

real	0m5.668s