Feature request: provide pre-compiled binaries (official builds) in npm packages

[Mithgol]

Currently npm install sqlite3 requires building node-sqlite3 from the source with node-gyp (on Windows the prerequisites are Python and MSVC, the latter is huge).

To get rid of this requirement (and thus to attract more end users and less building-related issues reported), please accomplish the following:

1. Require (and use) node-bindings.
2. Compile binary part of node-sqlite3 for the supported platforms (win32/ia32, darwin/x64, linux/ia32, linux/x64... sunos/ia32?), put the resulting binaries inside your npm package and let node-bindings pick one on user's side.
3. Put binding.gyp in .npmignore, otherwise npm will still attempt to build from the source. (Do not forget to add some contents of .gitignore to .npmignore when you create the latter, but only some: npm should not ignore the binaries.)
4. (optional) Put binaries in .gitignore to prevent them from being hosted on GitHub. (This step is necessary only if you hate mixing sources with binaries. This step may also be completed already if you build in build/, which is currently in .gitignore.)

After that npm install sqlite3 would involve only downloading and unpacking, and thus would just work.

[springmeyer]

great, thanks very much for this description. I've known this was possible, just had not ran into good docs yet.

[Mithgol]

I am glad to feel myself helpful, but also let me ask whether you are planning such builds, or waiting for Node 0.8 stable, or something else.

[Mithgol]

As I see now, the last commit in node-sqlite3 master branch happened in July 27 (≈50 days ago), so the module seems more or less complete.

If it indeed is, then you could build the OS-specific binaries once… and update them only when some new SQLite version happens or when a new Node version breaks binary compatibility.

[ErisDS]

Is there any chance that this will ever happen?

[springmeyer]

I think we should provide pre-compiled binaries for windows at least, so yes I think it will happen, either when I find the time or if someone else takes it on.

[ErisDS]

I'm glad there's agreement that it should happen, that's positive at least. I'd love to say I would do it, but it's not really my area of expertise (yet).

[calvinmetcalf]

I've had mixed luck building it on Open Shift (Red Hat's cloud), precompiled binaries that target RHEL/Fedora wouldn't be helpful.

[springmeyer]

update on this: I'd still like to provide windows binaries, but we need to devise a solid way to do this that:

• does not mix binaries for windows in everyone else's download
• does not preclude all other platforms from building using the binding.gyp

If anyone has ideas or best practices from other modules please share.

[Mithgol]

I have an idea.

You could have some preinstall npm script that checks process.platform. If that's 'win32', a pre-built binary is downloaded to /build/Release/node_sqlite3.node. Otherwise the binding.gyp is copied to the package's root from some subdirectory where it was previously stored.

Alternatively you could use binding.gyp's own language to prevent windows builds from building (I don't know any details, it just seems feasible), and then use some postinstall npm script to download a pre-built win32 binary if process.platform === 'win32'.

[springmeyer]

@Mithgol - great ideas. I like the idea of doing it in binding.gyp if possible. This way you could easily pass a flag to npm install that would be passed to node-gyp install that could request a source compile on windows. So the default behavior could be to pull a binary but people like me that want to compile from source could do it easily without having to move around bindings.gyp or remove pre/post install items from the package.json. Another reason I'm more inclined to try to embed the logic in binding.gyp actions is the preinstall confusion I mention at #151 (comment) which is upcoming in recent npm versions.

[Mithgol]

Having read #151 I have to admit that's a good reason… @isaacs toying with preinstall semantics does not leave you any other choice.

[ErisDS]

This all sounds really exciting, but are you only planning to add builds for windows?
Mac doesn't build nicely either unless you have xcode and a tonne of other dependencies installed, and although linux flavors will do a build out of the box, it's doesn't work so nicely on node hosting platforms like nodejitsu and appfog.

Ideally, the builds would be available for all common platforms with a switch to build if you want to.

[springmeyer]

I plan to take a look at hosting binaries on s3 and writing a custom gyp action to, by default/prefer, downloading those precompiled binaries. Some option like npm install sqlite3 --build-from-source could then trigger the current behavior. We could likely learn from https://github.com/mxcl/homebrew/wiki/Bottles#bottle-usage and will also need to check around to see if any other node c++ modules have or have not taken this approach.

[sirber]

Why is is to hard to provide a binary?

[springmeyer]

@sirber - working on it. Can you provide details about what platform and architecture you are seeking a binary for?

[sirber]

win64 please :)

[velizstartup]

win7 32bits :)

[springmeyer]

Okay, progress on this. First a summary of research I did before starting to code this (which I will describe in the next post):

• I looked for whether some of the most commonly installed node addons package binaries and I found they DID NOT. I checked contextify (nearly 20,000 downloads last week), bcrypt (~13,000 downloads last week), and node-ffi (~2500 downloads last week). Are there other highly used C++ modules people know of that do or do not provide binaries?
• I noted this issue which contains various different, but all valid, perspectives: bcrypt binaries for windows 32/64 should be included node_module to ease use/adoption kelektiv/node.bcrypt.js#103
• I gathered from this thread that anything short of an official and secure/checksum-based binary deployment system for node.js should be considered a workaround that will hopefully be removed in the future.
• I noticed that node-ffi in fact once provided binaries and then stopped because of the difficulty in keeping them up to date. Hopefully I will be able to sustain this because node-sqlite3 releases are not super frequent (the module is pretty stable). However @TooTallNate said in a node.js email thread:

I tried doing it for a while in node-ffi (precompiling releases for each platform and arch combo, like 10 total) and my conclusion is that it's a lot of extra work when you're potentially doing releases often. So I decided to stop bundling precompiled binaries for node-ffi with the hopes that a better precompiled binary solution comes along someday.

• I looked around for modules that do provide binaries. Both phantomjs and appjs do, but their use cases are different. Then I found node-fibers which seems to have thought this through well. node-fibers is slightly different than node-sqlite in that it does not depend on an external library, but the approach is solid: it looks by default for a binary and if not found will compile one.

[springmeyer]

1. The https://github.com/developmentseed/node-sqlite3/compare/pre-built branch now contains a bunch of binaries checked into git. THIS IS ONLY FOR TESTING.
2. It looks for a pre-compiled binary for a given <platform>-<architecture>-<V8 version>. If found, you are done. If the binary is not found then a build will commence.
3. So, if we want to support both mac, linux, and windows at both 32 and 64 bit and just one node/v8 version that is 3*2 combinations == 6. Multiply 6 * <number of major v8 api changes> over the range of Node versions you want to support and that will give a rough # of total binaries that people might likely wish for.
4. Looking at the inside of https://registry.npmjs.org/fibers/-/fibers-1.0.1.tgz (since node-fibers does not store binaries in github) I can see node-fibers is supporting back three versions of v8 (although some are missing, notably mine which is osx 64 bit and latest node.js-v8-3.14) so that seems to indicate we will likely need to provide around 6*3 = 18 different binaries. Given the node_sqlite2.node module is about 1MB that would be nearly 20 MB of binaries, so this rules out storing them both in github and the uploaded npm tarball (which node-fibers does).
5. Shrinking the binary is not an option. node_sqlite3.node is 1 MB after this commit: 663b4bf
6. We could start limiting features that are compiled into sqlite3 to reduce the size. For example the FTS feature is quite large: http://www.sqlite.org/footprint.html#relfootprint. But then people needing FTS would not be able to benefit from the pre-compiled binaries which would be bad and confusing.

[springmeyer]

Okay, I've pushed binaries for 32/64 arch + node v0.10.x + windows/mac/linux which tool me about 1 hour of booting up different VM's that I personally pay for, creating a node v0.10.x build environment, and creating and testing the builds.

These only work via a custom branch right now. Please test by doing:

git clone -b pre-built https://github.com/developmentseed/node-sqlite3.git
cd node-sqlite3
npm install mocha
npm test
uname -a

Report your results (if any failures) to a gist and then link here. If things work then provide your uname -a and the fact that things worked.

[springmeyer]

Next steps:

• - devise a sustainable plan for updating these binaries when new releases land, ideally automated and not costing a lot of $ in hardware. Perhaps crowdsourced, but... what about security (see IMPORTANT below)?
• - devise a place to put these binaries since, as per discussion above, they are too big to be in git and the npmjs upload. I'm thinking an s3 bucket.
• - devise a robust way to download them, and in the event of timeout/failure/bad network/no binary available gracefully fallback to building from source. Think of a way to do this so users are not double confused by the potential failure to download a binary and the potential failure to compile from source.
• - figure out a way to cleanly invalidate binaries when new releases of node-sqlite3 are made that change the c++
• - figure out a way for users to submit binaries for unsupported platforms.
• - modify build.js to detect a flag like --build-from-source and then force a source compile
• - IMPORTANT: figure out a way to enforce that those binaries are valid and do not contain viruses or malicious content that will wipe your harddrive.
• - figure out whether, once this is all working, binaries should be opt-in only or on by default. (they should be default provided)

[ErisDS]

Architecture: Linux vagrant-ubuntu-precise-64 3.2.0-43-generic #68-Ubuntu SMP Wed May 15 03:33:33 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Result: Tests run successfully

[ErisDS]

Architecture: Darwin unknown-28:37:37:1c:af:ce-2.home 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64 (Macbook air)
Result: Suitable image not found. Gist is here.

[javorszky]

Architecture: Darwin Gabors-iMac.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64 (iMac 27", mac osx 10.8.4)
Result: No suitable image found. Gist is here

[matthojo]

Architecture: Darwin unknown-28-cf-e9-48-fb-7f.home 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
Result: "no suitable image found." Gist is here

[springmeyer]

Thanks everyone for helping test! Based on these reports it looks the binaries basically all work. The caveat was 64 bit mac I made a mistake and shipped the 32 bit version in the 64 bit folder: Just just fixed this in 1b6d489. The test failures @calvinmetcalf hit can be explained by running a version of node old enough that I did not provide a binary yet and a test timeout (which is common on underpowered systems and not a big deal).

[calvinmetcalf]

I think I had an out of date version of linux but yes I agree that you should't bother to support that.

Thanks for all the hard work

[sirber]

Thank you!!! J

[cobbspur]

Architecture: MINGW32_NT-6.1 HORUS 1.0.12(0.46/3/2) 2011-07-20 17:52 i686 unknown (windows 7 x64 ultimate)

Result: failed to find module

[springmeyer]

@cobbspur - thanks for the error report. I'll try to research this and figure out what might be wrong. Are you around tomorrow/tuesday if I have ideas of a few other things to test?

[springmeyer]

@cobbspur - here is a couple things that would be great if you could try:

1. Try opening the module with pythons ctypes lib. This should produce an error as well, but might provide a popup with more detail. From within the node-sqlite3 folder do:

python -c "import ctypes as c;c.CDLL('bin/win32-x64-v8-3.14/node_sqlite3.node')"

2. Download depends.exe (Dependency Walker) and open up bin/win32-x64-v8-3.14/node_sqlite3.node in it. Then take a screenshot of the entire app view and paste here.

[cobbspur]

Here is the screenie you wanted.

[springmeyer]

Thanks @cobbspur - I have no idea what is wrong, but I'm sure if I can find some leads these screenshots will help.

[springmeyer]

New progress in remote-binary branch. This branch now can pull dynamically from binaries hosted on s3. Because binaries are now remote they are versioned by sqlite3 version (major.minor + manual ABI letter to be exact) as well as arch/v8 version so it is possible that if we do a minor release that only changes package.json or the js files we will not need to re-build any of the binaries (which only represent the c++ binding + libsqlite3`.

Available binaries can be viewed here: http://node-sqlite3.s3.amazonaws.com/index.html

To test do:

git clone -b remote-binary https://github.com/developmentseed/node-sqlite3.git
cd node-sqlite3
npm install
npm install mocha
npm test

[springmeyer]

now storing todo's before tagging and making this live here: https://github.com/developmentseed/node-sqlite3/blob/remote-binary/build.js#L5-L14

[ErisDS]

There are no words to express how nicely this works, so:

🍰 🌈 🎈 🍰 ⭐ 🎈 🍩 🌈 ⭐ 🎈 🍩 🌈 🎈 🍰

Here's the full gist for Win 7 home 64bit

[matthojo]

Mac OS X 10.8
Results: https://gist.github.com/matthojo/b95045b5ea35de0afdbc

[cobbspur]

win7x 64 ultimate as previous

no probs found
gist

[ErisDS]

Worked perfectly on my 64bit macbook air and 32bit Windows XP machines as well.

[springmeyer]

Just changed the binary naming approach to imprive, so now a few binaries are missing. I don't have access to a windows machine for the next several weeks, so @ErisDS I will plan to look to you to create windows binaries when I give you the word. Hopefully early next week, if not the next.

[springmeyer]

update: I was able to re-purpose the previous set of windows binaries against node v0.10.x under my new naming scheme. Just uploaded them, so pulling from the remote-binary branch should work again now without a source compile.

[springmeyer]

work on this is now staged in the binary-deployment branch and close to merging into master, see #192.

[springmeyer]

note: all work has been merged and tagged. sqlite@2.1.16 now provides binaries for common platforms.

[ErisDS]

🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈 🎈

[allouis]

using version 2.1.19 i'm not getting the prebuilt binaries on npm install, are these available in this version? i should note, that npm install is being run from a directory which has sqlite3 as a dependency in the package.json

[springmeyer]

Can you post your install log to a new issue and I will take a look at why binaries are not being installed.

On Feb 17, 2014, at 9:23 AM, allouis notifications@github.com wrote:

using version 2.1.19 i'm not getting the prebuilt binaries on npm install, are these available in this version? i should note, that npm install is being run from a directory which has sqlite3 as a dependency in the package.json

—
Reply to this email directly or view it on GitHub.

[allouis]

Hey just retested on a windows machine and all seems to be working.

[michael-ts]

I am experiencing a problem with the link to pre-compiled binaries provided here and from the documentation. The page just says "loading..." but it never does. If I bring up Chrome developer tools there are two errors:

• Mixed Content: The page at 'https://node-sqlite3.s3.amazonaws.com/index.html?path=Release/' was loaded over HTTPS, but requested an insecure script 'http://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js'. This request has been blocked; the content must be served over HTTPS.
• index.html?path=Release/:16 Uncaught ReferenceError: jQuery is not defined(anonymous function) @ index.html?path=Release/:16

I tried switching to http to get rid of the mixed content warning (not that that's a good solution!), but it automatically re-directs to https.

Is there a new updated URL to obtain the binaries?

Also, are there pre-compiled binaries with sqlcipher support? I'm trying to get this working in Windows 10 and the pre-requisites are ridiculous. I'm sure a lot of people would appreciate not having to go through this.

UPDATE: I got around this problem by using a different browser and bypassing the mixed content issue, but as I don't see any precompiled binaries for it my above question regarding sqlcipher support stands