Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Added support for "node-gyp" build #61

Merged
merged 36 commits into from

7 participants

@TooTallNate

Hey guys, so this should take care of #60. I started with the current windows branch and worked from there. I created a gyp file for sqlite3 itself, and then declared that as a dependency in node-sqlite3's bindings.gyp file.

Try it out and let me know what you think! Cheers!

springmeyer and others added some commits
@springmeyer springmeyer add ability to configure against boost threads if requested 8aec7ad
@springmeyer springmeyer add a threading compatibility shim - likely cleaner ways of doing thi…
…s but we don't want to depend on boost threads unless we really have to for now
9ee3c90
@springmeyer springmeyer enable usage of boost threads instead of direct posix/pthreads usage …
…- all tests are passing on OS X 10.7 with boost 1.47 - todo: test on linux - refs #48
357f57b
@springmeyer springmeyer be explicit about headers 1e57acc
@springmeyer springmeyer handle up front error in affected.test.js 0e4d62f
@springmeyer springmeyer Revert change to affected.test.js 501caf0
@springmeyer springmeyer fix configure 50c1d42
Konstantin Käfer Merge branch 'cross-platform-threading' into windows
Conflicts:
	src/async.h
	src/statement.h
	wscript
8d7eb6e
Konstantin Käfer remove trailing whitespace 963295e
Konstantin Käfer use win32 mutexes 3fb599b
Konstantin Kaefer actually define type ad6cda0
Konstantin Kaefer add visual studio projects
they still contain absolute paths
fed5417
Konstantin Kaefer some tweaks for windows support e7c3afb
Konstantin Kaefer add paths to gitignore and sample propsheet 78f2eec
@springmeyer springmeyer add gyp files for building without messing with visual studio UI e6b1f79
@springmeyer springmeyer fully clear out stale builds 55f7cfb
@TooTallNate TooTallNate Add "expresso" to devDependencies. 9b7b922
@TooTallNate TooTallNate Consistent spacing in the package.json. 7ef826c
@TooTallNate TooTallNate Prepare the bindings.gyp file. 82f63cb
@TooTallNate TooTallNate A useful .gitignore file. 2556985
@TooTallNate TooTallNate Remove compiled Visual Studio stuff. This shouldn't get comitted. b70da5d
@TooTallNate TooTallNate node-gyp in the ./configure and Makefile. 0a0b316
@TooTallNate TooTallNate Add unarchived sqlite3 bundled dependency.
This is how node does it so let's stay consistent.
c062c58
@TooTallNate TooTallNate Remove the tarball. 8a4e67d
@TooTallNate TooTallNate Use node-bindings. 2b91175
@TooTallNate TooTallNate Add a minimally working gyp file to build sqlite3 as a static lib. eeda206
@TooTallNate TooTallNate Ignore test/support/big.db. 5c5ab6c
@cmundi

node-gyp configure
node-gyp build
copy %BASE%\Release\node_sqlite3.node %BASE%\lib\node_sqlite3.node

All attempts to require('sqlite3') fail with "cannot find module 'bindings'" so I patched %BASE%\Release\sqlite3.js as follows

< var sqlite3 = module.exports = exports = require('bindings')('node_sqlite3.node');
---
> var sqlite3 = module.exports = exports = require('node_sqlite3.node');

and now the example code in the README.md runs as expected. I have not tried to run the test suite yet.

@cmundi

Sorry about the ugly formatting of that last comment. It looked fine before I hit the submit button.

@TooTallNate

@cmundi You need to npm install bindings :)

@TooTallNate

And the copy step should not be necessary.

@cmundi

D'oh! Yes. I am denser than usual tonight. Thanks!

@cmundi

FYI,anyone who cares about node-sqlite3 on Windows should be aware of how test/createdb.js seems to reveal a problem specific to Windows: #62

@TooTallNate

@cmundi I added some missing defines to sqlite3's gyp file. I'm not sure if it made any difference for windows, but on OS X I got 100% tests passing (and the big.db was created as expected).

@cmundi

Thanks. I started trying this out (on yet another machine, since I'm traveling) and got an uncaught exception from node-gyp when configuring node-sqlite3. The details are here: TooTallNate/node-gyp#22

@nick7ikin

Can't build sqlite3 with node-gyp, please help.

Environment: windows xp prof sp3, Microsoft Visual C++ 2010 Express.

My steps:
1. download the windows branch
2. cd to the target forlder and run node-gyp configure:

C:\Program Files\nodejs\temp\sqlite3>node-gyp configure
info it worked if it ends with ok
info downloading: http://nodejs.org/dist/v0.6.0/node-v0.6.0.tar.gz
info downloading `node.lib` http://nodejs.org/dist/v0.6.10/node.lib
spawn python [ '.node-gyp\\0.6\\tools\\gyp_addon',
  '-Dnode_root_dir=.node-gyp\\0.6',
  '-I',
  '.node-gyp\\0.6\\tools\\patch.gypi',
  '-Dtarget_arch=ia32' ]
Warning: Missing input file deps\sqlite3\sqlite3.c pwd=C:\Program Files\nodejs\temp\sqlite3
info done ok

it tells me that there is missing dependency file sqlite3.c.
3. I moved to the 'deps' folder and there is sqlite-autoconf-3070800.tar.gz. Unpack it in sqlite3 folder
4. Perform node-gyp configure again:

C:\Program Files\nodejs\temp\sqlite3>node-gyp configure
info it worked if it ends with ok
spawn python [ '.node-gyp\\0.6\\tools\\gyp_addon',
  '-Dnode_root_dir=.node-gyp\\0.6',
  '-I',
  '.node-gyp\\0.6\\tools\\patch.gypi',
  '-Dtarget_arch=ia32' ]
info done ok

It seems to be ok for now
5. Execute node-gyp build:

C:\Program Files\nodejs\temp\sqlite3>node-gyp build
info it worked if it ends with ok
spawn C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\msbuild.exe [ 'bindings.sln',
  '/clp:Verbosity=minimal',
  '/nologo',
  '/p:Configuration=Release' ]
MSBUILD : error MSB1009: ?? ???? ?? ???????.
????: bindings.sln
ERR! `C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\msbuild.exe` failed with exit code: 1
ERR! not ok

Sorry for question marks, it seems it have to be russian letters, but windows console isn't friendly with Russian. Anyway I see that error is caused by bindings.sln file, actually by its absence. I tried to find bindings.sln in sqlite3 sources tree, but have no success.

Please tell me where can I find this bindings.sln and where to put it. Thanks.

@TooTallNate

@nick7ikin Don't use the windows branch on the main repo here, instead use the gyp branch on my fork.

Here's a direct download link to the zipball: https://github.com/TooTallNate/node-sqlite3/zipball/gyp

@TooTallNate

@kkaefer or any other @developmentseed person. Have you guys tried out these changes yet? It seems like it would help out a lot of the Windows people if you merged these soon.

@nick7ikin

@TooTallNate
Googling for bindings.sln I found the answer here.
But your link works without any renaming, thanks for it.

@springmeyer
Admin

Hi @TooTallNate - I will take a look at integrating tomorrow, thanks! /cc @kkaefer

@kkaefer
Admin

How am I supposed to install node-sqlite3 with npm using gyp? Do I have to manually install node-gyp beforehand? Why is node-gyp downloading a tarball of node 0.6.0?

@TooTallNate

How am I supposed to install node-sqlite3 with npm using gyp?

I haven't tried this, but I think this should be the easiest way to test it using npm and node-gyp:

$ npm install -g node-gyp   # install node-gyp
$ npm install https://github.com/TooTallNate/node-sqlite3/tarball/gyp  # install my "gyp" branch fork
$ npm explore sqlite3  # enter the node-sqlite3 directory
$ node-gyp configure  # configure using node-gyp
$ node-gyp build  # build using node-gyp
$ exit  # leave the `npm explore` subshell
$ node -e "console.log(require('sqlite3'))"  # <-- should work fine

Do I have to manually install node-gyp beforehand?

Yes

Why is node-gyp downloading a tarball of node 0.6.0?

It does this because on Windows the dev header files never get installed anywhere, nor does the node.lib file required for compiling native modules. So node-gyp downloads these from http://nodejs.org/dist.

Cheers!

@springmeyer
Admin

Why is node-gyp downloading a tarball of node 0.6.0?

It does this because on Windows the dev header files never get installed anywhere, nor does the node.lib file required for compiling native modules. So node-gyp downloads these from http://nodejs.org/dist.

So, is node-gyp just for windows? I'm seeing this on osx, where the headers for node 0.6.0 are being downloaded even though I'm running node 0.6.11, installed from source (with headers) - this seems like trouble.

@springmeyer
Admin

Trying your example I get: https://gist.github.com/1994428. It appears it is still trying to use the wscript.

@TooTallNate

@springmeyer node-gyp is for all platforms that node supports. So yes, it downloads the dev files on OS X as well, for consistency. Basically the .gyp files from the node source tree are set up to expect to be run inside the node source tree. This leaves the headers installed from make install rather useless, so node-gyp downloads them manually and places them in the proper directory structure.

So in the end: you are seeing normal behavior.

@TooTallNate

@springmeyer The wscript file still being invoked seems to be some annoying npm behavior for when it sees a wscript file. I'll delete the old wscript file from my branch to fix that.

@TooTallNate TooTallNate Remove the wscript file.
So that npm doesn't try to automatically compile and use it on an
`npm install` of the tarball.
f82a33f
@TooTallNate

@springmeyer Ok try one more time :) I just tried the procedure I laid out above and it worked for me.

@kkaefer
Admin

so you can no longer type 'npm install sqlite3' but have to manually install node-gyp beforehand? I'm a bit worried about the added complexity and users failing to install node-sqlite3. Do you recommend adding "npm install -g node-gyp" to the preinstall script?

@kkaefer
Admin

Still, shouldn't node-gyp look at the current node version and download the appropriate distribution?

@TooTallNate

@kkaefer Well I know that @isaacs is leaning towards phasing out the preinstall script, most likely in favor of precompiled binaries, but that will take some modifications to npm I believe.

The idea is that normal users shouldn't have to worry about compiling binaries (similar to how they download a .pkg or .msi now), and npm should have the module binaries ready to go when the user types in npm install sqlite3. That's the idea, but there's a transition phase which we are in right now that makes the setup seem more awkward.

@TooTallNate

Still, shouldn't node-gyp look at the current node version and download the appropriate distribution?

That's what it does! Downloads the .tar.gz for the current (minor) version that node is running.

@kkaefer
Admin

So you're saying that there are no changes in header files between patch levels?

@TooTallNate

@kkaefer Correct, that and node minor versions have binary stability, so you don't need to target your specific node version, only the minor version.

@springmeyer
Admin

@kkaefer Correct, that and node minor versions have binary stability, so you don't need to target your specific node version, only the minor version.

node might but not the v8 version that is bundled, in my experience.

@springmeyer
Admin

@kkaefer Correct, that and node minor versions have binary stability, so you don't need to target your specific node version, only the minor version.

node might but not the v8 version that is bundled, in my experience.

wow, I just checked and I'm delighted I'm wrong on that:

~/src$ diff node-v0.6.0/deps/v8/include/v8.h node-v0.6.11/deps/v8/include/v8.h 
~/src$ 
@springmeyer
Admin

however a difference like this could mean that node 0.6.0 headers might not work on windows:

~/src$ diff -u node-v0.6.0/src/node_object_wrap.h node-v0.6.11/src/node_object_wrap.h 
--- node-v0.6.0/src/node_object_wrap.h  2011-11-04 23:48:57.000000000 -0700
+++ node-v0.6.11/src/node_object_wrap.h 2012-02-17 12:39:45.000000000 -0800
@@ -22,12 +22,13 @@
 #ifndef object_wrap_h
 #define object_wrap_h

+#include <node.h>
 #include <v8.h>
 #include <assert.h>

 namespace node {

-class ObjectWrap {
+class NODE_EXTERN ObjectWrap {
  public:
   ObjectWrap ( ) {
     refs_ = 0;
@TooTallNate

@springmeyer Thanks for pointing that out. node_buffer.h will probably be similar. That may be a good reason to bump the tar.gz download version for the 0.6 target.

This shouldn't happen anymore though, and I'll be keeping an eye on the node commits to make sure.

@cmundi
@TooTallNate

@springmeyer The latest version of node-gyp (which is now bundled with npm, starting from v1.1.8) actually downloads dev files for the specific version of node that is currently running, not the minor version, so your concerns about differences in the header files should be taken care of now.

@Mithgol Mithgol referenced this pull request
Closed

windows port #48

@kkaefer
Admin

Looked at this again, but it seems like the most recent node stable (0.6.15) still comes with 1.1.16 which doesn't seem to include node-gyp.

@isaacs

Yes, it does: https://github.com/joyent/node/blob/v0.6.15/deps/npm/node_modules/node-gyp/package.json

It just doesn't install it separately. If you have a wscript, and a binding.gyp, and no explicit install script defined, then npm will default to using the binding.gyp in recent versions, or the wscript in older versions that didn't have node-gyp yet.

@Mithgol
Collaborator

See also https://github.com/isaacs/npm/blob/master/lib/utils/read-json.js#L360 and a dozen of immediately following lines.

@kkaefer
Admin

Ah, so it's there but not in PATH? Or did nvm play a trick on my and not add that to the path?

@Mithgol
Collaborator

It's not in PATH.

At least, npm update npm -g never puts node-gyp in PATH (and should not, because node-gyp is not installed separately this way, but only as a subfolder of npm/node_modules/).

If you need a separate install of node-gyp for some reason, then run npm install -g node-gyp (as README at https://github.com/TooTallNate/node-gyp recommends).

And after a quick look at package.json it seems that it won't extend PATH anyway, so you have to add that module's path in PATH manually if you need it there.

I know that node-gyp's README file contain just node-gyp commands without any path, as if they were in your PATH already, but I guess actually they are not. (That's a guess and not my personal experience, so @TooTallNate may correct me if I'm wrong.)

@TooTallNate

@kkaefer It's like @isaacs said: npm bundles it's own (internal) copy of node-gyp which is used when installing packages that don't have an "install" script phase, but do have a "binding.gyp" file in the root of the package. This internal copy does not get installed into your PATH; it's only used by npm for commands like npm install.

If you want to have node-gyp in your PATH (which is useful for development of native modules, but not necessary for user installation, since npm has its own copy), the you can npm install -g node-gyp and have your own global copy installed.

Hope that clears things up!

@kkaefer kkaefer merged commit 427fd32 into from
@isaacs

When npm runs node-gyp, it runs the node-gyp that it bundles, always. It sets up the PATH in the subprocess to be such that it will work out that way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 6, 2011
  1. @springmeyer
  2. @springmeyer

    add a threading compatibility shim - likely cleaner ways of doing thi…

    springmeyer authored
    …s but we don't want to depend on boost threads unless we really have to for now
  3. @springmeyer

    enable usage of boost threads instead of direct posix/pthreads usage …

    springmeyer authored
    …- all tests are passing on OS X 10.7 with boost 1.47 - todo: test on linux - refs #48
  4. @springmeyer

    be explicit about headers

    springmeyer authored
Commits on Nov 7, 2011
  1. @springmeyer
  2. @springmeyer
  3. @springmeyer

    fix configure

    springmeyer authored
Commits on Nov 8, 2011
  1. Merge branch 'cross-platform-threading' into windows

    Konstantin Käfer authored
    Conflicts:
    	src/async.h
    	src/statement.h
    	wscript
  2. remove trailing whitespace

    Konstantin Käfer authored
  3. use win32 mutexes

    Konstantin Käfer authored
  4. actually define type

    Konstantin Kaefer authored
  5. add visual studio projects

    Konstantin Kaefer authored
    they still contain absolute paths
Commits on Nov 19, 2011
  1. some tweaks for windows support

    Konstantin Kaefer authored
  2. add paths to gitignore and sample propsheet

    Konstantin Kaefer authored
Commits on Jan 6, 2012
  1. @springmeyer
Commits on Jan 24, 2012
  1. @springmeyer
Commits on Feb 13, 2012
  1. @TooTallNate
  2. @TooTallNate
  3. @TooTallNate
  4. @TooTallNate

    A useful .gitignore file.

    TooTallNate authored
  5. @TooTallNate
  6. @TooTallNate
  7. @TooTallNate

    Add unarchived sqlite3 bundled dependency.

    TooTallNate authored
    This is how node does it so let's stay consistent.
  8. @TooTallNate

    Remove the tarball.

    TooTallNate authored
  9. @TooTallNate

    Use node-bindings.

    TooTallNate authored
  10. @TooTallNate
  11. @TooTallNate
  12. @TooTallNate
  13. @TooTallNate

    Add some defines from the autoconf configuration.

    TooTallNate authored
    No idea really if this helps anything...
  14. @TooTallNate
Commits on Mar 7, 2012
  1. @TooTallNate

    Remove the wscript file.

    TooTallNate authored
    So that npm doesn't try to automatically compile and use it on an
    `npm install` of the tarball.
  2. @TooTallNate
  3. @TooTallNate

    Revert "Use node-bindings."

    TooTallNate authored
    This reverts commit 2b91175.
  4. @TooTallNate
Commits on Mar 26, 2012
  1. @TooTallNate

    Revert "Remove the wscript file."

    TooTallNate authored
    This reverts commit f82a33f.
  2. @TooTallNate
Something went wrong with that request. Please try again.