pkg+sequelize+sqlite fails

[grempe]

I've created the following repository to be a simple test case you can run to see the issue I am having packaging up a simple app that makes use of Sequelize w/ a sqlite data store. Clone this repo for a working example of the errors we are seeing. Is it possible to make this work?

https://github.com/grempe/pkgtest

Here is a summary of the issue:

Bundle with pkg using yarn build throws warnings:

$ yarn build
yarn build v0.24.6
$ ./node_modules/.bin/pkg . --out-path ./build/
> pkg@4.1.0
> Targets not specified. Assuming:
  node8-linux-x64, node8-macos-x64, node8-win-x64
> Warning Cannot resolve 'config'
  /Users/glenn/Desktop/pkgtest/node_modules/pkg/lib-es5/index.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> Warning Cannot resolve '_path2.default.join(dd, file)'
  /Users/glenn/Desktop/pkgtest/node_modules/pkg/lib-es5/walker.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> Warning Cannot resolve 'envFilePath'
  /Users/glenn/Desktop/pkgtest/node_modules/env-cmd/lib/index.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
✨  Done in 12.29s.

Running the resultant executable throws the following errors:

 $ ./build/pkgtest-macos
/snapshot/pkgtest/node_modules/sequelize/lib/dialects/sqlite/connection-manager.js:31
        throw new Error('Please install sqlite3 package manually');
        ^

Error: Please install sqlite3 package manually
    at new ConnectionManager (/snapshot/pkgtest/node_modules/sequelize/lib/dialects/sqlite/connection-manager.js:31:15)
    at new SqliteDialect (/snapshot/pkgtest/node_modules/sequelize/lib/dialects/sqlite/index.js:14:30)
    at new Sequelize (/snapshot/pkgtest/node_modules/sequelize/lib/sequelize.js:226:20)
    at Object.<anonymous> (/snapshot/pkgtest/index.js:3:19)
    at Module._compile (pkg/prelude/bootstrap.js:1136:22)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (pkg/prelude/bootstrap.js:1192:12)
    at startup (bootstrap_node.js:200:16)
    at bootstrap_node.js:617:3

Thanks very much for any guidance.

[igorklopov]

Fixed in pkg@4.1.1. Please try again.

[grempe]

Yes, it looks like that fixed it and the build no longer throws an error. Thank you.

➜  pkgtest git:(master) ✗ yarn build
yarn build v0.24.6
$ ./node_modules/.bin/pkg . --out-path ./build/
> pkg@4.1.1
> Targets not specified. Assuming:
  node8-linux-x64, node8-macos-x64, node8-win-x64
> Warning Cannot resolve 'config'
  /Users/glenn/src/grempe/pkgtest/node_modules/pkg/lib-es5/index.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> Warning Cannot resolve '_path2.default.join(dd, file)'
  /Users/glenn/src/grempe/pkgtest/node_modules/pkg/lib-es5/walker.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> Warning Cannot resolve 'envFilePath'
  /Users/glenn/src/grempe/pkgtest/node_modules/env-cmd/lib/index.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
✨  Done in 11.43s.

➜  pkgtest git:(master) ✗ ./build/pkgtest-macos
Executing (default): DROP TABLE IF EXISTS `users`;
Executing (default): CREATE TABLE IF NOT EXISTS `users` (`id` INTEGER PRIMARY KEY AUTOINCREMENT, `name` VARCHAR(255), `createdAt` DATETIME NOT NULL, `updatedAt` DATETIME NOT NULL);
Executing (default): PRAGMA INDEX_LIST(`users`)
Executing (default): INSERT INTO `users` (`id`,`name`,`createdAt`,`updatedAt`) VALUES (NULL,'John','2017-06-27 05:52:52.582 +00:00','2017-06-27 05:52:52.582 +00:00');
Executing (default): SELECT count(*) AS `count` FROM `users` AS `user`;
Executing (default): SELECT `id`, `name`, `createdAt`, `updatedAt` FROM `users` AS `user`;
1

[mawelsh]

@grempe Did you ever distribute the resulting binary anywhere (move it to another machine or anything)? I'm having a terrible time with the sqlite package and portability. Even across identical machines running the same base images where I'm building on one and running on the other I'm getting the same:

throw new Error('Please install sqlite3 package manually');

behavior I was seeing before 4.1.1. I'm just curious if you've seen this behavior at all and if you've done anything to resolve it.

[igorklopov]

1 . Include sqlite3 package to dependencies in your package.json and use pkg package.json syntax (or just pkg .)
or
2 . Use pkg index.js syntax, but insert dummy require('sqlite3') into your index.js. To make pkg include sqlite3 package explicitly.

[grempe]

@mawelsh We didn't. We were short on time, and it wasn't known if this issue was being addressed or not, and so we implemented and alternative local DB in our CLI client (NeDB), replacing Sequelize + sqlite3, which has worked well for us in testing so far and has no binary dependencies and so avoids these issues.

I only tested that in my simple test case repository that the build seemed to work correctly now where before it did not.

[mawelsh]

Thanks for the responses. It's in my package.json and installs locally. I'm beginning to assume the resulting binary sqlite3-lib that sqlite3 is building is non-portable. The pkg build runs fine as long as it's run from the folder it's created in, but after a bit more expirimentation copying it anywhere (even locally on the building machine) causes a failure.

The full error I'm seeing is:
Error: Please install sqlite3 package manually
at new ConnectionManager (C:\snapshot\policy-management-api\node_modules\sequelize\lib\dialects\sqlite\connection-manager.js:25:13)

so it looks like the pkg'd exe is looking in the right place (local to the exe) but sqlite3 is complaining for some reason.

[igorklopov]

Do you distribute sqlite3.node along with final executable? You should.

[mawelsh]

Thanks a bunch, I didn't totally understand what that note at the bottom of the readme was getting at. Adding a gulp task to go hunt down native add ons and copy them out to my release folder. Thanks for the help!

[igorklopov]

Well, then i kindly ask you to rephrase that note. Please tell what idea the note lacks? What would you add for other users who face the same problem?

[mawelsh]

Just user error on my part. I'm mostly an embedded developer that's been tossed into node recently. It's probably mostly on the sqlite3 maintainers not calling it out as a native module. I understood that sqlite was building its own sqlite-lib (to some extent their docs make it sound like if your system doesn't have sqlite3 installed/in the path it builds it but I'm not totally sure that's the case either) but didn't realize that was actually tied into node through a native module. New users to node may miss that basically any package that is calling itself a binding is probably a "native module" and will result in a .node file being built/pulled through gyp that needs to be included.

[grempe]

Perhaps an example of what one would actually do (maybe use sqlite as the example) to resolve it to go along with the text in the readme. Its not clear to me how I would resolve that.

[PerspectivesLab]

@grempe Did you ever distribute the resulting binary anywhere (move it to another machine or anything)? I'm having a terrible time with the sqlite package and portability. Even across identical machines running the same base images where I'm building on one and running on the other I'm getting the same:

throw new Error('Please install sqlite3 package manually');

behavior I was seeing before 4.1.1. I'm just curious if you've seen this behavior at all and if you've done anything to resolve it.

portability :
this is because the sqlite plugin is compiled with visual studio, and depending the vc runtime compiled with you need the users to install the compatible runtime... else executing the app will fail,
one of the options there is to create a msi installer, that installs your pkg file and the runtimes...

[mxvsh]

Do you distribute sqlite3.node along with final executable? You should.

how do you do it 😢