Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker based test dbs #3157

Merged
merged 16 commits into from May 13, 2019
Merged

Docker based test dbs #3157

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
bc06f25
Fixed some of the tests failing with various dialect combinations
elhigu Apr 22, 2019
37f6b60
Setup all test databases to run in docker
elhigu Apr 22, 2019
4181790
Fixed test suite and updated travis to use docker
elhigu Apr 22, 2019
ebdcd09
Fixed some tests in master and disabled oracle tests while setting ev…
elhigu May 5, 2019
1080d68
Changed docker to use latest postgresql alpine image
elhigu May 6, 2019
eb0a025
Re-enabled all oracledb tests
elhigu May 6, 2019
255d273
Install oracle client libs to oracle server and copy them to travis t…
elhigu May 7, 2019
3c0f97c
Disabled random failing oracle tests and added test to show that orac…
elhigu May 10, 2019
d2fc5c2
Update package.json
kibertoad May 12, 2019
6a7862c
Update package.json
kibertoad May 12, 2019
1f7a141
Fixed linting issues
elhigu May 12, 2019
ae2149b
Fixed waiting for postgres to use the same DB image that database is …
elhigu May 12, 2019
043ae20
Removed installing oracledb driver in travis if not needed .travis.yml
elhigu May 12, 2019
1bf149d
Cleaned up package.json more
elhigu May 12, 2019
754af01
Created static name for oracledb container to allow copying files out…
elhigu May 12, 2019
fa57e47
Removed webpack from deps
elhigu May 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
68 changes: 23 additions & 45 deletions .travis.yml
View file
Open in desktop
Expand Up @@ -5,64 +5,42 @@ sudo: required

cache:
directories:
- "node_modules"
- 'node_modules'

matrix:
fast_finish: true
include:
- node_js: "12"
env: DB="mssql mysql mysql2 postgres sqlite3" CXX=g++-4.8 KNEX_TEST_TIMEOUT=60000
- node_js: "10"
env: DB="mssql mysql mysql2 postgres sqlite3" CXX=g++-4.8 KNEX_TEST_TIMEOUT=60000
- node_js: "8"
env: DB="mssql mysql mysql2 postgres sqlite3" CXX=g++-4.8 KNEX_TEST_TIMEOUT=60000
- node_js: "6"
env: DB="mssql mysql mysql2 postgres sqlite3" CXX=g++-4.8 KNEX_TEST_TIMEOUT=60000
- node_js: "10"
env: TEST_ORACLEDB=true NODE_VER=11 KNEX_TEST_TIMEOUT=60000
- node_js: "10" # node_js env is just used to have npm command availabe, all test code is ran in docker container
env: TEST_ORACLEDB=true NODE_VER=10 KNEX_TEST_TIMEOUT=60000
- node_js: "10"
env: TEST_ORACLEDB=true NODE_VER=8 KNEX_TEST_TIMEOUT=60000
- node_js: "10"
env: TEST_ORACLEDB=true NODE_VER=6 KNEX_TEST_TIMEOUT=60000
allow_failures:
- node_js: "10"
env: TEST_ORACLEDB=true NODE_VER=11 KNEX_TEST_TIMEOUT=60000
- node_js: "10"
env: TEST_ORACLEDB=true NODE_VER=10 KNEX_TEST_TIMEOUT=60000
- node_js: "10"
env: TEST_ORACLEDB=true NODE_VER=8 KNEX_TEST_TIMEOUT=60000
- node_js: "10"
env: TEST_ORACLEDB=true NODE_VER=6 KNEX_TEST_TIMEOUT=60000

- node_js: '12'
env: DB="mssql mysql mysql2 postgres sqlite3" KNEX_TEST_TIMEOUT=60000
- node_js: '10'
env: DB="oracledb mssql mysql mysql2 postgres sqlite3" KNEX_TEST_TIMEOUT=60000
- node_js: '8'
env: DB="oracledb mssql mysql mysql2 postgres sqlite3" KNEX_TEST_TIMEOUT=60000
- node_js: '6'
env: DB="oracledb mssql mysql mysql2 postgres sqlite3" KNEX_TEST_TIMEOUT=60000
install:
- if [ -z $TEST_ORACLEDB ]; then npm i; fi
- npm i
- npm run build

before_script:
- if [ -z $TEST_ORACLEDB ]; then psql -c 'create database knex_test;' -U postgres; fi
- if [ -z $TEST_ORACLEDB ]; then mysql -e 'create database knex_test;'; fi
- if [ -z $TEST_ORACLEDB ]; then npm run mssql:init; fi
- if [ -z $TEST_ORACLEDB ]; then docker ps -a; fi
- if [ -z $TEST_ORACLEDB ]; then netstat -tulpn; fi
- npm run db:start
- echo "Install oracle client libs to db container and copy to hosts ~/lib for node oracledb driver..."
- docker-compose -f scripts/docker-compose.yml exec oracledbxe curl http://yum.oracle.com/public-yum-ol7.repo -o /etc/yum.repos.d/public-yum-ol7.repo
- docker-compose -f scripts/docker-compose.yml exec oracledbxe yum install -y yum-utils
- docker-compose -f scripts/docker-compose.yml exec oracledbxe yum-config-manager --enable ol7_oracle_instantclient
- docker-compose -f scripts/docker-compose.yml exec oracledbxe yum install -y oracle-instantclient18.3-basiclite
- docker cp scripts_oracledbxe_1:/usr/lib/oracle/18.3/client64/lib/ ~/
- sudo sh -c "echo $HOME/lib > /etc/ld.so.conf.d/oracle-instantclient.conf"
- sudo ldconfig

script:
- if [ -z $TEST_ORACLEDB ]; then npm test; else npm run oracledb:test; fi
- npm test

after_success:
- if [ -z $TEST_ORACLEDB ]; then npm run coveralls; fi
- npm run coveralls

notifications:
email: false

services:
- mysql
addons:
postgresql: '9.6'
apt:
packages:
- libmysqlclient-dev
- libmysqlclient20
- mysql-common
- mysql-server
- mysql-client
- docker
73 changes: 28 additions & 45 deletions CONTRIBUTING.md
View file
Open in desktop
Expand Up @@ -97,16 +97,37 @@ main();
Usually issues without reproduction code available are just closed and if the same issue is reported multiple
times maybe someone looks into it.

One easy way to setup database for your reproduction is to use database from knex's docker-compose setup (npm run db:start) and by checking the connection settings from tests' `test/knexfile.js`.

## Integration Tests

### The Easy Way

By default, Knex runs tests against MySQL (using [mysql](https://github.com/felixge/node-mysql) and [mysql2](https://github.com/sidorares/node-mysql2)), Postgres, and SQLite. The easiest way to run the tests is by creating the database `'knex_test'` and granting permissions to the database's default username:
By default, Knex runs tests against sqlite3, postgresql, mysql, mysql2, mssql and oracledb drivers. All databases can be initialized and ran with docker.

Docker databases can be started and initialized and started with:

```bash
npm run db:start
```

and stopped with:

```bash
npm run db:stop
```

### Installing support for oracledb

Oracle has started providing precompiled driver libs for all the platforms, which makes it viable to run oracle tests also locally against oracledb running in docker.

- **MySQL**: _root_
- **Postgres**: _postgres_
Check message when running

No setup is required for SQLite.
```bash
npm install oracledb
```

and download driver library binary packages and unzip it to ~/lib directory.

### Specifying Databases

Expand All @@ -133,7 +154,9 @@ $ KNEX_TEST='./path/to/my/config.js' npm test

### Creating Postgres User

Depending on your setup you might not have the default postgres user. To create a new user, login to Postgres and use the following queries to add the user. This assumes you've already created the `knex_test` database.
If you are running tests agains own local database one might need to setup test user and databse for knex to connect.

To create a new user, login to Postgres and use the following queries to add the user. This assumes you've already created the `knex_test` database.

```
CREATE ROLE postgres WITH LOGIN PASSWORD '';
Expand All @@ -146,46 +169,6 @@ Once this is done, check it works by attempting to login:
psql -h localhost -U postgres -d knex_test
```

### Running OracleDB tests in docker

Since node-oracledb driver is so hard to install on every platform, oracle tests
are actually ran inside docker container. Container has Oracle XE g11,
node 8 and node-oracledb driver installed and copies local knex directory in
to be able to run the tests.

```
NODE_VER=10 npm run oracledb:test
```

You can also manually start shell in the docker image and run build commands manually:

```
docker run -i -t knex-test-oracledb /bin/bash

root@34f1f1cd20cf:/#

/usr/sbin/startup.sh
cd knex
npm install
npm install oracledb
npm test
```

### Runnin MSSQL SQL Server tests

SQL Server needs to be started as docker container before running tests

```
# start mssql docker container
npm run mssql:init

# run tests, do changes etc.
npm run mssql:test

# stop mssql container
npm run mssql:destroy
```

## Want to be Collaborator?

There is always room for more collaborators. Be active on resolving github issues / sending pull requests / reviewing code and we will ask you to join.
Expand Down
85 changes: 45 additions & 40 deletions README.md
View file
Open in desktop
Expand Up @@ -28,6 +28,7 @@ If upgrading from older version, see [Upgrading instructions](https://github.com
For support and questions, join the `#bookshelf` channel on freenode IRC

For an Object Relational Mapper, see:

- http://bookshelfjs.org
- https://github.com/Vincit/objection.js

Expand All @@ -41,47 +42,51 @@ We have several examples [on the website](http://knexjs.org). Here is the first
const knex = require('knex')({
dialect: 'sqlite3',
connection: {
filename: './data.db'
}
filename: './data.db',
},
});

// Create a table
knex.schema.createTable('users', function(table) {
table.increments('id');
table.string('user_name');
})

// ...and another
.createTable('accounts', function(table) {
table.increments('id');
table.string('account_name');
table.integer('user_id').unsigned().references('users.id');
})

// Then query the table...
.then(function() {
return knex.insert({user_name: 'Tim'}).into('users');
})

// ...and using the insert id, insert into the other table.
.then(function(rows) {
return knex.table('accounts').insert({account_name: 'knex', user_id: rows[0]});
})

// Query both of the rows.
.then(function() {
return knex('users')
.join('accounts', 'users.id', 'accounts.user_id')
.select('users.user_name as user', 'accounts.account_name as account');
})

// .map over the results
.map(function(row) {
console.log(row);
})

// Finally, add a .catch handler for the promise chain
.catch(function(e) {
console.error(e);
});
knex.schema
.createTable('users', function(table) {
table.increments('id');
table.string('user_name');
})

// ...and another
.createTable('accounts', function(table) {
table.increments('id');
table.string('account_name');
table
.integer('user_id')
.unsigned()
.references('users.id');
})

// Then query the table...
.then(function() {
return knex('users').insert({ user_name: 'Tim' });
})

// ...and using the insert id, insert into the other table.
.then(function(rows) {
return knex('accounts').insert({ account_name: 'knex', user_id: rows[0] });
})

// Query both of the rows.
.then(function() {
return knex('users')
.join('accounts', 'users.id', 'accounts.user_id')
.select('users.user_name as user', 'accounts.account_name as account');
})

// .map over the results
.map(function(row) {
console.log(row);
})

// Finally, add a .catch handler for the promise chain
.catch(function(e) {
console.error(e);
});
```
25 changes: 18 additions & 7 deletions package.json
View file
Open in desktop
Expand Up @@ -20,11 +20,13 @@
"liftoff": "3.1.0",
"lodash": "^4.17.11",
"mkdirp": "^0.5.1",
"oracledb": "^3.1.2",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be prodDependency?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, we don't have any of the drivers as knex deps to be able to install only one that is used in the project.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be moved then

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oracledb is still in prod dependencies :D

"pg-connection-string": "2.0.0",
"tarn": "^1.1.5",
"tildify": "1.2.0",
"uuid": "^3.3.2",
"v8flags": "^3.1.3"
"v8flags": "^3.1.3",
"webpack": "^4.30.0"
elhigu marked this conversation as resolved.
Show resolved Hide resolved
},
"lint-staged": {
"*.{js,json}": [
Expand Down Expand Up @@ -71,7 +73,8 @@
"tape": "^4.10.1",
"typescript": "^3.4.5",
"through": "^2.3.8",
"toxiproxy-node-client": "^2.0.6"
"toxiproxy-node-client": "^2.0.6",
"webpack-cli": "^3.3.1"
},
"buildDependencies": [
"@babel/cli",
Expand All @@ -80,21 +83,22 @@
"rimraf"
],
"scripts": {
"babel": "rimraf ./lib && babel src --out-dir lib --copy-files",
"format": "prettier --write \"{src,bin,scripts,test}/**/*.js\"",
"build": "npm run babel",
"debug:test": "node --inspect-brk ./node_modules/.bin/_mocha -- --exit -t 0 test/index.js",
"debug_test": "mocha --inspect-brk --exit -t 0 test/index.js",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate, there's already "debug:test"

"babel": "rimraf ./lib && babel src --out-dir lib --copy-files",
"debug:test": "mocha --inspect-brk --exit -t 0 test/index.js",
"debug:tape": "node --inspect-brk test/tape/index.js",
"format": "prettier --write \"{src,bin,scripts,test}/**/*.js\"",
"coveralls": "nyc report --reporter=text-lcov | coveralls",
"dev": "rimraf ./lib && babel -w src --out-dir lib --copy-files",
"lint": "eslint '{src,test}/**/*.js'",
"lint:types": "dtslint types",
"plaintest": "mocha --exit -t 10000 -b -R spec test/index.js && npm run tape",
"plaintest": "mocha --exit -t 10000 test/index.js && npm run tape && npm run test:cli",
"plaintest:sqlite": "cross-env DB=sqlite3 npm run plaintest",
"prepare": "npm run babel",
"prepublishOnly": "npm run babel",
"pretest": "npm run lint && npm run lint:types && npm run babel",
"test": "nyc mocha --exit --check-leaks --globals __core-js_shared__ -t 10000 -R spec test/index.js && npm run test:tape && npm run test:cli",
"test": "nyc mocha --exit --check-leaks --globals __core-js_shared__ -t 10000 test/index.js && npm run tape && npm run bin_test",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be "npm run test:tape && npm run test:cli""

"test:tape": "node test/tape/index.js | tap-spec",
"test:cli": "cross-env KNEX_PATH=../knex.js KNEX=bin/cli.js jake -f test/jake/Jakefile",
"oracledb:test": "docker rmi -f --no-prune knex-test-oracledb && docker build -f scripts/oracle-tests-Dockerfile --tag knex-test-oracledb . && docker run --rm -i -t -e KNEX_TEST_TIMEOUT=$KNEX_TEST_TIMEOUT -e NODE_VER=$NODE_VER knex-test-oracledb",
Expand All @@ -103,6 +107,13 @@
"mssql:test": "DB=mssql npm run plaintest",
"mssql:destroy": "docker-compose -f scripts/mssql-docker-compose.yml stop",
"postmssql:init": "node scripts/wait-for-mssql-server.js && npm run mssql:logs || (npm run mssql:logs;false)",
"prepublish": "npm run babel",
"pre_test": "npm run lint && npm run lint_types",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate, there's pretest

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is pretest run automatically before publish?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aa you meant pre_test 👍

"bin_test": "cross-env KNEX_PATH=../knex.js KNEX=bin/cli.js jake -f test/jake/Jakefile",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate with test:cli

"tape": "node test/tape/index.js | tap-spec",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate with test:tape

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about that "prepublishOnly", when that is ran?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

npm is throwing deprecation warning about prepublish now. Now there are two separate hooks: prepublishOnly for publish and prepare for after install. prepublish should actually be removed, next major npm version won't support it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I'll remove prepublish already then. Its really unlikely that we would publish anything without running babel first, since its pretty much included in every hook there :)

"debug_tape": "node --inspect-brk test/tape/index.js",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate, there's already "debug:tape"

"db:start": "docker-compose -f scripts/docker-compose.yml up -d mysql oracledbxe postgres mssql; docker-compose -f scripts/docker-compose.yml up initmssqlknexdb waitmysql waitpostgres waitoracledbxe",
"db:stop": "docker-compose -f scripts/docker-compose.yml down",
"stress:init": "docker-compose -f scripts/stress-test/docker-compose.yml up --no-start && docker-compose -f scripts/stress-test/docker-compose.yml start",
"stress:test": "node scripts/stress-test/knex-stress-test.js | grep -A 5 -B 60 -- '- STATS '",
"stress:destroy": "docker-compose -f scripts/stress-test/docker-compose.yml stop"
Expand Down