Docker based test dbs #3157

elhigu · 2019-04-22T14:34:34Z

Probably will still fail, because travis might not have oracle client libs installed yet.

Also found out that:

current test suite is not closing down all started databases (so mocha must be ran with --exit switch)
acquireConnectionTimeout doesn't work for some reason with oracledb

scripts/docker-compose.yml

test/unit/dialects/oracledb.js

.travis.yml

test/unit/knex.js

elhigu · 2019-05-09T18:07:16Z

scripts/docker-compose.yml

+      - bash
+      - -c
+      - 'until /opt/oracle/product/18c/dbhomeXE/bin/sqlplus -s sys/Oracle18@oracledbxe/XE as sysdba <<< "SELECT 13376411 FROM DUAL; exit;" | grep "13376411"; do echo "Could not connect to oracle... sleep for a while"; sleep 5; done'
+#  testrunner:


I was able to run oracle tests with this locally by copying needed client libs from oracledb container, so if nothing else works I'll add one more container, which actually runs the tests instead of travis trusty.

elhigu · 2019-05-09T18:36:16Z

Only potential problem still here is that mysql server startup hangs some times... I'll try to rerun these to check out if that is still a problem.

elhigu · 2019-05-09T21:42:15Z

Yep, still various tests are unstable with oracledb (so far I've catched 3 when running tests in loop)... I'll skip them on oracle and add an issue about them. Still seems like the problem is that the way how knex does transactions with oracle is not completely stable...

Or we it might be less strict about when changes in DB is available for read for other connections when updating row in another... I really would like to write more multiconnection / concurrent reads / writes tests to be able to catch and reproduce those problems.

elhigu · 2019-05-10T09:21:16Z

test/integration/builder/updates.js

@@ -70,6 +70,80 @@ module.exports = function(knex) {
        });
    });

+    it('should immediately return updated value for other connections when updating row to DB returns', function() {


This is the new test case... if we can setup knex oracledb dialect to work in a way that this tests passes, then probably most of the other failing oracle tests will start to work

elhigu · 2019-05-10T09:36:28Z

And still gotta fix those linting issues on sunday.

kibertoad · 2019-05-10T23:20:09Z

scripts/docker-compose.yml

+      - POSTGRES_PASSWORD=knextest
+      - POSTGRES_DB=knex_test
+  waitpostgres:
+    image: postgres:10-alpine


why do we need two different postgres images?

kibertoad · 2019-05-11T22:07:08Z

@elhigu Merged changes from master that enabled running tests on Node 12. However, Oracle driver still doesn't support Node 12. Can we not execute Oracle tests on it?

kibertoad · 2019-05-11T22:25:54Z

package.json

@@ -20,11 +20,13 @@
    "liftoff": "3.1.0",
    "lodash": "^4.17.11",
    "mkdirp": "^0.5.1",
+    "oracledb": "^3.1.2",


should it be prodDependency?

nope, we don't have any of the drivers as knex deps to be able to install only one that is used in the project.

should be moved then

oracledb is still in prod dependencies :D

elhigu · 2019-05-12T17:59:22Z

@elhigu Merged changes from master that enabled running tests on Node 12. However, Oracle driver still doesn't support Node 12. Can we not execute Oracle tests on it?

No, oracle tests cannot be ran with node 12 yet. Also looks like that something else came with you merge here, since there is new failing test (TypeError: qb(...).table(...).having(...).clearHaving is not a function)...

elhigu · 2019-05-12T18:04:21Z

@elhigu Pull changes from master, I would expect that to fix clearHaving problem

Yep... I'm rebasing back to top of master... now there are just bunch of conflict to fix 😬

…erything else to work again

…o make node oracledb package to work

…le update in transaction does not seem to work ACID

kibertoad · 2019-05-12T18:09:19Z

@elhigu I can resolve conflicts if you want.

…using.

elhigu · 2019-05-12T18:14:44Z

Looks like all of them was for package.json + travis (but had to fix them on every commit). You could check afterwards that they look fine after I'll update this branch (I might have lost some of the changes / package version there).

kibertoad · 2019-05-12T18:18:32Z

package.json

    "build": "npm run babel",
-    "debug:test": "node --inspect-brk ./node_modules/.bin/_mocha -- --exit -t 0 test/index.js",
+    "debug_test": "mocha --inspect-brk --exit -t 0 test/index.js",


duplicate, there's already "debug:test"

kibertoad · 2019-05-12T18:19:03Z

package.json

    "plaintest:sqlite": "cross-env DB=sqlite3 npm run plaintest",
    "prepare": "npm run babel",
    "prepublishOnly": "npm run babel",
    "pretest": "npm run lint && npm run lint:types && npm run babel",
-    "test": "nyc mocha --exit --check-leaks --globals __core-js_shared__ -t 10000 -R spec test/index.js && npm run test:tape && npm run test:cli",
+    "test": "nyc mocha --exit --check-leaks --globals __core-js_shared__ -t 10000 test/index.js && npm run tape && npm run bin_test",


Should be "npm run test:tape && npm run test:cli""

kibertoad · 2019-05-12T18:19:21Z

package.json

+    "pre_test": "npm run lint && npm run lint_types",
+    "bin_test": "cross-env KNEX_PATH=../knex.js KNEX=bin/cli.js jake -f test/jake/Jakefile",
+    "tape": "node test/tape/index.js | tap-spec",
+    "debug_tape": "node --inspect-brk test/tape/index.js",


duplicate, there's already "debug:tape"

kibertoad · 2019-05-12T18:19:33Z

package.json

@@ -103,6 +107,13 @@
    "mssql:test": "DB=mssql npm run plaintest",
    "mssql:destroy": "docker-compose -f scripts/mssql-docker-compose.yml stop",
    "postmssql:init": "node scripts/wait-for-mssql-server.js && npm run mssql:logs || (npm run mssql:logs;false)",
+    "prepublish": "npm run babel",
+    "pre_test": "npm run lint && npm run lint_types",


duplicate, there's pretest

is pretest run automatically before publish?

aa you meant pre_test 👍

kibertoad · 2019-05-12T18:19:43Z

package.json

@@ -103,6 +107,13 @@
    "mssql:test": "DB=mssql npm run plaintest",
    "mssql:destroy": "docker-compose -f scripts/mssql-docker-compose.yml stop",
    "postmssql:init": "node scripts/wait-for-mssql-server.js && npm run mssql:logs || (npm run mssql:logs;false)",
+    "prepublish": "npm run babel",
+    "pre_test": "npm run lint && npm run lint_types",
+    "bin_test": "cross-env KNEX_PATH=../knex.js KNEX=bin/cli.js jake -f test/jake/Jakefile",


duplicate with test:cli

kibertoad · 2019-05-12T18:19:51Z

package.json

+    "prepublish": "npm run babel",
+    "pre_test": "npm run lint && npm run lint_types",
+    "bin_test": "cross-env KNEX_PATH=../knex.js KNEX=bin/cli.js jake -f test/jake/Jakefile",
+    "tape": "node test/tape/index.js | tap-spec",


duplicate with test:tape

how about that "prepublishOnly", when that is ran?

npm is throwing deprecation warning about prepublish now. Now there are two separate hooks: prepublishOnly for publish and prepare for after install. prepublish should actually be removed, next major npm version won't support it.

thanks, I'll remove prepublish already then. Its really unlikely that we would publish anything without running babel first, since its pretty much included in every hook there :)

elhigu · 2019-05-12T20:15:25Z

Again something really strange happens in travis that doesn't happen locally... created another simpler project to explore that https://github.com/elhigu/travis-node-oracle-18-xe

Basically locally installed libs are found in /usr/lib/oracle... inside docker container, but in travis that directory doesn't seem to be created, even that both uses the same docker image where to install them.

… of it on different docker versions

kibertoad · 2019-05-12T23:01:26Z

Finally green! Should we merge this?

package.json

elhigu · 2019-05-13T09:23:57Z

huh finally its over.... I saw one random hanging after tests, which might mean that some connection is left open in some cases. or pool cannot tear down properly. If that happens we could add flag that forces mocha to exit even if there are connections left open and see if stuff works better when tests are written in a more clean manner with async / await .

kibertoad reviewed Apr 22, 2019

View reviewed changes

scripts/docker-compose.yml Outdated Show resolved Hide resolved

elhigu force-pushed the docker-based-test-dbs branch from 44acc89 to 2906709 Compare April 22, 2019 14:59

kibertoad approved these changes Apr 22, 2019

View reviewed changes

test/unit/dialects/oracledb.js Outdated Show resolved Hide resolved

elhigu force-pushed the docker-based-test-dbs branch 2 times, most recently from 8e2deb5 to 4155e4c Compare April 22, 2019 15:27

kibertoad reviewed Apr 28, 2019

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

kibertoad reviewed Apr 28, 2019

View reviewed changes

test/unit/knex.js Show resolved Hide resolved

elhigu force-pushed the docker-based-test-dbs branch 9 times, most recently from 3f8ed41 to 2d3502d Compare May 9, 2019 18:04

elhigu commented May 9, 2019

View reviewed changes

elhigu force-pushed the docker-based-test-dbs branch from 2d3502d to fcb48cf Compare May 9, 2019 18:09

elhigu force-pushed the docker-based-test-dbs branch 4 times, most recently from d74ebdc to fa31550 Compare May 9, 2019 19:40

elhigu commented May 10, 2019

View reviewed changes

kibertoad reviewed May 10, 2019

View reviewed changes

kibertoad reviewed May 11, 2019

View reviewed changes

elhigu added 6 commits May 12, 2019 21:07

Fixed test suite and updated travis to use docker

4181790

Fixed some tests in master and disabled oracle tests while setting ev…

ebdcd09

…erything else to work again

Changed docker to use latest postgresql alpine image

1080d68

Re-enabled all oracledb tests

eb0a025

Install oracle client libs to oracle server and copy them to travis t…

255d273

…o make node oracledb package to work

Disabled random failing oracle tests and added test to show that orac…

3c0f97c

…le update in transaction does not seem to work ACID

kibertoad and others added 4 commits May 12, 2019 21:11

Update package.json

d2fc5c2

Update package.json

6a7862c

Fixed linting issues

1f7a141

Fixed waiting for postgres to use the same DB image that database is …

ae2149b

…using.

elhigu force-pushed the docker-based-test-dbs branch from 6e20bc2 to ae2149b Compare May 12, 2019 18:16

kibertoad reviewed May 12, 2019

View reviewed changes

elhigu added 2 commits May 12, 2019 21:41

Removed installing oracledb driver in travis if not needed .travis.yml

043ae20

Cleaned up package.json more

1bf149d

elhigu force-pushed the docker-based-test-dbs branch from 047a990 to 1bf149d Compare May 12, 2019 19:23

Created static name for oracledb container to allow copying files out…

754af01

… of it on different docker versions

kibertoad reviewed May 12, 2019

View reviewed changes

package.json Outdated Show resolved Hide resolved

Removed webpack from deps

fa57e47

elhigu merged commit 8a9a648 into knex:master May 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker based test dbs #3157

Docker based test dbs #3157

elhigu commented Apr 22, 2019

elhigu May 9, 2019

elhigu commented May 9, 2019

elhigu commented May 9, 2019 •

edited

elhigu May 10, 2019

elhigu commented May 10, 2019

kibertoad May 10, 2019

elhigu May 12, 2019

kibertoad commented May 11, 2019

kibertoad May 11, 2019

elhigu May 12, 2019

kibertoad May 12, 2019

kibertoad May 12, 2019

elhigu commented May 12, 2019

elhigu commented May 12, 2019

kibertoad commented May 12, 2019

elhigu commented May 12, 2019

kibertoad May 12, 2019

kibertoad May 12, 2019

kibertoad May 12, 2019

kibertoad May 12, 2019

elhigu May 12, 2019

elhigu May 12, 2019

kibertoad May 12, 2019

kibertoad May 12, 2019

elhigu May 12, 2019

kibertoad May 12, 2019

elhigu May 12, 2019

elhigu commented May 12, 2019

kibertoad commented May 12, 2019

elhigu commented May 13, 2019

Docker based test dbs #3157

Docker based test dbs #3157

Conversation

elhigu commented Apr 22, 2019

Choose a reason for hiding this comment

elhigu commented May 9, 2019

elhigu commented May 9, 2019 • edited

Choose a reason for hiding this comment

elhigu commented May 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibertoad commented May 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elhigu commented May 12, 2019

elhigu commented May 12, 2019

kibertoad commented May 12, 2019

elhigu commented May 12, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elhigu commented May 12, 2019

kibertoad commented May 12, 2019

elhigu commented May 13, 2019

elhigu commented May 9, 2019 •

edited