Enable testing with MySQL and Postgres on Travis-CI #171

atodorov · 2017-03-10T14:48:14Z

This PR enables MySQL and Postgres on Travis. I have a support ticket opened with their team to figure out how to add MariaDB alongside MySQL.

The build will fail b/c there are some issue with both MySQL and Postgres. I will try to fix them in subsequent commits.

coveralls · 2017-03-10T14:54:03Z

Coverage increased (+0.01%) to 45.189% when pulling c6c2794 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T19:30:58Z

Coverage increased (+0.01%) to 45.189% when pulling f486a3d on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T19:36:59Z

Coverage increased (+0.01%) to 45.189% when pulling 297333c on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T19:41:45Z

Coverage increased (+0.01%) to 45.189% when pulling b9e8ab6 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T19:42:39Z

Coverage increased (+0.01%) to 45.189% when pulling b9e8ab6 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T21:15:55Z

Coverage increased (+0.01%) to 45.189% when pulling cefa25a on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T21:34:00Z

Coverage increased (+0.01%) to 45.189% when pulling 6f9fdcd on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T21:46:03Z

Coverage increased (+0.01%) to 45.189% when pulling e34804a on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T22:20:26Z

Coverage increased (+0.01%) to 45.189% when pulling 1c2d2fd on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T22:46:05Z

Coverage increased (+0.03%) to 45.209% when pulling bad6c4c on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T22:46:05Z

Coverage increased (+0.03%) to 45.209% when pulling bad6c4c on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T22:54:56Z

Coverage increased (+0.03%) to 45.209% when pulling 482f977 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T22:54:56Z

Coverage increased (+0.03%) to 45.209% when pulling 482f977 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T22:58:59Z

Coverage increased (+0.03%) to 45.209% when pulling 482f977 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T23:13:08Z

Coverage increased (+0.03%) to 45.209% when pulling 443c5bb on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-12T23:13:08Z

Coverage increased (+0.03%) to 45.209% when pulling 443c5bb on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-13T08:37:08Z

Coverage increased (+0.03%) to 45.209% when pulling a5920d7 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-13T08:55:04Z

Coverage increased (+0.03%) to 45.209% when pulling 414963e on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-13T09:01:26Z

Coverage increased (+0.03%) to 45.209% when pulling 288c9fc on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-13T09:04:38Z

Coverage increased (+0.03%) to 45.209% when pulling 288c9fc on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

coveralls · 2017-03-13T09:15:02Z

Coverage increased (+0.03%) to 45.209% when pulling 11b4fb9 on MrSenko:travisci_more_db_testing into ae3745a on Nitrate:develop.

atodorov · 2017-04-11T20:39:49Z

@tkdchen I think this is ready to merge so we can continue forward. I hate to merge with failing Postgres but MySQL, MariaDB and sqlite all work fine and some of the fixes are covered into other PRs. I don't think we have a lot of other options here otherwise this PR will become very big and there are other problems to worry about too.

tkdchen · 2017-04-12T15:50:27Z

This is a big step forward to the testing. However, the problem is there is dependency of Travis in Nitrate source code, and this testing matrix cannot run locally by developer. I prefer to a solution to avoid these two major disadvantages.

Basically, the testing matrix should

allow developer to run tests locally without any other external services required.
allow developer to run tests by selecting specific database, SQLite, MySQL, MariaDB or PostgreSQL, or even all of them at once whatever in parallel or in sequence.
by default, tests could be run with SQLite, which is a much fast way to run tests and verify something quickly.

The second ability should be able to be used in Travis-CI to run the testing matrix.

In my mind, docker would be a good tool to help us to build this testing matrix, and obviously some scripts would be required.

Currently, to run whole test suite, I can simple run make test in a well-provisioned vagrant machine. However, running test suite from scratch should be an easy thing now in Nitrate as following steps,

provision machine, installing necessary packages required in next steps
make a virtual environment
install dependencies from requirements/devel.txt
make test

It should be easy to make these steps happen in a docker container. Fedora 25 docker image can be used and it is ok to use the version of MySQL, MariaDB and PostgreSQL in Fedora repo.

Finally, there could be these ways to run tests matrix

make test, run with SQLite by default
DB=[dbname] make test, dbname could be mysql, mariadb, or pgsql
DB=all make test, run tests with mysql, mariadb, and pgsql individually

Thank you for making this improvement. It's valuable and develop our ideas. Let's make it better and better.

I'll look at other commits in this PR from tomorrow. Have a good day. :)

atodorov · 2017-04-12T21:02:10Z

make test, run with SQLite by default
DB=[dbname] make test, dbname could be mysql, mariadb, or pgsql
DB=all make test, run tests with mysql, mariadb, and pgsql individually

I think this can easily be done in the settings and then simply ask Travis to execute whichever command fits its matrix. In this way we move the infrastructure dependency in .travis.yml and not in the project source code.

I will rework the settings for that. It is up to the developer to prepare their local environment for whichever DB they want to test on. I will also update the docs with commands how to install dependencies and run the tests against different databases.

Note: vagrant or docker will be a great help but that's up to each developer to decide how to use. IMO this needs to come at a later stage because this PR is quite big already. What do you think?

coveralls · 2017-04-13T13:43:04Z

Coverage increased (+0.05%) to 45.425% when pulling bd74f39 on MrSenko:travisci_more_db_testing into 8f45beb on Nitrate:develop.

atodorov · 2017-04-13T13:43:59Z

@tkdchen see the updated documentation and the last 3 commits for the latest changes. It is not possible to execute the tests on the local machine by specifying which database to use. The same commands are integrated into Travis as well. All of the integration happens in .travis.yml.

tkdchen · 2017-04-14T13:23:46Z

Using Travis to run tests does not mean there is a dependency of Travis to run tests. Simply say I don't want this line of code appears in source code

if 'TRAVIS' in os.environ:

Documentation is good but not always. For running tests, I prefer to make it as easy as possible with just one command, no need of steps

find the documentation
read through the documentation to learn the steps
try each step to provision environment and launch tests
repeat again and again if something is wrong by misoperation

Instead, as mentioned in my previous comment, just make test or DB=pgsql make test is enough. Easy and save much time, isn't it?

If someone is interested in the steps to build testing environment, just open and read the relative scripts. That should be more straightforward than lots of words in documentation and easy understand.

coveralls · 2017-04-14T13:31:22Z

Coverage increased (+0.03%) to 45.411% when pulling 61355c2 on MrSenko:travisci_more_db_testing into 8f45beb on Nitrate:develop.

atodorov · 2017-04-14T13:34:33Z

if 'TRAVIS' in os.environ:

Mistake, this has now been updated. The rest (make test commands) are as requested and described in more details in the docs. Anything else missing ?

coveralls · 2017-04-14T13:42:33Z

Coverage increased (+0.03%) to 45.406% when pulling 8b42d6f on MrSenko:travisci_more_db_testing into 8f45beb on Nitrate:develop.

tkdchen · 2017-04-15T04:07:50Z

@atodorov We talked a lot in IRC channel #nitrate. You did a good job to make it possible to run tests on different database engines. Hopefully, you could continue improving based on the changes in this PR to make running tests more easily.

I'd like to repeat my thought again

Do not depend on Travis-CI to run tests.
To achieve the first goal, if those database provision steps in .travis.yml are necessary, they should happen locally, not in Travis-CI.
As a developer, I can run tests simply in following steps without the need of reading whole documentation before that.
1. clone Nitrate
2. DB=mysql make test

Detailed documentation in this PR is good, I think it could be as reference for developers who are interested in hacking the tests for themselves.

Is this clear and make sense?

tkdchen · 2017-04-15T04:41:40Z

I merged these 3 commits

a1c47ec Compare PKs with what was written in DB, not hard-coded
fc2343c Update pre_check_product() and pre_process_ids() to include long
7d10b49 Don't use unique test object names

tkdchen · 2017-04-15T04:43:43Z

tcms/testruns/tests.py

@@ -37,7 +36,7 @@ def setUpTestData(cls):
        cls.client = test.Client()

    def test_404_if_run_does_not_exist(self):
-        nonexisting_run_pk = TestRun.objects.count() + 1
+        nonexisting_run_pk = 999999


It would be helpful to understand if you can add explanation to this change. Then, I'll merge it.

OK, for some reason ( I think because PKs are global autoincrements on MySQL, generated via LAST_INSERT_ID())

TestRun.objects.count() == 1 nonexisting_run_pk == 2

but the only TestRun object in the database has an actual ID of 2 hence the failure!

On other DB engines the only TestRun object has an ID of 1. I will update this code with
TestRun.objects.last().pk + 1 which will be more robust.

However I have found one more thing. In testruns/urls/run_urls.py there are two URLs which use the same view:
.../ordercase/$ and .../ordercaserun/$. Do you know why is that? Sounds like a bug (let me know and I will send a separate PR for it).

Fix Nitrate#169 Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

- the older Travis CI images have problems with MariaDB not picking up its configuration - also manually create DB and test DB and grant privileges to travis b/c connection will fail otherwise. Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

On MySQL the default AutoField() translates to AUTO_INCREMENT which is shared between all tables! Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

testing is now possible via the following commands: make test (uses SQlite) TEST_DB=all make test (will test on all DBs) TEST_DB=MySQL make test TEST_DB=MariaDB make test TEST_DB=Postgres make test In addition when preparing the local virtualenv we now have DB specific requirements files: requirements/mysql.txt (for MySQL and MariaDB) requirements/postgres.txt (for Postgres) requirements/base.txt (for SQLite) requirements/devel.txt (additional development libraries) Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

also updates the usage of new requirements files Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

atodorov · 2017-04-15T19:45:48Z

Updated and rebased.

@tkdchen let's move this comment #171 (comment) into a separate issue. I understand the value proposition of it and it is definitely doable even in the short term. However not everyone would like to follow exactly the same steps so we shouldn't force it. Plus a separate issue will allow us to discuss into more details. What do you think?

coveralls · 2017-04-15T19:47:34Z

Coverage increased (+0.04%) to 45.406% when pulling fa98cb2 on MrSenko:travisci_more_db_testing into a1c47ec on Nitrate:develop.

coveralls · 2017-04-15T19:47:34Z

Coverage increased (+0.04%) to 45.406% when pulling fa98cb2 on MrSenko:travisci_more_db_testing into a1c47ec on Nitrate:develop.

tkdchen · 2019-02-04T07:33:30Z

This was done in 2e22b8a and subsequent commits that make it possible to run with PostgreSQL.

atodorov force-pushed the travisci_more_db_testing branch from f486a3d to 7f7b61b Compare March 12, 2017 19:30

atodorov force-pushed the travisci_more_db_testing branch from 7f7b61b to b9e8ab6 Compare March 12, 2017 19:32

atodorov force-pushed the travisci_more_db_testing branch from 297333c to b9e8ab6 Compare March 12, 2017 19:38

atodorov force-pushed the travisci_more_db_testing branch from 46121bb to cefa25a Compare March 12, 2017 21:11

atodorov force-pushed the travisci_more_db_testing branch from cefa25a to 6f9fdcd Compare March 12, 2017 21:28

atodorov force-pushed the travisci_more_db_testing branch from 6f9fdcd to e34804a Compare March 12, 2017 21:38

atodorov force-pushed the travisci_more_db_testing branch from e34804a to 1c2d2fd Compare March 12, 2017 22:16

atodorov force-pushed the travisci_more_db_testing branch from f19200d to bad6c4c Compare March 12, 2017 22:41

atodorov force-pushed the travisci_more_db_testing branch from 0a1158e to 482f977 Compare March 12, 2017 22:52

atodorov mentioned this pull request Apr 11, 2017

Don't use deprecated request.REQUEST #156

Closed

atodorov force-pushed the travisci_more_db_testing branch 2 times, most recently from 4888d77 to bd74f39 Compare April 13, 2017 13:38

atodorov force-pushed the travisci_more_db_testing branch from bd74f39 to 61355c2 Compare April 14, 2017 13:27

atodorov force-pushed the travisci_more_db_testing branch from 61355c2 to 8b42d6f Compare April 14, 2017 13:38

tkdchen reviewed Apr 15, 2017

View reviewed changes

atodorov added 6 commits April 15, 2017 21:53

Enable testing with MySQL and Postgres on Travis-CI

b3672a3

Fix Nitrate#169 Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

Fix test failures on MySQL for non-existing PKs

80cdbb0

On MySQL the default AutoField() translates to AUTO_INCREMENT which is shared between all tables! Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

Handle imports if not running on MySQL

b6e4d15

Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

Update documentation to reflect new test commands

fa98cb2

also updates the usage of new requirements files Signed-off-by: Mr. Senko <atodorov@mrsenko.com>

atodorov force-pushed the travisci_more_db_testing branch from 8b42d6f to fa98cb2 Compare April 15, 2017 19:43

atodorov mentioned this pull request Apr 26, 2017

Fix Issue #181 #182

Closed

tkdchen closed this Feb 4, 2019

Enable testing with MySQL and Postgres on Travis-CI #171

Enable testing with MySQL and Postgres on Travis-CI #171

Conversation

atodorov commented Mar 10, 2017

coveralls commented Mar 10, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017 • edited

coveralls commented Mar 12, 2017

coveralls commented Mar 12, 2017

coveralls commented Mar 13, 2017 • edited

coveralls commented Mar 13, 2017 • edited

coveralls commented Mar 13, 2017 • edited

coveralls commented Mar 13, 2017 • edited

coveralls commented Mar 13, 2017 • edited

atodorov commented Apr 11, 2017

tkdchen commented Apr 12, 2017 • edited

atodorov commented Apr 12, 2017

coveralls commented Apr 13, 2017 • edited

atodorov commented Apr 13, 2017

tkdchen commented Apr 14, 2017

coveralls commented Apr 14, 2017 • edited

atodorov commented Apr 14, 2017

coveralls commented Apr 14, 2017 • edited

tkdchen commented Apr 15, 2017

tkdchen commented Apr 15, 2017

tkdchen Apr 15, 2017

Choose a reason for hiding this comment

atodorov Apr 15, 2017

Choose a reason for hiding this comment

atodorov commented Apr 15, 2017

coveralls commented Apr 15, 2017

coveralls commented Apr 15, 2017 • edited

tkdchen commented Feb 4, 2019

coveralls commented Mar 12, 2017 •

edited

coveralls commented Mar 13, 2017 •

edited

coveralls commented Mar 13, 2017 •

edited

coveralls commented Mar 13, 2017 •

edited

coveralls commented Mar 13, 2017 •

edited

coveralls commented Mar 13, 2017 •

edited

tkdchen commented Apr 12, 2017 •

edited

coveralls commented Apr 13, 2017 •

edited

coveralls commented Apr 14, 2017 •

edited

coveralls commented Apr 14, 2017 •

edited

coveralls commented Apr 15, 2017 •

edited