Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run mysql in docker and load sql scripts in test/integration/sql/ #25

Open
wants to merge 1 commit into
base: v3
Choose a base branch
from

Conversation

dotnetCarpenter
Copy link

@dotnetCarpenter dotnetCarpenter commented May 8, 2017

This is a highly experimental PR but so far I have made good progress, running all tests in a mysql docker instance.

This requires docker to be installed: https://docs.docker.com/engine/installation/
I have not found an easy way to auto install docker cross platform.

Usage

In the terminal write: composer docker:setup, which is a shorthand for the following:

1 composer docker:network:create
2 composer docker:create
3 composer docker:start

Then start the tests via composer test as usual.

Feature

  • runs mysql in docker (does not touch you local database)
  • loads .sql scripts found in test/integration/sql/ on creation
  • remove all data creation code from PHP
  • can run alongside of a native mysql server (the docker instance listens on TCP/33555)

I'm on linux ubuntu, so am eager to know how the setup works in other OS'es, especially Windows and macOS.

Give it a shot!

@dotnetCarpenter
Copy link
Author

If you get ConnectionException: Connection to 'mysql:host=172.18.0.2' failed: SQLSTATE[HY000] [2002] Connection refused. Then wait a few seconds before running composer test again.

@coveralls
Copy link

coveralls commented May 8, 2017

Coverage Status

Coverage decreased (-24.0%) to 75.641% when pulling 70a6ec3 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@coveralls
Copy link

coveralls commented May 8, 2017

Coverage Status

Coverage decreased (-24.0%) to 75.641% when pulling 22bd197 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@dotnetCarpenter
Copy link
Author

dotnetCarpenter commented May 9, 2017

Now passing on travis. I had to use another port than 3306, and had some issues around port forwarding. But it's now resolved. It did require some small changes to FullSyncTest.php and TestAbstract.php, to parse port number from the host key in config.php. A few things were surprising here.

  1. Setting host to localhost doesn't work - it's parsed wrong
  2. setting port in config.php doesn't work - it's parsed wrong.

I have a feeling that the host/port parsing need a little love.

@coveralls
Copy link

coveralls commented May 9, 2017

Coverage Status

Coverage decreased (-24.0%) to 75.641% when pulling b908dcd on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@coveralls
Copy link

coveralls commented May 9, 2017

Coverage Status

Coverage remained the same at 99.605% when pulling f06520c on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@mrjgreen
Copy link
Owner

This looks great! Thanks for taking the time to do this.

I was wondering if a docker-compose.yml might be a nice alternative to the individual setup commands on the composer.json?

What do you think?

@dotnetCarpenter
Copy link
Author

@mrjgreen As far as I understand, then docker-compose.yml require users to install another docker tool, namely Docker Compose.

As with Docker itself, there is no easy cross platform install. So you have to throw an error, instructing people to go to docker.com and install tools. It's already bad that there is no auto-install of Docker. Adding more required tools, to run composer test, is not helpful.

When trying out docker configurations, you really want to be able to create/remove different layers easily. I find that the composer scripts does a very good job at that. docker-compose COMMAND basically does the same but more abstracted from configuration, since you can not see the docker commands but only have a YAML file (docker-compose.yml) and a pretty lengthy YAML file documentation.

docker-compose run network db-sync-network vs composer docker:network:create

docker-compose up vs composer docker:setup

docker-compose stop vs composer docker:stop

docker-compose down vs composer docker:clean

I think it's easier to grok these things, when you can see the implementation right there in composer.json and you will learn docker commands at the same time (even if the configuration looks a bit messy).

It makes sense to use Docker Compose though, when managing multiple images for the same app. With the little caveat that a database docker image can not serve requests before the database is initialized and Docker Compose will start each container at the same time, so a container that needs to make DB requests will need to be paused for arbitrary seconds before making the first request.

BTW, the configuration for docker:create is unique for the mysql image. If you want to use another image, you'll probably have to change everything there. E.i. the docker mysql image from the docker team is slightly different - using a different base linux image etc.

@coveralls
Copy link

coveralls commented May 10, 2017

Coverage Status

Coverage remained the same at 99.605% when pulling 2b38bb3 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@dotnetCarpenter
Copy link
Author

dotnetCarpenter commented May 10, 2017

One thing that would simplify the docker process, is if we could use the default bridge network. Creating a new network risk a collision of IP addresses on 172.18.0.X. But I can not get port forwarding (TCP://localhost:33555 to TCP://docker_image:3306) to work on the default bridge network.

Well, we still risk IP address collision but at least it would shave off two composer scripts.

@coveralls
Copy link

coveralls commented May 11, 2017

Coverage Status

Coverage remained the same at 99.605% when pulling 33a5869 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@coveralls
Copy link

coveralls commented May 11, 2017

Coverage Status

Coverage remained the same at 99.605% when pulling 1b9d706 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@dotnetCarpenter
Copy link
Author

@mrjgreen So I eventually found out why I could not connect to the docker instance through the proxy port, when using an dynamic ip address in the default bridge network.

If you want to inspect the network, I can recommend iptables -t nat -L -v and docker network inspect bridge. I used netstat -at but I could not see any docker containers listening IPv4, even though they were.

There is now no risk of IP collision, since docker will assign IP in its network. If the user change the default network range, docker will assign the mysql image to that range.

So all is good

I can't think of any thing else to amend to this PR. I think you should handle the data conversion from PHP land to .sql script file, since I don't have intimate knowledge about the integration tests. Nor, do I want to.

Better parsing of host:port would be nice. But unless I run it that problem in my project. I will probably not fix it.

@dotnetCarpenter
Copy link
Author

dotnetCarpenter commented May 11, 2017

A few pointers to help you inspect Docker.

  1. docker exec -it db-sync-test-mysql bash - get a bash terminal inside the container
  2. mysql --port 33556 -h 0.0.0.0 -u root -p - connect via mysql cli to the mysql server inside the container (using port forwarding)
  3. docker exec -it db-sync-test-mysql mysql -u root -p (directly through docker - no port forwarding)
  4. docker ps - see what is currently running in docker
  5. ifconfig - see network interfaces

You can use 3 if you really just want to get inside mysql but I recommend 2, since that is the access every container or PHP script you write will gain access. 3 can work, even if 2 doesn't.

@dotnetCarpenter
Copy link
Author

This post helped me a lot to understand the workings of docker network: moby/moby#2174 (comment)

@coveralls
Copy link

coveralls commented May 17, 2017

Coverage Status

Coverage remained the same at 99.605% when pulling bec2275 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

- setup docker on travis

- speed up travis build by caching the vendor directory - used by composer install

- Describe how to run tests with Docker

- be double sure that the docker instance has started before we run tests - this implementation asures that double dash parameters are forwarded correctly to phpunit from outside (e.i. .travis.yml)
@coveralls
Copy link

Coverage Status

Coverage decreased (-0.08%) to 99.528% when pulling 1603262 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

@coveralls
Copy link

coveralls commented May 17, 2017

Coverage Status

Coverage remained the same at 99.605% when pulling 1603262 on dotnetCarpenter:docker into 020fa60 on mrjgreen:v3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants