Migrate to Docker from Vagrant

[electricjones]

Picking up on Issues #120 and #123 . This is not ready to merge yet

Alright, after a lot of learning, I managed to create a setup that works for orientdb for now. It should be easy enough to add neo and gremlin.

clone the docker branch
docker-compose up -d
docker run -it --link spider_orientdb:orientdb --net spider_default -v /Mac/users/michael/code/spider/spider:/spider spider/php bash

The second command is specific to my system. replace the /mac/users... path with the absolute path of spider. (for windows /c/path/to/spider).

This will all be abstracted away beneath a spider-dev.sh bash file. After a long chat with a coworker who manages our docker system, I think it'll be best to skip docker-compose for now. If we do everything through docker build and docker run abstracted behind a spider-dev bash file, then we give the users the ability to cherry pick and mix-and-match the versions of php and database.

Also, I had to use a php image from ubuntu:15.10 which I hate. It's huge. I tried several others, but got frustrated b/c I either couldn't install xdebug or they weren't compiled with sockets enabled (I can't figure out why). I am sure there exists, or we can define a better one. For now, though, I just wanted to get it working.

Still ToDo:

• Remove vagrant
• Setup PHP and Orient to work with shell
• Move fixture and testing credentials to .env file (that's why tests fail right now)
• Add Neo4j
• Add Gremlin
• Add a php 5.6 container
• Add versioning for supported versions to docker-compose.yml (https://docs.docker.com/compose/compose-file/#variable-substitution)
• Initialize spider-dev.sh to abstract away docker commands (from [App] Introduce a (basic) spider-cli application #123)
• Add testing commands to spider-dev (spider-dev test -iau)
• Integrate with Travis
• Documentation and lots of testing

[electricjones]

Neo4j and OrientDB are up and running. I ran into some problems adding the Gremlin container. Will try later this week.

To get this running at this point:

docker-compose up
docker run -it --link spider_orientdb:orientdb --link spider_neo4j:neo4j --net spider_default -e "SPIDER_DOCKER=true" -v /Mac/users/michael/code/spider/spider:/spider spider/php bash

Replacing the volume path with your own. The next step is to get docker-compose.yml to handle these complexities and multiple versions of databases and php.

[smolinari]

The only way I can see a docker-compose.yml handle different versions is to have different docker-compose.yml files with a different base dockerfile.

I've been working with Docker now too the last couple of days and learning a lot. 😄 I've also gotten a PHP image created with Alpine Linux with PHP7 and it has composer + phpunit globally pre-installed. You can get the image with

FROM skooppaos/php7

I can easily create a php5.6 image too. Would we need any other PHP versions?

Scott

[electricjones]

That's great! does that php container come with sockets enabled? The last time I tried an alpine container, I couldn't get sockets to work, which is ridiculous. I think 5.6 and 7.0 are all we are really supporting right now. It would be great to have them both installed on the same container, but two different boxes would be fine as well.

And docker-compose.yml supports Variable Substitution which I think should do the trick. Actually, if we push a working php container to hub.docker.io, then we won't need any dockerfiles whatsoever.

[smolinari]

I have sockets enabled in the PHP compilation. Is that what you mean?

Scott

[electricjones]

Yes, that is exactly what I mean. Thank you.

[electricjones]

And, I take it back. Two different tags (FROM skooppaos/php:7.0, FROM skooppaos/php:5.6) or something similar would be perfect. Also, would you consider making that repo public on github? I'd like to see the Dockerfile, and others may as well.

[smolinari]

Ok. I'll push them with those tags.

It isn't as easy as I thought to make a 5.6 image. Working on it though.

Scott

[electricjones]

Your container worked great.

I have also simplified what is needed to run the containers:

docker-compose run php sleep 15;phpunit will run all the phpunit tests

The first time you run it, it will pull in and build everything. The "sleep 15" needs a better solution. The php container cannot start running its tests until the db services are up and ready. The best alternative (though more complex than I like) is to have the php bootstrap file check for connections, and hold up the tests until the connections are good.

I'd love to find a better solution.

To open a full terminal, you must still

docker-compose up -d
docker run -it --link spider_orientdb:orientdb --link spider_neo4j:neo4j --net spider_default -e "SPIDER_DOCKER=true" -v /Mac/users/michael/code/spider/spider:/spider spider/php bash

Also open to a better solution here.

One last note: I have moved all the credentials to environment variables. This means you can use the docker's php container, or use php locally. Simply add a .env file to the spider root with the services credentials and you can run the tests and such from your host machine. For me, this was a huge performance boost.

[electricjones]

I have pushed a basic shell script to handle docker testing and management. I'm happy with it, though I'm sure it could use some refactoring and testing.

See bin/spider-dev for api.

From spider directory bin/spider-dev test to run all tests, bin/spider-dev up to start services, etc.

Two important notes

• If you (like me) are using the docker private beta for mac, you must include the flag --mac for now. This will be removed once beta goes out.
• The options MUST come before the command. At least for now.

@smolinari Have you gotten any luck with a php5.6 container? I also noticed that this one doesn't support bash or shell. At least not that I've found. We need a way to do something like

docker run -it skooppaos/php7 bash and open up a terminal.

Next step is to finish versioning in the docker-compose file, and then add gremlin support.

[smolinari]

@chrismichaels84 - I haven't finished the 5.6 version yet. Working on it today. The php7 version is also finished and coincidentally, I also added bash, because of the timeout issue I'm having with ODB (which you also mentioned), I was trying to incorporate wait-for-it. So the new image should work for you now. It is also under skooppaos/php:7.0 as you wished.

I like the script you made. I guess I'll have to test it. 😄

Edit: I've also uploaded the files to a repo: https://github.com/skooppaos/docker-files

Edit2: I actually won't be getting the 5.6 version done today. I had too much work on my house and now I have to go shopping for the weekend. So, I'll get to it tomorrow.

Scott

[electricjones]

The container looks great. Good work. And wait-for-it could come in handy. I've run into a problem where I have to make containers sleep before running tests b/c we have to wait for the databases to be ready. Something like this could be super helpful. And have a good weekend.

[electricjones]

Just pushed update with full-ish versioning control. If you try to use an unsupported version, you will get an error, but it will let you try. If there is a docker image with that tag, it will load.

Also added spider-dev versions which prints some info.

Does NOT include actual supported versions. That will have to be its own ticket to see what spider actually supports. This way, though, we get to keep each branch and its supported versions together, which is nice.

Last cleanup items before this gets tested and merged (other than documentation)

• Some simple refactoring
• Better handling of working directories
• Add gremlin container

Would be nice:

• Integrate wait-for-it so that spider-dev doesn't run tests or open a shell until the environment is ready. I first thought to do this is phpunit, but I think those tests should fail if the service is unavailable. This deserves its own ticket and is unimportant atm.

My last thought is along with #123, add a spider.php cli utility with a command to build this bash file automatically from a supported_versions.json file or something similar. We can use the same meta file in Travis or anything else. One place to change whatever.

[smolinari]

Just an FYI. Spider can't support ODB 2.2 yet, because the OrientPHP driver doesn't work properly with 2.2, unfortunately.

Ostico/PhpOrient#71
orientechnologies/orientdb#6143

Scott

[smolinari]

Yup, I can't get Docker to do as I want on my Win7 system and I've tried all sorts of things. I am giving up, which I hate to do. I guess I'll stick with Vagrant for now as you suggest.

The only thing missing is PHP7 support in the spider-box. I'll see if I can add that in the near future.

Scott

[coveralls]

Coverage decreased (-10.02%) to 84.578% when pulling f9bf9d8 on docker into 3880e00 on develop.

[electricjones]

Well, this is actually working -- and all finished with gremlin and all -- but there are two external problems:

• The Gremlin container does not work with transactions. I have opened a ticket Transactions fail dmill-bz/docker-gremlin-server#2. For now, that one test is skipped for gremlin
• Travis seems to be having trouble with the neo container. However it all works locally.

[electricjones]

So, I've run into a really weird problem. It seems that (only on travis) the neo4j tests can't connect to neo4j when the gremlin database is running. Works fine on my local machine. When the gremlin db is disabled, all works well. Even when the gremlin tests are skipped -- if the db is up, then neo4j isn't.

But only on travis.

@PommeVerte any idea what may be causing this?

[electricjones]

I have decided to revert to a point where everything is working except gremlin, merge this into develop and then into the other two PRs. This will allow me to use it in real situations for testing.

We just have to be sure not to release develop until the gremlin container is working as expected. If I find myself with time and @PommeVerte isn't able to look at it, I may fork the gremlin container and see what I can do. I need to learn tinkerpop better anyway.

[coveralls]

Coverage decreased (-65.5%) to 29.058% when pulling d31fa8d on docker into 3880e00 on develop.