Docker Compose Based Development Environment #41

conatus · 2017-06-26T16:08:21Z

This allows you to run: docker-compose up and have a working development environment on your machine. Or follow the instructions in the README.md for a bit more detail.

@chriscroome have a go with this and see if it works for you.

I'll then add in the piece which updates the images programmatically.

conatus · 2017-06-26T16:31:59Z

@nicksellen

Would be cool if you took a look too.

nicksellen

Ok, review added! A few minor details then a suggestion to add an "assets" container to do cotech images and gulp watching.

nicksellen · 2017-06-27T08:05:43Z

docker-compose.yml

+    ports:
+      - 13306:3306
+    volumes:
+      - ./docker/db/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d


We should use consistent folder mapping names, I used - ./docker/db/data:/docker-entrypoint-initdb.d in af.

See https://gitlab.outlandish.com/audience-agency/audience-finder/blob/docker-compose/docker-compose.yml#L38

nicksellen · 2017-06-27T08:06:55Z

docker-compose.yml

+      WP_ENV: development
+      WP_HOME: http://localhost:18080
+      WP_SITEURL: http://localhost:18080/wp
+      COMPOSER_ALLOW_SUPERUSER: 1


Would be better in Dockerfile as an ENV setting (as it's not a user/app specific configuration)

Like here https://github.com/api-platform/api-platform/blob/master/Dockerfile#L38

nicksellen · 2017-06-27T08:08:07Z

docker/wordpress/docker-entrypoint.sh

+export COMPOSER_NO_INTERACTION=1
+
+# Composer does not warn when we are superuser inside the Docker container
+export COMPOSER_ALLOW_SUPERUSER=1


Ah, defined in another place too! I think Dockerfile is still the best place for it, set and forget.

nicksellen · 2017-06-27T08:09:45Z

docker/wordpress/docker-entrypoint.sh

+
+cd /var/www/html
+
+composer update


I switched to using composer install, the update command will actually change the composer.lock file to find newer versions where possible, we want it to just install what composer.lock says.

This is also why we need an image registry so we aren't copying this things around any more :)

This is a problem overall. If we iterate on this docker-entrypoint.sh, how do we make sure it is kept up to date across projects as we improve it.

nicksellen · 2017-06-27T08:27:41Z

docker/wordpress/docker-entrypoint.sh

+
+echo "Compiling WordPress theme"
+
+cd /var/www/html/web/app/themes/coop-tech-oowp-theme


I think we need to think a bit more about this. When we have a standard wordpress base image we will not be able to use this approach there.

In this case I would create another container for the gulp tasks for two reasons:

keeps the wordpress image focused on doing one thing in preparation for our custom image

we can provide a better dev experience by running gulp watch to recompile-on-change (as it uses this internally https://laravel.com/docs/5.0/elixir#gulp)

Having to run composer install at top-level and inside the theme seems less than ideal to me. One approach is to define the theme directory as a local composer repository and so long as it is marked as being a theme, the wordpress install would ensure it ends up in the correct place (or if it is already in the correct place I think the repository configuration has some options for that). See the "path" method on https://getcomposer.org/doc/05-repositories.md.

One of the outlandish projects uses this approach previously, but then after moving to bedrock layout it was easier to just add the plugins to the repo directly (they did not need any composer install).

... an alternative to the local directory composer repo would be to get the entrypoint script to descend the themes (and plugins/mu-plugins?) and run composer install if it finds a composer.json file...

I agree it would be nice for a separate container to run gulp watch etc but this sort of stuff would still be needed.

However, the problem is that the WordPress image is not in a functioning state without running these commands. The two are intertwined. You need the asset compilation and you need the other bits to be present or the WordPress theme doesn't work. If you said "you need both containers running to be this site" then we have defeated the object of the exercise.

The reason for it running inside the theme was I was copying the install instructions in the repository, but looking at the composer.json this is likely not needed.

nicksellen · 2017-06-27T08:31:20Z

docker/wordpress/docker-entrypoint.sh

+./node_modules/gulp/bin/gulp.js
+
+cd /var/www/html
+curl https://www.coops.tech/cotech-images.tgz > cotech-images.tgz


Same comment about keeping wordpress image dedicated. Also these will get outdated over time, although not much we can do about that for now.

Perhaps if there was a gulp container, the command (CMD override in docker-compose.yml) could run this before running gulp watch? So the gulp container purpose is about "assets".

Also, don't want to run this each time we start the container, so should check if it is already there...

Same response - the container isn't in a functioning state until these assets are compiled.

How can we move forward on this?

Talked on Slack - basically going to go with an assets container here.

chriscroome · 2017-06-28T10:44:05Z

This is great :-) There were a few small issues like I needed to do a recursive chown, but it seems to work really well!

conatus · 2017-06-28T10:48:36Z

@chriscroome What recursive chown is needed mate? Need to add this in.

chriscroome · 2017-06-28T11:09:20Z

This is the error I had:

~/cotech/website/web/app/themes/coop-tech-oowp-theme$ npm install
npm ERR! Linux 4.9.0-3-amd64
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install"
npm ERR! node v6.11.0
npm ERR! npm  v3.10.10
npm ERR! path /home/chris/cotech/website/web/app/themes/coop-tech-oowp-theme/node_modules/.staging
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall mkdir

npm ERR! Error: EACCES: permission denied, mkdir '/home/chris/cotech/website/web/app/themes/coop-tech-oowp-theme/node_modules/.staging'
npm ERR!     at Error (native)
npm ERR!  { Error: EACCES: permission denied, mkdir '/home/chris/cotech/website/web/app/themes/coop-tech-oowp-theme/node_modules/.staging'
npm ERR!     at Error (native)
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/home/chris/cotech/website/web/app/themes/coop-tech-oowp-theme/node_modules/.staging' }
npm ERR! 
npm ERR! Please try running this command again as root/Administrator.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/chris/cotech/website/web/app/themes/coop-tech-oowp-theme/npm-debug.log

I'm sorry I don't recall exactly which directory was www-data:www-data -- I was trying to get it all up and running quickly, I can repeat the process from scratch on another machine when I get a chance -- I simply did a chown chris:chris -R ~/cotech/website/ to fix it.

chriscroome · 2017-06-28T20:54:40Z

I have installed everything from scratch on a Debian Stretch machine and documented what I did on the wiki, when I got to the tasks in the post-merge git hooks script I hit the permissions issue:

composer update
Loading composer repositories with package information
                                                      
Could not fetch https://api.github.com/repos/outlandishideas/routemaster/contents/composer.json?ref=master, please create a GitHub OAuth token to go over the API rate limit
Head to https://github.com/settings/tokens/new?scopes=repo&description=Composer+on+XXX+2017-06-28+2138
to retrieve a token. It will be stored in "/home/chris/.composer/auth.json" for future use by Composer.
Token (hidden): 
Token stored successfully.
Updating dependencies (including require-dev)                        
Nothing to install or update

                                                                                                                          
  [ErrorException]                                                                                                        
  file_put_contents(/home/chris/cotech/website/vendor/composer/installed.json): failed to open stream: Permission denied  
                                                                                                                          

update [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--lock] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [--with-dependencies] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [-i|--interactive] [--root-reqs] [--] [<packages>]...

It is the website/vendor directory that is owned by www-data:

ls -lah /home/chris/cotech/website
total 208K
drwxr-xr-x  8 chris    chris    4.0K Jun 28 21:20 .
drwxr-xr-x  4 chris    chris    4.0K Jun 22 07:55 ..
-rw-r--r--  1 chris    chris    3.8K Jun 22 07:55 CHANGELOG.md
-rw-r--r--  1 chris    chris    2.9K Jun 28 21:40 composer.json
-rw-r--r--  1 chris    chris    107K Jun 28 21:03 composer.lock
drwxr-xr-x  3 chris    chris    4.0K Jun 22 07:55 config
drwxr-xr-x  5 chris    chris    4.0K Jun 28 21:03 docker
-rw-r--r--  1 chris    chris    1.2K Jun 28 21:03 docker-compose.yml
-rw-r--r--  1 chris    chris     663 Jun 22 07:55 .env.example
drwxr-xr-x  8 chris    chris    4.0K Jun 28 21:31 .git
-rw-r--r--  1 chris    chris     127 Jun 28 21:03 .gitattributes
drwxr-xr-x  2 chris    chris    4.0K Jun 28 21:30 git-hooks
-rw-r--r--  1 chris    chris     325 Jun 22 07:55 .gitignore
-rw-r--r--  1 chris    chris     35K Jun 22 07:55 LICENSE.md
-rw-r--r--  1 chris    chris    1.5K Jun 28 21:03 README.md
drwxr-xr-x 22 www-data www-data 4.0K Jun 28 21:19 vendor
drwxr-xr-x  5 chris    chris    4.0K Jun 28 21:18 web
-rw-r--r--  1 chris    chris      13 Jun 22 07:55 wp-cli.yml

So a sudo chown -R ${SUDO_USER}:${SUDO_USER} solves it... but perhaps there is another way that would be cross platform compatible?

conatus · 2017-06-29T06:53:48Z

@chriscroome

I haven't changed anything about the Git hooks stuff here, see the diff, so if you are having problems its not the result of this merge request which is only about Docker Compose and doesn't touch these files.

chriscroome · 2017-06-29T11:29:11Z

if you are having problems its not the result of this merge request

The only issue I have raised is the ownership of the vendor directory and it was you who suggested that something needed to be done about this, I don't believe that running composer update caused the ownership issue, it simply exposed it. I have documented how to reproduce this on Debian Stretch.

chriscroome · 2017-06-29T18:23:04Z

Just to add that without running the following tasks after running docker-compose up -d you have a site with no CSS or images, so I think we should document that these things need to be run, together with a chown, or am I misunderstanding something here?

composer update
cd web/app/themes/coop-tech-oowp-theme
composer update
npm install
nodejs ./node_modules/gulp/bin/gulp.js

These are the things that are run when the live or dev site are updated.

nicksellen · 2017-06-30T11:38:29Z

@conatus so, a summary of my changes:

move image downloading to assets container (as they are assets)
download images only if uploads/2017 does not exist (+ option to force a new download, see readme)
hopefully fix yarn/node-sass issue (make it explicit dep in package.json and run yarn add --force node-sass)
remove instructions from readme about composer/gulp outside of container
make node image use explicit version so it is deterministic (:latest will change over time)

nicksellen · 2017-06-30T12:01:17Z

And final change:

the wordpress entrypoint script will now check if there is composer.json in the theme dir and run composer install there if so

nicksellen · 2017-06-30T12:04:00Z

@chriscroome all this should be addressed now, just to make it clear where they each happen:

composer update

This is to install composer deps from root dir, runs in wordpress docker entrypoint script each time container is started. Uses composer install to use versions pinned from composer.lock.

cd web/app/themes/coop-tech-oowp-theme
composer update

This is also run in wordpress docker entrypoint by checking for a composer.json in the theme directory (as described in previous comment). Also runs each time container is started.

npm install

This is run (although using yarn) in the assets container now, and runs each time the container is started. Defined in docker-compose.yml.

nodejs ./node_modules/gulp/bin/gulp.js

This also runs in the assets container and is the command for that container. Defined in docker-compose.yml (as gulp watch though, so gets compiled on change).

nicksellen · 2017-06-30T12:06:46Z

@chriscroome and to clarify - all this docker stuff only applies for development environments, perhaps production deployment using docker could be looked into in the future, but this is not addressed here at all.

conatus

I think this is great stuff. Nice one.

conatus · 2017-06-30T14:11:55Z

docker/assets/download-images.sh

+# we just check for one path that we assume will be created when we unpack
+TEST_PATH="$DEST/app/uploads/2017"
+
+if [ ! -e $TEST_PATH ] || [[ $* == *--force* ]]; then


More shell scripts is it?

Indeed, it runs in a controlled environment (i.e. a docker image that we choose and is always the same), and is pretty simple. I don't see an alternative, but happy if you have a suggestion.

Requirements are probably:

downloads images automatically on first use

does not download every time the container is started if they already exist

can override if needed to get download even if they already exist

I was just joking - I see no other way you could do this - you know my feelings about shell scripts, but this is cool.

conatus · 2017-06-30T14:12:05Z

docker/assets/download-images.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+


Fixed, thanks

nicksellen · 2017-06-30T21:06:07Z

@chriscroome I just read https://wiki.coops.tech/wiki/CoTech_WordPress#Debian_Stretch_dev_env - no need to install anything other than docker on your dev machine. Everything else (composer, nodejs, php, php extensions, etc) is installed inside docker (this is why it's so useful!).

So, if you feel like it you can do a fresh install and skip the Install Nodejs, Install Composer and PHP modules, and Run Composer steps.

conatus · 2017-07-01T01:37:54Z

@chriscroome

If you could look to @nicksellen's instructions - i.e. you don't need to install anything apart from Docker - it would be really cool to get this reviewed as soon as possible. Thanks a lot mate!

chriscroome · 2017-07-03T20:25:25Z

Sorry not to have had the time and energy to looks at this sooner.

So, removing the www-data:www-data owned vendor directory and the unneeded packages and restarting Docker and updating the code:

sudo rm -rf vendor/
sudo aptitude remove nodejs docker-compose php7.0-mbstring php7.0-xml php7.0-curl
sudo service docker restart
git pull

Checking with docker ps and there are no containers running, so tab completion suggestions for docker run have some options, none of which seen to work?

docker run website_wordpress
Starting up entrypoint script
Composer could not find a composer.json file in /var/www/html
To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section

docker run adamyeats/docker-nginx-hhvm-bedrock
error: missing DB_HOST and MYSQL_PORT_3306_TCP environment variables
  Did you forget to --link some_mysql_container:mysql or set an external db
  with -e DB_HOST=hostname:port?

Sorry I'm unclear how to start the docker container in order to be able to do things like running composer in the container? I guess I'm missing something obvious?

chriscroome · 2017-07-03T20:40:17Z

I seem to be getting somewhere:

docker run wordpress

Results in a WordPress web install running at http://172.17.0.2/wp-admin/setup-config.php

chriscroome · 2017-07-03T21:13:25Z

Is docker-compose installed in a container or do I need to install that locally in addition to docker-ce?

conatus · 2017-07-03T22:50:05Z

Install locally. Then things should just work. Its the only command you need, normal use of Docker isn't required.

…

On Mon, 3 Jul 2017 22:13 Chris Croome, ***@***.***> wrote: Is docker-compose installed in a container <https://docs.docker.com/compose/install/#install-as-a-container> or so I need to install that locally in addition to docker-ce? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATZJgDikpuXvSvum1bdCdzb6XXg-65vks5sKVl2gaJpZM4OFhom> .

nicksellen · 2017-07-04T07:39:39Z

@chriscroome

docker run wordpress

This command downloads and starts the official wordpress docker image from docker hub [1]. It doesn't relate to the cotech docker-compose setup.

docker-compose often comes bundled with docker setups as it's an official tool now, but you might need to install it separately. Anything version >=1.6 should be sufficient.

To update an existing docker-compose installation you may need to rebuild the images, you can add the ---build args to the up command, so:

docker-compose up -d --build

If nothing changed it won't actually do much (you'll see lots of ---> Using cache) so it doesn't hurt to run it like that.

The README should hopefully contain enough instructions anyway :) Let me know if I can help more, feel free to ping me on cotech slack too.

[1] https://hub.docker.com/_/wordpress/

chriscroome · 2017-07-04T08:18:30Z

Thanks for you patience, it is up and running now and I have updated the wiki, Debian Stretch has docker-compose (1.8.0-2) and that seems to work OK.

conatus added 9 commits June 23, 2017 15:53

First pass at WordPress docker-compose setup

bf40607

Re-arrange file

e5342e5

Update dependencies

8381bc8

Run commands to install dependencies

a5bb496

Compile assets, add Yarn lock file

e269c62

Install Node.js and Yarn

07363b6

Set environment variables for Composer overrides

567fb64

Add a database sync, with a seed to begin with

44ca590

Add treatment for images

d40189e

conatus requested a review from chriscroome June 26, 2017 16:08

Update README with Docker Compose instructions

2cbadea

Make a fresh download of uploads on each start of the containers

485ab51

nicksellen reviewed Jun 27, 2017

View reviewed changes

conatus added 5 commits June 27, 2017 10:34

Move COMPOSER_ALLOW_SUPERUSER to the highest level

aa404d0

Switch to Composer install

a0be00c

Move location of database seed

de50d83

Simple assets container, that needs completion

9f05761

Docker compose for assets container

2917b63

bash shebang corrected

92556dd

nicksellen added 3 commits June 30, 2017 13:18

Specify node image version explicitly

e54a3bd

Add image download script

53bd243

Remove image downloading from wordpress image

9fb14b2

nicksellen added 7 commits June 30, 2017 13:20

Docker build context is now docker/wordpress

39de10f

Simplify build option, fixup node-sass install

2e05f81

Explicitly require node-sass

e0e4a55

Update yarn.lock

8322567

Remove outdated README instruction

a623a5a

Nicer output messages

c8b1cc7

Add note about how to redownload wordpress images

7ca61c3

Run composer install for theme if needed

2b2fc55

conatus commented Jun 30, 2017

View reviewed changes

Add set -e

14d3319

chriscroome merged commit 89a51c5 into master Jul 4, 2017

chriscroome deleted the docker-compose branch September 7, 2017 17:25


		echo "Compiling WordPress theme"

		cd /var/www/html/web/app/themes/coop-tech-oowp-theme

Docker Compose Based Development Environment #41

Docker Compose Based Development Environment #41

Conversation

conatus commented Jun 26, 2017 • edited Loading

conatus commented Jun 26, 2017

nicksellen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chriscroome commented Jun 28, 2017

conatus commented Jun 28, 2017

chriscroome commented Jun 28, 2017 • edited Loading

chriscroome commented Jun 28, 2017

conatus commented Jun 29, 2017

chriscroome commented Jun 29, 2017

chriscroome commented Jun 29, 2017

nicksellen commented Jun 30, 2017

nicksellen commented Jun 30, 2017

nicksellen commented Jun 30, 2017 • edited Loading

nicksellen commented Jun 30, 2017

conatus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksellen commented Jun 30, 2017 • edited Loading

conatus commented Jul 1, 2017

chriscroome commented Jul 3, 2017

chriscroome commented Jul 3, 2017

chriscroome commented Jul 3, 2017 • edited Loading

conatus commented Jul 3, 2017 via email

nicksellen commented Jul 4, 2017

chriscroome commented Jul 4, 2017

conatus commented Jun 26, 2017 •

edited

Loading

chriscroome commented Jun 28, 2017 •

edited

Loading

nicksellen commented Jun 30, 2017 •

edited

Loading

nicksellen commented Jun 30, 2017 •

edited

Loading

chriscroome commented Jul 3, 2017 •

edited

Loading