Docker Update and Continuous Deployment #19

rowanmanning · 2016-03-09T14:03:59Z

Outline

This PR contains the work required to get the build service continuously deploying, addressing #7. In order to do this we've had to overhaul the way that Docker is used, moving to Heroku's improved Docker support (which is currently in private beta). Here's what's been done:

Improved Docker support: Deployment is now done by pushing a built Docker image to Heroku's registry. I've simplified the commands required to do this:
- Local development: docker-compose build and docker-compose up
- Deploy: npm run build and npm run deploy
Continuous deployment: The above build/deploy commands no longer need to be run manually as they'll be run in CircleCI. Any push to the master branch will be deployed to QA once the tests have passed. Production is not deployed to through CI (see pipelines).
Pipelines: We're using a Heroku pipeline to deploy from QA to production. This makes a production deploy a simple button press, and the image that gets pushed into production is the exact same code that is running successfully in QA.
Cleanup: Removal of any files which are no longer necessary, and a cleanup of documentation/environment variables which are no longer used.

What Needs Doing Before/After Merge

First we need to agree that this is approach to continuous deployment that we'd like to take. I can demo everything to the team and we can have a chat about it.

Code-wise, everything is currently deploying happily to origami-buildservice-rowan-qa and origami-buildservice-rowan. We need to do a few things before this is ready to merge:

~~In circle.yml, change the deployment branch to master.~~
~~Change references in package.json from origami-buildservice-rowan-qa to origami-buildservice-qa.~~
~~Merge this PR.~~
~~Tag a new version on master and push it (wait for the deploy to work).~~
~~Verify that the new version is on QA and everything works.~~
Create a Heroku pipeline from origami-buildservice-qa to origami-buildservice-eu.
Promote the QA deploy to production.
Verify that the new version is on production and everything works.

Next Steps

These are things that I'd like to get done after this has been merged, but don't need to be done to get continuous deployment up-and-running:

Suggestion: Use .env for development environment variables instead of docker-compose.yml #18: using .env instead of docker-compose.yml for environment variables
Refactor the tests to run in the container #21: run the integration tests against a running container in CI

rowanmanning · 2016-03-09T14:09:15Z

@JakeChampion, @AlbertoElias, @alicebartlett, @onishiweb, @j-mes, would be good to get your thoughts/feedback on this. We can discuss in-person as a group if it's easier.

JakeChampion · 2016-03-09T14:12:33Z

.editorconfig

+insert_final_newline = true
+charset = utf-8
+
+[**{.js,.scss,.html,.mustache,.md,.yml}]


.yml forbids tabs

Oops, I got that wrong, and preventing yml from having tabs is the entire reason I added this file 😂

JakeChampion · 2016-03-09T14:15:41Z

Dockerfile

@@ -1,44 +1,22 @@
-FROM heroku/cedar:14
+FROM mhart/alpine-node:4


It is not mentioned anywhere in this PR why the change from cedar to alpine was made.

Below is the explanation given in the heroku-docker-node-alpine-example repository

Heroku's Cedar base image is based upon Ubuntu, which was not developed with the idea of being run inside a container, incidentally making the image size rather large. Heroku's Cedar also installs a lot of packages to cover the needs of most Heroku developers' applications.

Alpine Linux is built around musl libc and busybox. This makes it smaller and more resource efficient than traditional GNU/Linux distributions. An Alpine Linux image requires no more than 8 MB.

A smaller base image with fewer dependencies gives benefits such as a faster build/deploy time, smaller artifacts created for each build, and a smaller surface area to check for security vulnerabilities.

Is alpine likely to continue to be maintained? Is that a concern?

Not a major concern, the community around Alpine is large.

JakeChampion · 2016-03-09T14:19:20Z

README.md


-To SSH into the web container, you first need to SSH into the Docker VM, and then into the container you want:
+```sh
+open "http://$(docker-machine ip dev):8080/"


remove dev

JakeChampion · 2016-03-09T14:22:07Z

circle.yml

+  production:
+    branch: docker
+    commands:
+      - 'echo "machine api.heroku.com login rowan.manning@ft.com password ${HEROKU_AUTH_TOKEN}" >> ${HOME}/.netrc; chmod 0600 /home/ubuntu/.netrc;'


Note: This application will fail to deploy from CI is @rowanmanning ever renews his Heroku authentication token.

We could maybe do with an Origami Heroku account for this kind of thing? It would also fail if I ever get removed from the Heroku org.

I believe this is not something the Heroku Working Group within the FT advocate people to do.

OK, we'll just need to keep an eye on it then

Yeah, this kind of sucks. I'm using my account for browser tests with nightwatch

in Next we use my HEROKU_AUTH_TOKEN and push it out to all our CircleCI configs. I can setup origami-build-service with this if you like and then Next can manage this for you ^^

For the unitiated, what does the machine command do and how is it available in the circle environment?

@triblondon run this on your mac cat ~/.netrc

You can create an Origami heroku account by the way, I spoke with the Heroku Working Group and they verified this is a good approach for Origami to take with it's deployments.

alicebartlett · 2016-03-09T14:27:31Z

this is a huge improvement to our deployment process, and this PR explains your changes really well too.

AlbertoElias · 2016-03-10T13:00:46Z

.editorconfig

+
+[**{.yml}]
+indent_style = space
+indent_size = 2


should me move this stuff into obt?

We could have a copy in there, yes, but the editorconfig plugins rely on there being a .editorconfig file in the project root.

Do you mean the IDE plugins? We should probably discuss what's the best way
to use editorconfig

This email was sent by a company owned by Financial Times Group Limited
("FT Group http://aboutus.ft.com/corporate-information/#axzz3rajCSIAt"),
registered office at Number One Southwark Bridge, London SE1 9HL.
Registered in England and Wales with company number 879531. This e-mail may
contain confidential information. If you are not the intended recipient,
please notify the sender immediately, delete all copies and do not
distribute it further. It could also contain personal views which are not
necessarily those of the FT Group. We may monitor outgoing or
incoming emails as permitted by law.

triblondon · 2016-03-10T14:21:58Z

Very nice.

Docker Update and Continuous Deployment

triblondon · 2016-03-23T04:08:49Z

package.json

-    "build": "touch ./appversion; docker-compose build; rm -f ./appversion"
+
+    "build": "docker build -t registry.heroku.com/origami-buildservice-qa/web .",
+    "deploy": "docker push registry.heroku.com/origami-buildservice-qa/web"


Just been thinking a bit more about this. It seems like these scripts are a mixture of tasks run inside the container and tasks run outside the container. Maybe we should follow Next and use Make to standardise outside-container-tasks like building the container and deploying it?

Sure thing, these dont depend on node modules so I see no issue extracting into a makefile.

I'm definitely up for adding a Makefile

rowanmanning and others added 12 commits March 7, 2016 14:51

Move to Heroku's official Docker support

af7327c

Remove npm dedupe

143a738

Deploy to QA rather than production

8f9589b

Fix a whitespace issue in docker-compose

ffe3cbc

Tidy up the cache removal

284594c

Remove some unnecessary files

b46cf23

Fix the broken npm cache deletion

e98fc6a

Document the new Docker setup

7554c98

Add an .editorconfig file for consistency

55c68a7

Tweak the README based on feedback

cc2820f

Deploy through CI

2d7619a

Make a few additional README tweaks

8f693eb

JakeChampion reviewed Mar 9, 2016
View reviewed changes

Require two space indentation in YAML files

56d652d

JakeChampion reviewed Mar 9, 2016
View reviewed changes

Re-add git config to enforce use of HTTPS

05cb2d2

JakeChampion reviewed Mar 9, 2016
View reviewed changes

Remove "dev" from a few Docker commands

fbf1203

JakeChampion reviewed Mar 9, 2016
View reviewed changes

rowanmanning added 2 commits March 9, 2016 14:24

Remove a TODO in the Dockerfile

e045d92

Remove nodemon from package.json

043e60c

AlbertoElias reviewed Mar 10, 2016
View reviewed changes

Ignore .e* files explicitly

d97ad07

rowanmanning changed the title ~~[do not merge] Docker Update and Continuous Deployment~~ Docker Update and Continuous Deployment Mar 15, 2016

Prep the config for going live

e81beea

rowanmanning added a commit that referenced this pull request Mar 15, 2016

Merge pull request #19 from Financial-Times/docker

97cd304

Docker Update and Continuous Deployment

rowanmanning merged commit 97cd304 into master Mar 15, 2016

rowanmanning deleted the docker branch March 15, 2016 09:56

triblondon reviewed Mar 23, 2016
View reviewed changes

conor-mullen mentioned this pull request Mar 15, 2021

[Snyk] Security upgrade mem from 1.1.0 to 4.0.0 #488

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Update and Continuous Deployment #19

Docker Update and Continuous Deployment #19

rowanmanning commented Mar 9, 2016

rowanmanning commented Mar 9, 2016

JakeChampion Mar 9, 2016

rowanmanning Mar 9, 2016

JakeChampion Mar 9, 2016

triblondon Mar 10, 2016

JakeChampion Mar 10, 2016

JakeChampion Mar 9, 2016

rowanmanning Mar 9, 2016

JakeChampion Mar 9, 2016

rowanmanning Mar 9, 2016

JakeChampion Mar 9, 2016

rowanmanning Mar 9, 2016

AlbertoElias Mar 10, 2016

matthew-andrews Mar 10, 2016

triblondon Mar 10, 2016

matthew-andrews Mar 10, 2016

JakeChampion Mar 10, 2016

alicebartlett commented Mar 9, 2016

AlbertoElias Mar 10, 2016

rowanmanning Mar 11, 2016

AlbertoElias Mar 11, 2016

triblondon commented Mar 10, 2016

triblondon Mar 23, 2016

JakeChampion Mar 23, 2016

rowanmanning Mar 23, 2016

		@@ -1,44 +1,22 @@
		FROM heroku/cedar:14
		FROM mhart/alpine-node:4

Docker Update and Continuous Deployment #19

Docker Update and Continuous Deployment #19

Conversation

rowanmanning commented Mar 9, 2016

Outline

What Needs Doing Before/After Merge

Next Steps

rowanmanning commented Mar 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alicebartlett commented Mar 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

triblondon commented Mar 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment