WIP: Add Dockerfile #23

pedrommone · 2016-10-04T23:36:04Z

This is a partial PR of #12.

Dockerfile

This is just a simple image with official Python 3.5 and all dependencies installed.

docker-compose.yml

I've split the whole application into four containers and one of them is the official postgres 9.6. This is just a base to work.

.travis.ci.yml

I've added some new cool stuff into the CI pipeline. Now we generate the docker images too, yay! I hope to improve the CI pipeline even more for fully CI/CD.

coveralls · 2016-10-05T00:10:05Z

Coverage remained the same at 69.454% when pulling d3a43fa on pedrommone:add-docker-image into 76b6271 on datasciencebr:master.

pedrommone · 2016-10-05T00:10:59Z

@cuducos I need your help here! :)
In order to generate secured tokens for Docker Hub, I need you to sign up the org and generate the credentials for me, after that you need encrypt them with this Travis feature

coveralls · 2016-10-05T00:14:16Z

Coverage remained the same at 69.454% when pulling d3a43fa on pedrommone:add-docker-image into 76b6271 on datasciencebr:master.

coveralls · 2016-10-05T00:26:52Z

Coverage remained the same at 69.454% when pulling 879c4f9 on pedrommone:add-docker-image into 76b6271 on datasciencebr:master.

cuducos · 2016-10-05T15:06:26Z

@pedrommone I added you as a contributor to hub.docker.com/u/datasciencebr

Could you generate the keys yet?

cuducos

Many many thanks, @pedrommone!

I left some inline comments. Please, don't take me wrong, I'm not a dick (I guess).

I was just trying to learn from your files, I'm a newbie to Docker and most of the lines were pretty straightforward — so I just asked about bits that I had doubts, just as a matter of curiosity ; )

Finally, one general question: what's the advantage of having separated containers from migration and seeding? Why not just running python manage.py migrate (at every deploy) and python manage.py loaddatasets etc. (on provision) within Jarbas main container?

cuducos · 2016-10-05T15:08:25Z

.travis.yml

+
+env:
+  global:
+    - secure: "" # DOCKER_EMAIL


How does that secure works? Should I set these variables in Travis CI? Drop a line at telegram.me/cuducos so we can sort that out.

I just discovered this feature too, travis has something to encrypt sensible data! You learn more here.

That's awesome — not sure if this encryption is really safe hahaha… I still tend to prefer to leave credentials in the Settings panel ; )

Of course, that's a lot better. We can change that.

cuducos · 2016-10-05T15:08:56Z

.travis.yml

+    - secure: "" # DOCKER_USER
+    - secure: "" # DOCKER_PASS
+    - REPOSITORY: "datasciencebr/jarbas"
+    - COMMIT=${TRAVIS_COMMIT::8}


Just out of curiosity: what does this line do?

Just to avoid some DRY.

But what is this exactly? The name/hash of the commit?

Yep, it just put into environment some useful information.

cuducos · 2016-10-05T15:10:06Z

.travis.yml

@@ -28,3 +39,9 @@ script:

 after_success:
  - coveralls
+  - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASS


Sorry, n00b question again: what's the point of building and pushing the image (?) after the tests?

Actually is is something I want to improve, build a image every time and run tests on top of it.

Got it. Thus maybe it should be on before_script section, so when you got to the script (the tests) section, the Docker image is ready…

If these commands take a while to exit, take a look on travis_wait.

it's just a login procedure, we need to call it before pushing images. Anyway I'll move it because we'll need in future.

cuducos · 2016-10-05T15:10:23Z

Dockerfile

@@ -0,0 +1,17 @@
+FROM python:3.5
+
+MAINTAINER Pedro Maia <pedro@pedromm.com>


Yay! Thanks for that ; )

You're welcome.

cuducos · 2016-10-05T15:11:46Z

docker-compose.yml

+  postgres:
+    image: postgres:9.6
+    container_name: jarbas-postgres
+    environment:


What is a good practice when it comes to sensible info like this? Does Docker reads from server envvar or something?

Actually we need to improve that too, I'll spend more time on it when we get a infrastructure to work on.

cuducos · 2016-10-05T15:19:16Z

docker-compose.yml

+  populate:
+    image: jarbas
+    container_name: jarbas-populate
+    command: python manage.py loaddatasets


Actually there are two extras commands to populate the database, making a total of three:

python manage.py loaddatasets

python manage.py loadsuppliers

python manage.py ceapdatasets

Is this any of these syntax valid?

command: - python manage.py loaddatasets - python manage.py loadsuppliers - python manage.py ceapdatasets

Or (worse since if one command fails, the following ones will not be executed):

command: python manage.py loaddatasets && python manage.py loadsuppliers && python manage.py ceapdatasets

Actually append them with & doesn't work (something related to django). I'll improve it.

pedrommone · 2016-10-07T13:44:49Z

@cuducos I saw you request, I'll try to code more today.

pedrommone · 2016-10-11T12:31:36Z

Just some feedback. I've faced some issues:

In order to make it functional, I need to build an container with NodeJS and Python and this is not cool.
For fully CI integration, we'll need to move to another CI SaaS that provide pipelines.

How to run what I've done?

Clone this branch and run docker build -t jarbas . then run docker-compose up. It will provision all containers and will fail into nodejs requirement step.

cuducos · 2016-10-11T12:59:51Z

In order to make it functional, I need to build an container with NodeJS and Python and this is not cool.

Actually this is a good thing. We can create another container to run NodeJS and this is the first step to address Issue #18 (splitting front-end and backend cc @leomeloxp).

What I can do is to some edit the repo in a separate branch and send a PR to this branch here. I make the Django and the Elm commands independent of each other — i.e. The elm-make command will be ran outside Django (e.g. npm run build instead of python manage.py assets build).

The only link between both worlds will be that the output file will be saved directly into the staticfiles/ (where Django HTML template will look for app.js). Surely later we can use nginx to have NodeJs creating the HTML too, but I'd say this is mostly a matter of Issue #18 (we're gonna address it later anyway).

Does that help?

For fully CI integration, we'll need to move to another CI SaaS that provide pipelines.

I have no idea about this limitation, but I don't mind leaving Travis CI either. Any ideas? cc @luiz-simples @vitallan @gwmoura @ayr-ton

Clone this branch and run docker build -t jarbas . then run docker-compose up. It will provision all containers and will fail into nodejs requirement step.

Does the separation I mentioned in my first point here helps? IMHO it worth it to have a working local version before depending on a CI tool. So if we fix the NodeJS thing the only pending issue will be the CI, am I right?

pedrommone · 2016-10-11T13:29:21Z

Does that help?

Yep. it helps a lot.

Does the separation I mentioned in my first point here helps? IMHO it worth it to have a working local version before depending on a CI tool. So if we fix the NodeJS thing the only pending issue will be the CI, am I right?

Yep, we'll have some work to be done here. :)

cuducos · 2016-10-11T14:00:28Z

Done in extract-nodejs. No Django command depends on NodeJS anymore.

app.js is generated by npm run assets (not by python manage.py assets build)
ceapdatasets.html loads the CSS from a CDN (not from node_modules/)

Is there anything I might have left behind?

I felt that if I isolate it more I'd be splitting the repo and I'd like to do this with Docker linking front-end and back end. Otherwise I don't know how to locally have a static front-end (generated by NodeJS) at root (e.g. http://localhost:8000/) and a Django app at /api/ (e.g. http://localhost:8000/api/) serving it.

gwmoura · 2016-10-17T13:15:45Z

For fully CI integration, we'll need to move to another CI SaaS that provide pipelines.

@cuducos we can create our simple pipeline on Travis CI like: test > build > push to docker hub > deploy container on server. One script for each step, the big problem is the deploy on the server.
Today I am using tsuru.io, but as we need deploy one app, I think better study docker swarm (a docker orchestration tool) or use some service of container deploy.

gwmoura · 2016-10-17T13:25:43Z

e a static front-end (generated by NodeJS) at root (e.g. http://localhost:8000/) and a Django app at /api/ (e.g. http://localhost:8000/api/) serving it

@cuducos about this, we can create on the project two folders and use docker-compose to manage this environments:

web/
   |__ Dockerfile
   |__ files to frontend app
api/
   |__ Dockerfile
   |__ files to back-end app
docker-compose.yml

# docker-compose.yml content some like this
version: '2'
services:
  db:
   images: postgresql
  web:
    build: ./web
    ...
  api:
    build: ./api
    ...

pedrommone · 2016-10-17T15:18:36Z

@gwmoura the main problem with this approach is the Travis itself, its made for Continuous Integration, not Continuous Delivery. We can't build a pipeline with Travis.

I'm sorry for holding this MR for too long, I've been busy with work and college (I need to finish my thesis).

cuducos · 2016-11-26T01:07:54Z

Closing this PR since we get a working docker-compose.yml on master branch 🎉

pedrommone added 6 commits October 4, 2016 20:34

Add very first version for Dockerfile

e939068

WIP: Add docker-compose.yml setup

e0b9333

Remove space

c4379c6

Update python version to match travis ci

8b19056

Just some..

7ddc984

Add docker build step

d3a43fa

Add end of line

879c4f9

cuducos reviewed Oct 5, 2016

View reviewed changes

cuducos mentioned this pull request Oct 20, 2016

Automate deploy (and, eventually, provision) #12

Closed

cuducos added the work in progress label Oct 24, 2016

cuducos closed this Nov 26, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add Dockerfile #23

WIP: Add Dockerfile #23

pedrommone commented Oct 4, 2016 •

edited by cuducos

Loading

coveralls commented Oct 5, 2016

pedrommone commented Oct 5, 2016

coveralls commented Oct 5, 2016

coveralls commented Oct 5, 2016

cuducos commented Oct 5, 2016

cuducos left a comment

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

cuducos Oct 5, 2016

pedrommone Oct 11, 2016

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

cuducos Oct 5, 2016

pedrommone Oct 5, 2016

pedrommone commented Oct 7, 2016

pedrommone commented Oct 11, 2016

cuducos commented Oct 11, 2016 •

edited

Loading

pedrommone commented Oct 11, 2016

cuducos commented Oct 11, 2016

gwmoura commented Oct 17, 2016

gwmoura commented Oct 17, 2016

pedrommone commented Oct 17, 2016

cuducos commented Nov 26, 2016

		@@ -0,0 +1,17 @@
		FROM python:3.5

		MAINTAINER Pedro Maia <pedro@pedromm.com>

WIP: Add Dockerfile #23

WIP: Add Dockerfile #23

Conversation

pedrommone commented Oct 4, 2016 • edited by cuducos Loading

Dockerfile

docker-compose.yml

.travis.ci.yml

coveralls commented Oct 5, 2016

pedrommone commented Oct 5, 2016

coveralls commented Oct 5, 2016

coveralls commented Oct 5, 2016

cuducos commented Oct 5, 2016

cuducos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pedrommone commented Oct 7, 2016

pedrommone commented Oct 11, 2016

How to run what I've done?

cuducos commented Oct 11, 2016 • edited Loading

pedrommone commented Oct 11, 2016

cuducos commented Oct 11, 2016

gwmoura commented Oct 17, 2016

gwmoura commented Oct 17, 2016

pedrommone commented Oct 17, 2016

cuducos commented Nov 26, 2016

pedrommone commented Oct 4, 2016 •

edited by cuducos

Loading

cuducos commented Oct 11, 2016 •

edited

Loading