Automate deploy (and, eventually, provision) #12
Comments
|
Velho, posso auxiliar você a montar deploy de ambientes complexos usando Docker. Segue um flow parecido com o que venho montando: |
|
Great! I have little experience with Docker but with a tutor I guess I can use it properly. When are you available to talk about it? Boa! Eu tenho pouca experiência com Docker, mas com um tutor eu acho que eu consigo usar certinho. Quando tu pode conversar sobre isso? |
|
Me passa o seu telefone ou liga pra mim. (48) 8810-8161 |
|
Sure thing — I'm gonna drop you a line at your Gmail. |
|
How this one goes? The idea is to create a dockerfile and use travis to build image and deploy the application? |
|
@vitallan, I think yes. The ideia is delivery the app in a container, we can any CI service to build a image, save on registry and after deploy the app. |
|
@cuducos can I help you too? My Gmail is gwmoura@gmail.com |
|
Many thanks, @vitallan and @gwmoura for jumping in! This Tuesday I was in a call with @luiz-simples and he's probably working on a branch to fix this issue. In baby steps probably we'll start just with Docker for a dev environment to later integrate with CI (Travis is already in use for tests here, so probably we'll stick to Travis) and automate the deploy. Feel free to talk to @luiz-simples and you guys can find the best way to help him. I'm gonna to introduce you all via email. And count on me for doubts on the repo/app, on pairing etc. |
Looks good to me. I'd love to learn bits of Ansible — it's in my TODO list anyway… |
|
How this architecture must be? Something like We dont need build all the stuff, thats why Docker is magic! :) |
|
@luiz-simples you'll need something to orchestrate the containers, main it hard coded is a waste of time. In my opinion, something like Rancher (hosted) or AWS EC2 (PaaS) is alot better (I can help in both). |
|
@pedrommone many thanks!
I have almost no experience with Docker, but I'll try to help, please follow up if what I'm saying is non-sense. This install steps (i.e. everything we need to do to get stuff running) is detailed in the
Is this of any help?
We used to use a PaaS but now we are powered (supported, sponsored) by Digital Ocean, so we are deploying to a Digital Ocean droplet, is that ok? |
|
@luiz-simples started a branch with Docker env - https://github.com/luiz-simples/jarbas/tree/docker-environment. Looking the branch is good for me, we gonna wait the PR for test in dev and after we can work the deploy on Digital Ocean. |
|
@cuducos thanks for the awesome info. About the DO, is totally fine. I can help you with that and make something working well. As soon I get some time work on the containers. I will trace the following steps
What do you guys think? |
|
Two minor doubts:
|
|
Here is an exemple of two services |
|
@pedrommone this steps are this same, for me is good option, I don't work with hancher but I listen good opinions 😄. @luiz-simples is working in create a base image for the project e images for dev and production, when possible looks the branch.
I think better start creating a Docker env for development to help more people to use and contribute with the project and after we split the project in backend and front-end |
|
@gwmoura you can easily make everything up with |
|
I think better too, I prefer create everything on docker-compose.yml, as talk in 12factor.net - prod and dev should be the same. @luiz-simples is using docker-compose.yml and created some scripts to help and automate the deploy. I am a developer, then I don't know if creating a simple container can impact negative in production... we can work together and create a simple and efficient environment |
|
Simple containers are the "same" as VMs. If you split the services with containers you can benefit of all Docker official images. I'll write something and show you. |
|
@pedrommone I know, you don't need write 😄. Looks the branch - https://github.com/luiz-simples/jarbas/tree/docker-environment and comments what you think we need remove |
|
@cuducos witch |
|
Python 3.5.2, Django is the latest version (1.10.something specified at |
|
@cuducos yeah!!!! Now we need to configure the gunicorn, right? How can I open the app and see this title? |
|
Guys testing the Dockerfile pasted here, I received 2 errors, because the elm: I added more commands on Dockerfile to install elm-make My DockerfileFROM python:3.5
COPY requirements-dev.txt /requirements-dev.txt
RUN python -m pip install -r requirements-dev.txt
COPY ./ /code
WORKDIR /code
RUN apt-get update && apt-get install -y postgresql \
postgresql-contrib \
nodejs \
npm
RUN python manage.py migrate
RUN ln -s /usr/bin/nodejs /usr/bin/node
RUN npm install
# RUN python manage.py createsuperuser
# RUN python manage.py loaddatasets
# RUN python manage.py loadsuppliers
# RUN python manage.py ceapdatasets
# RUN python manage.py assets buildRunning the tests:$ docker run --rm jarbas python manage.py test
....................................................EE..
======================================================================
ERROR: test_status_code (jarbas.frontend.tests.test_home_view.TestGet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/webassets/filter/__init__.py", line 514, in subprocess
shell=os.name == 'nt')
File "/usr/local/lib/python3.5/subprocess.py", line 947, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'elm-make'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/code/jarbas/frontend/tests/test_home_view.py", line 8, in test_status_code
resp = self.client.get(resolve_url('home'))
File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 529, in get
**extra)
File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 333, in get
return self.generic('GET', path, secure=secure, **r)
File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 409, in generic
return self.request(**r)
File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 494, in request
six.reraise(*exc_info)
File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 686, in reraise
raise value
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 39, in inner
response = get_response(request)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/code/jarbas/frontend/views.py", line 10, in home
return render(request, 'frontend/home.html', context=context)
File "/usr/local/lib/python3.5/site-packages/django/shortcuts.py", line 30, in render
content = loader.render_to_string(template_name, context, request, using=using)
File "/usr/local/lib/python3.5/site-packages/django/template/loader.py", line 68, in render_to_string
return template.render(context, request)
File "/usr/local/lib/python3.5/site-packages/django/template/backends/django.py", line 66, in render
return self.template.render(context)
File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 208, in render
return self._render(context)
File "/usr/local/lib/python3.5/site-packages/django/test/utils.py", line 94, in instrumented_test_render
return self.nodelist.render(context)
File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 994, in render
bit = node.render_annotated(context)
File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 961, in render_annotated
return self.render(context)
File "/usr/local/lib/python3.5/site-packages/django_assets/templatetags/assets.py", line 87, in render
for url in bundle.urls():
File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 787, in urls
urls.extend(bundle._urls(new_ctx, extra_filters, *args, **kwargs))
File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 746, in _urls
*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 600, in _build
force, disable_cache=disable_cache, extra_filters=extra_filters)
File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 524, in _merge_and_apply
kwargs=item_data)
File "/usr/local/lib/python3.5/site-packages/webassets/merge.py", line 276, in apply
return self._wrap_cache(key, func)
File "/usr/local/lib/python3.5/site-packages/webassets/merge.py", line 218, in _wrap_cache
content = func().getvalue()
File "/usr/local/lib/python3.5/site-packages/webassets/merge.py", line 251, in func
getattr(filter, type)(data, out, **kwargs_final)
File "/usr/local/lib/python3.5/site-packages/webassets_elm/__init__.py", line 39, in input
self.subprocess(write_args, fake_write_obj)
File "/usr/local/lib/python3.5/site-packages/webassets/filter/__init__.py", line 516, in subprocess
raise FilterError('Program file not found: %s.' % argv[0])
webassets.exceptions.FilterError: Program file not found: elm-make.
======================================================================
ERROR: test_js (jarbas.frontend.tests.test_static_files.TestCompiledStatic)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.5/unittest/mock.py", line 1157, in patched
return func(*args, **keywargs)
File "/code/jarbas/frontend/tests/test_static_files.py", line 27, in test_js
call_command('assets', 'build', 'elm', '--no-cache')
File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 130, in call_command
return command.execute(*args, **defaults)
File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python3.5/site-packages/django_assets/management/commands/assets.py", line 87, in handle
'a non-zero exit code (%d).' % retval)
django.core.management.base.CommandError: The webassets build script exited with a non-zero exit code (2).
----------------------------------------------------------------------
Ran 56 tests in 2.027s
FAILED (errors=2)
Creating test database for alias 'default'...
Destroying test database for alias 'default'... |
|
@gomex Good to hear from you! Are you recovering well? I really hope so! Starting the app is done with commands such as @gwmoura The erros are there probably because you're using the |
|
@cuducos it's the my case, I gonna change my branch and test once more. |
|
Hey @cuducos I am trying to build the elm assets and static files. "python manage.py assets build" and "python manage.py collectstatic" are enough or should I do something else? Where are these files? |
|
Hi @gomex, The commands are a little bit different in
One option maybe is to:
Does that make sense? |
|
@cuducos @gomex using a container for NodeJS and running repo: https://github.com/gwmoura/jarbas/tree/extract-nodejs Dockerfile used for create a nodejs container: # Dockerfile-node
FROM node:6.9.1
COPY elm-package.json elm-package.json
COPY package.json package.json
RUN npm i
COPY . /code
WORKDIR /code
VOLUME /code
RUN npm install gulp gulp-elm gulp-rename gulp-uglify elm
RUN npm install -g gulp
CMD npm run assets
The error after execute the container with the command $ docker run --rm -v "`pwd`":/code nodejarbas
npm info it worked if it ends with ok
npm info using npm@3.10.8
npm info using node@v6.9.1
npm info lifecycle @~preassets: @
npm info lifecycle @~assets: @
> @ assets /code
> gulp elm
[13:56:08] Using gulpfile /code/gulpfile.js
[13:56:08] Starting 'elm'...
events.js:160
throw er; // Unhandled 'error' event
^
Error: spawn /code/node_modules/elm/Elm-Platform/0.17.1/.cabal-sandbox/bin/elm-make ENOENT
at exports._errnoException (util.js:1026:11)
at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
at onErrorNT (internal/child_process.js:359:16)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)
at Module.runMain (module.js:606:11)
at run (bootstrap_node.js:394:7)
at startup (bootstrap_node.js:149:9)
at bootstrap_node.js:509:3
Compile error
/node_modules/q/q.js:155
throw e;
^
Error: Compile error
npm info lifecycle @~assets: Failed to exec assets script
npm ERR! Linux 4.4.0-47-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "assets"
npm ERR! node v6.9.1
npm ERR! npm v3.10.8
npm ERR! code ELIFECYCLE
npm ERR! @ assets: `gulp elm`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ assets script 'gulp elm'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! gulp elm
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /code/npm-debug.log |
|
Hi @gwmoura. I just tested with a FROM node:6.9.1
COPY elm-package.json elm-package.json
COPY package.json package.json
RUN npm install -g gulp
RUN npm install
COPY . /code
WORKDIR /code
VOLUME /code
CMD npm run assetsIt raised no error and created the expected $ docker run jarbas ls jarbas/frontend/static
app.js
ceap-datasets.html
digitalocean.pngI ran it in the |
|
@cuducos your Dockerfile not work for me :(, before build the container you deleted the file app.js? $ cat Dockerfile-node
FROM node:6.9.1
COPY elm-package.json elm-package.json
COPY package.json package.json
RUN npm install -g gulp
RUN npm install
COPY . /code
WORKDIR /code
VOLUME /code
CMD npm run assets
george@george-ubuntu:~/workspace/datasciencebr/jarbas$ docker build -t nodejarbas -f Dockerfile-node .
Sending build context to Docker daemon 365.6 MB
Step 1 : FROM node:6.9.1
---> 130ebee906ca
Step 2 : COPY elm-package.json elm-package.json
---> Using cache
---> d336bf466a4a
Step 3 : COPY package.json package.json
---> Using cache
---> ef4015364789
Step 4 : RUN npm install -g gulp
---> Using cache
---> 4670f4553021
Step 5 : RUN npm install
---> Using cache
---> ca31fc97fee5
Step 6 : COPY . /code
---> Using cache
---> 9a61c46c5b97
Step 7 : WORKDIR /code
---> Using cache
---> dc25b7a5fd37
Step 8 : VOLUME /code
---> Using cache
---> 126c30a69517
Step 9 : CMD npm run assets
---> Using cache
---> bbdc5dd4234f
Successfully built bbdc5dd4234f
george@george-ubuntu:~/workspace/datasciencebr/jarbas$ docker run --rm nodejarbas ls jarbas/frontend/static
app.css
digitalocean.png
george@george-ubuntu:~/workspace/datasciencebr/jarbas$ docker run --rm nodejarbas
npm info it worked if it ends with ok
npm info using npm@3.10.8
npm info using node@v6.9.1
npm info lifecycle @~preassets: @
npm info lifecycle @~assets: @
> @ assets /code
> gulp elm
[14:47:01] Using gulpfile /code/gulpfile.js
[14:47:01] Starting 'elm'...
events.js:160
throw er; // Unhandled 'error' event
^
Error: spawn /code/node_modules/elm/Elm-Platform/0.17.1/.cabal-sandbox/bin/elm-make ENOENT
at exports._errnoException (util.js:1026:11)
at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
at onErrorNT (internal/child_process.js:359:16)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)
at Module.runMain (module.js:606:11)
at run (bootstrap_node.js:394:7)
at startup (bootstrap_node.js:149:9)
at bootstrap_node.js:509:3
Compile error
/node_modules/q/q.js:155
throw e;
^
Error: Compile error
npm info lifecycle @~assets: Failed to exec assets script
npm ERR! Linux 4.4.0-47-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "assets"
npm ERR! node v6.9.1
npm ERR! npm v3.10.8
npm ERR! code ELIFECYCLE
npm ERR! @ assets: `gulp elm`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ assets script 'gulp elm'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! gulp elm
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /code/npm-debug.log |
|
Yep, you're right: I haven't deleted the
Good that it fixed, @gwmoura , but it shouldn't be that way. I have to explore the issue deeper I guess. Here are my concerns:
Maybe a solution: When we install Elm locally I'll try to find a workaround for that. |
|
I created 2 docker image, one for python and another onde to node (It is hard/right? way, but let's try it) https://github.com/gomex/jarbas To reproduce, use these commands: The first command will build the assets But the page is still blank and I don't know why :( |
|
@gomex the route Guys, I rewrote the commit of the @gomex and adapted to stay more easy to develop. My changes are here: gwmoura@c30fb06 To test follow this: docker-compose run --rm elm # this command compile the elm and generate the app.js
docker-compose run --rm web python manage.py migrate # runs the migrations
docker-compose up web # lifts up the api app listening on 8001 portAccessing |
|
@gwmoura why did you removed postgres installation? I guess it is needed. In the elm image, "RUN mkdir /code" it is not necessary. If you don't copy the code to elm image on the build, you should just run the npm -i after start the container, because node_modules folder won't present on container, it is because when you mount on /code folder, that content created by image on that folder will be overwritten on that container. |
|
@cuducos I tested here... Still blank :( |
|
|
@gomex and I just paired for a while. The code in TODO:
Pipeline:
|
|
@gomex @cuducos, I has some updates (gwmoura@449afa4), based in what @gomex talked. On my commits has:
|
|
I made some changes and applied on that PR: https://github.com/datasciencebr/jarbas/pull/48/files |
|
Many thanks @gomes and @gwmoura! Great PR! I merged it to
I ran into this issue here, not sure why — ImportError at /
No module named 'jarbas.frontend.views'
Request Method: GET
Request URL: http://localhost/
Django Version: 1.10.3
Exception Type: ImportError
Exception Value:
No module named 'jarbas.frontend.views'
Exception Location: /code/jarbas/urls.py in <module>, line 21
…Or, from Internal Server Error: /
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 39, in inner
response = get_response(request)
File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 172, in _get_response
resolver_match = resolver.resolve(request.path_info)
File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 270, in resolve
for pattern in self.url_patterns:
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 35, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 313, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 35, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 306, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 665, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/code/jarbas/urls.py", line 21, in <module>
from jarbas.frontend.views import home
ImportError: No module named 'jarbas.frontend.views'This is bizarre, because of three reasons: Reason 1 If I import it through the container, it works: $ docker run -it --rm jarbas_jarbas python
Python 3.5.2 (default, Nov 10 2016, 08:25:20)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jarbas.frontend.views import home
>>>Reason 2 The tests pass (and yes, this endpoint is tested): docker run -it --rm jarbas_jarbas python manage.py test
Creating test database for alias 'default'...
.........................................................
----------------------------------------------------------------------
Ran 57 tests in 0.385s
OK
Destroying test database for alias 'default'...Reason 3 Running the same code locally (not through Docker also works). Any ideas what did I got wrong here? AFAIK I deleted all previous Docker images and containers to start form scratch ( |
|
@cuducos how do you ran? You ran using
We can configure the
The Makefile is a shortcut to docker-compose commands, Windows users can run docker-compose commands or maybe we can write a @gomex What do you think? |
|
Hey @cuducos Can you please tell me how can I reproduce that? Did you use make or just docker-compose? |
We copy entire code from your folder, your .env file will be transfered to docker image too.
You are right. We can remove Makefile and just user docker-compose commands :(
I already fixed this problem: #49 |
|
Many thanks, @gomex! Got it about teh envvars, tks. I'm testing the PR and probably gonna merge it soon. About the |
|
I'm closing this issue since it was an open ended discussion (Docker? Travis? Ansible? Capistrano? Fabric?) and now that we have Docker stuff on master we can focus on automating deploy and provision with Docker (just opened a fresh new issue for that). Many many thanks for all of you that coded to get us further — specially @gomex @gwmoura @luiz-simples @pedrommone ; ) |
The Digital Ocean droplet was set manually and the only piece of software that makes deploy easier is a git hook. Yet,
nginxandgunicornprocesses are owned byroot— that's not cool.Something like Ansible might help.
Personally I don't have devops know-how to fix the users and processes issues, and I've never used Ansible — so any help is appreciated. Surely I can share the application specific logic and help anyone willing to close that issue ; )
The text was updated successfully, but these errors were encountered: