New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion - Guide to deploy an Elixir/Phoenix app to AWS ECS #20

Open
joaquimadraz opened this Issue Feb 16, 2018 · 17 comments

Comments

Projects
None yet
8 participants
@joaquimadraz
Owner

joaquimadraz commented Feb 16, 2018

I'm opening this issue for a possible discussion of my blog post:
https://joaquimadraz.com/guide-to-deploy-an-elixir-phoenix-app-to-aws-ecs

@tim2CF

This comment has been minimized.

tim2CF commented Feb 28, 2018

Hello! Thanks for article.
I have some questions

  1. What's a point to create Distillery releases when you have full-operational Erlang and Elixir inside Docker image (that is used to distribute program)?
  2. How distributed Erlang works in case of using of AWS ECS? I mean it seems we need EPMD (Erlang Port Mapper Daemon) configuration and some other magical stuff that will create Erlang nodes cluster from our EC2 instances.
  3. How you update your apps in runtime? Any examples of Erlang hot code reloading for EC2 instance?
@joaquimadraz

This comment has been minimized.

Owner

joaquimadraz commented Feb 28, 2018

Hey @tim2CF, thanks for the feedback.

The Dockerfile is split into two stages, build and release. The build stage uses Distillery to compile the application into Erlang binaries which also includes the Erlang Runtime (I forgot to mention this detail in the article: rel/config.exs#L37). Then release starts with plain Alpine Linux. Nothing from the build phase (meaning Erlang, Elixir, Phoenix, Mix, Node) is carried. There's no Erlang in there until we copy the compiled files into the release image: Dockerfile#L102.

In the end, the release image just has the Erlang runtime and our application binaries.

I still didn't dive into distributed Erlang on AWS. I totally going to use this project to dive into it.
Same with hot code reloading. I'll keep you posted.

@tim2CF

This comment has been minimized.

tim2CF commented Feb 28, 2018

Thanks!

@Merff

This comment has been minimized.

Merff commented Mar 13, 2018

@joaquimadraz Thanks for the guide!
I am currently on step docker run --rm -it -p 4002:4002 -e PORT=4002 subs:latest

docker build -t subs:latest \
  --build-arg PHOENIX_SECRET_KEY_BASE=super_secret_phoenix_key_base \ ...

This command works without any errors. But docker run --rm -it -p 4002:4002 -e PORT=4002 subs:latest have error:

Postgrex.Protocol (#PID<0.113.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (localhost:5432): connection refused - :econnrefused
message=><<"connection not available because of disconnection">>

I am new in Docker, like that docker can not connect to the Postgres.
May be you know how to fix it?

@joaquimadraz

This comment has been minimized.

Owner

joaquimadraz commented Mar 13, 2018

Hey @Merff. Yes, what's probably happening is that docker is not able to connect to the host's Postgres.
Try using @docker.for.mac.host.internal instead of @localhost on the DATABASE_URL:

DATABASE_URL=postgresql://postgres:postgres@docker.for.mac.host.internal/subs_prod

Depending on your Docker version, you might need to use @docker.for.mac.host.internal instead of @ localhost for the database url so that the container can access the host’s Postgres. Read more about it here.

There's a note regarding that issue on the guide. I guess it's not properly placed :)

@Merff

This comment has been minimized.

Merff commented Mar 14, 2018

Hi @joaquimadraz. Thanks for the answer) But I don't use Mac, I use Ubuntu. And my project not an Umbrella, just standard Phoenix 1.3 project. How to correct change Dockerfile and may be other files? Thank you!)

@Merff

This comment has been minimized.

Merff commented Mar 30, 2018

Hi! I get "no basic auth credentials" while

docker push "$AWS_ECS_URL"/"$AWS_ECS_DOCKER_IMAGE"

Login works fine with

eval $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
@markdev

This comment has been minimized.

markdev commented Apr 4, 2018

Hi @joaquimadraz ,

I'm stuck on this step:

$ PORT=4001 _build/prod/rel/subs/bin/subs foreground
>>> Elixir.Subs.Tasks.ReleaseTasks.setup is either not defined or has a non-zero arity

I have module ReleaseTasks in apps/subs/priv/tasks/release_tasks.ex, under defmodule Elixir.Subs.Tasks.ReleaseTasks, with def setup do .... Any idea what could be causing this?

@joaquimadraz

This comment has been minimized.

Owner

joaquimadraz commented Apr 4, 2018

@Merff sorry for the late reply. Did you figure out what was the problem with the database connection?

Regarding the credentials problem, I'm not sure what can it be but I would say that it's a credentials problem. Make sure that the correct AWS keys are being used. Start by looking at ~/.aws/credentials default credentials.

@joaquimadraz

This comment has been minimized.

Owner

joaquimadraz commented Apr 4, 2018

@markdev Did you add the priv/tasks to the elixir compilation paths?
https://github.com/joaquimadraz/opensubs.io/blob/deployment/apps/subs/mix.exs#L32

@sean-clayton

This comment has been minimized.

sean-clayton commented Apr 13, 2018

UPDATE

I got it all working!!


So I followed the steps and everything worked out in the end except... my app doesn't load. I'm completely new to AWS/any kind of devops stuff and am just an application developer.

Here is my project URL: https://gitlab.com/project-bourbon/mash

and here is my public url for the project: http://ec2-54-165-95-3.compute-1.amazonaws.com/

If there's anything glaringly obvious it'd be nice to know. I don't even know how to look at logs or anything :\

Here is my last deploy log as well: https://gitlab.com/project-bourbon/mash/-/jobs/62738365

Locally I can build production and provide the environment variables for my local db and it all works perfectly running from a docker container, so I know it's not that.

@joaquimadraz

This comment has been minimized.

Owner

joaquimadraz commented Apr 13, 2018

@sean-clayton I'm glad that you were able to do it!

I was checking you project's source code it has something to do with the connection with the database. Do you want to explain a bit the problem you faced?

Regarding the logs, what I do for now is accessing the EC2 instance, grab the docker container logs.

  1. ssh to the EC2 instance
  2. run docker ps to check the containers that are running
CONTAINER ID IMAGE COMMAND ... PORTS
YourContainerID subs_web:latest subs_web/bin/subs... ... tcp, 0.0.0.0:443->443 ...
  1. Grab the container id (YourContainerID) and run docker logs YourContainerID
05:16:51.223 request_id=ar2f8mj2kg6se42lo5sabmgfl0cn6gi5 [info] Sent 200 in 168µs
05:59:35.608 request_id=mqgv4mqcvfgql16jgv9prnaptjce98e8 [info] GET /
05:59:35.608 request_id=mqgv4mqcvfgql16jgv9prnaptjce98e8 [info] Sent 200 in 173µs
06:35:06.920 request_id=5ge7814chchnn91qvvnpuhn75opulo8a [info] HEAD /
06:35:06.920 request_id=5ge7814chchnn91qvvnpuhn75opulo8a [info] Sent 200 in 144µs

This is not ideal but it's a good starting point to check if something went wrong on boot.

I'm working on logging everything to AWS CloudWatch. Would that be interesting for you?

@sean-clayton

This comment has been minimized.

sean-clayton commented Apr 13, 2018

@joaquimadraz Yup, I dug a little bit and forgot that I had ssh access to my EC2 instance 😅 Quickly got access to logs after that. And about CloudWatch: That would definitely interest me. I'm kinda surprised there isn't a step on that in the wizard.

And about the problem I had... I'm not 100% sure why it started working. I think it was because I wasn't passing the DATABASE_URL var as an environment variable so Phoenix couldn't see it. Not 100% sure about that since I wasn't able to look at logs when that was occurring. I then passed down my DB settings in the docker-compose.yml file and it seemed to work after that. Normally, Phoenix would still start even if no DB connection could be made (meaning I could still see my routes and stuff, it would just fail on Repo things), but for me it wasn't even doing that. Then I remembered that I'm running migrations in the before script so I think that those kept failing and preventing Phoenix from even booting up, but these are all just hunches and suspicions.

My next steps that I'm going to try learning (as I said, completely new to this side of development):

  • See if WebSockets work and if they don't, try to fix it
  • Figure out how to SSL in AWS. Lots of acronyms now... (ACM, EC2, ECS, ELB, etc)
  • Stick nginx in front of my ecs cluster to hold onto the certificate

Are these things "good practice" with AWS? Namely wondering about nginx there. I have no idea what the norm is on this field.


Update

Got an elastic load balancer with SSL support working :D

@blueangel0723

This comment has been minimized.

blueangel0723 commented Apr 18, 2018

@joaquimadraz I have the trouble to deploy the Elixir to AWS. Can you help me, please?

@KalvinHom

This comment has been minimized.

KalvinHom commented Oct 3, 2018

@joaquimadraz Do you have any thoughts on adjusting the deploy.sh to not have the slight downtime between the the old service going down and the new one finishing starting up?

@joaquimadraz

This comment has been minimized.

Owner

joaquimadraz commented Oct 4, 2018

Hey @KalvinHom. I didn’t have a change to play with it but you can accomplish that using AWS ALB (Application Load Balancer).
You can have an ALB pointing to a ECS cluster which allows to do blue/green deployments. After starting the new containers, the ALB will drain the connections from the old containers until all connections are for the new containers.
This is as far as I can help you. Haven’t set it up myself.

@stupidusernametoreportissues

This comment has been minimized.

stupidusernametoreportissues commented Nov 27, 2018

Some minor issues.

What you'll need

  • docker
  • jq
  • awscli
  • ecs-cli
  1. git clone git@github.com:joaquimadraz/opensubs.io.git
  2. cd subs && git checkout 91c25e4
    clone creates directory opensubs.io not subs

Now in order for the files under priv/tasks to be compiled and available on the pre_hook_script we need to add the path to elixirc_paths on mix.exs:
could add more context such as apps/subs/mix.exs since previously edited root mix.exs

Great job with the commits on deployment branch. Made following along slight ambiguities like this much easier

just noticed oddness
https://github.com/joaquimadraz/opensubs.io/blob/deployment/config/deploy/shipit.sh
has
--region $AWS_ECS_REGION
which you corrected elsewhere

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment