Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion - Guide to deploy an Elixir/Phoenix app to AWS ECS #20

Open
joaquimadraz opened this issue Feb 16, 2018 · 25 comments
Open

Discussion - Guide to deploy an Elixir/Phoenix app to AWS ECS #20

joaquimadraz opened this issue Feb 16, 2018 · 25 comments
Labels

Comments

@joaquimadraz
Copy link
Owner

@joaquimadraz joaquimadraz commented Feb 16, 2018

I'm opening this issue for a possible discussion of my blog post:
https://joaquimadraz.com/guide-to-deploy-an-elixir-phoenix-app-to-aws-ecs

@tim2CF
Copy link

@tim2CF tim2CF commented Feb 28, 2018

Hello! Thanks for article.
I have some questions

  1. What's a point to create Distillery releases when you have full-operational Erlang and Elixir inside Docker image (that is used to distribute program)?
  2. How distributed Erlang works in case of using of AWS ECS? I mean it seems we need EPMD (Erlang Port Mapper Daemon) configuration and some other magical stuff that will create Erlang nodes cluster from our EC2 instances.
  3. How you update your apps in runtime? Any examples of Erlang hot code reloading for EC2 instance?
@joaquimadraz
Copy link
Owner Author

@joaquimadraz joaquimadraz commented Feb 28, 2018

Hey @tim2CF, thanks for the feedback.

The Dockerfile is split into two stages, build and release. The build stage uses Distillery to compile the application into Erlang binaries which also includes the Erlang Runtime (I forgot to mention this detail in the article: rel/config.exs#L37). Then release starts with plain Alpine Linux. Nothing from the build phase (meaning Erlang, Elixir, Phoenix, Mix, Node) is carried. There's no Erlang in there until we copy the compiled files into the release image: Dockerfile#L102.

In the end, the release image just has the Erlang runtime and our application binaries.

I still didn't dive into distributed Erlang on AWS. I totally going to use this project to dive into it.
Same with hot code reloading. I'll keep you posted.

@tim2CF
Copy link

@tim2CF tim2CF commented Feb 28, 2018

Thanks!

@Merff
Copy link

@Merff Merff commented Mar 13, 2018

@joaquimadraz Thanks for the guide!
I am currently on step docker run --rm -it -p 4002:4002 -e PORT=4002 subs:latest

docker build -t subs:latest \
  --build-arg PHOENIX_SECRET_KEY_BASE=super_secret_phoenix_key_base \ ...

This command works without any errors. But docker run --rm -it -p 4002:4002 -e PORT=4002 subs:latest have error:

Postgrex.Protocol (#PID<0.113.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (localhost:5432): connection refused - :econnrefused
message=><<"connection not available because of disconnection">>

I am new in Docker, like that docker can not connect to the Postgres.
May be you know how to fix it?

@joaquimadraz
Copy link
Owner Author

@joaquimadraz joaquimadraz commented Mar 13, 2018

Hey @Merff. Yes, what's probably happening is that docker is not able to connect to the host's Postgres.
Try using @docker.for.mac.host.internal instead of @localhost on the DATABASE_URL:

DATABASE_URL=postgresql://postgres:postgres@docker.for.mac.host.internal/subs_prod

Depending on your Docker version, you might need to use @docker.for.mac.host.internal instead of @ localhost for the database url so that the container can access the host’s Postgres. Read more about it here.

There's a note regarding that issue on the guide. I guess it's not properly placed :)

@Merff
Copy link

@Merff Merff commented Mar 14, 2018

Hi @joaquimadraz. Thanks for the answer) But I don't use Mac, I use Ubuntu. And my project not an Umbrella, just standard Phoenix 1.3 project. How to correct change Dockerfile and may be other files? Thank you!)

@Merff
Copy link

@Merff Merff commented Mar 30, 2018

Hi! I get "no basic auth credentials" while

docker push "$AWS_ECS_URL"/"$AWS_ECS_DOCKER_IMAGE"

Login works fine with

eval $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
@markdev
Copy link

@markdev markdev commented Apr 4, 2018

Hi @joaquimadraz ,

I'm stuck on this step:

$ PORT=4001 _build/prod/rel/subs/bin/subs foreground
>>> Elixir.Subs.Tasks.ReleaseTasks.setup is either not defined or has a non-zero arity

I have module ReleaseTasks in apps/subs/priv/tasks/release_tasks.ex, under defmodule Elixir.Subs.Tasks.ReleaseTasks, with def setup do .... Any idea what could be causing this?

@joaquimadraz
Copy link
Owner Author

@joaquimadraz joaquimadraz commented Apr 4, 2018

@Merff sorry for the late reply. Did you figure out what was the problem with the database connection?

Regarding the credentials problem, I'm not sure what can it be but I would say that it's a credentials problem. Make sure that the correct AWS keys are being used. Start by looking at ~/.aws/credentials default credentials.

@joaquimadraz
Copy link
Owner Author

@joaquimadraz joaquimadraz commented Apr 4, 2018

@markdev Did you add the priv/tasks to the elixir compilation paths?
https://github.com/joaquimadraz/opensubs.io/blob/deployment/apps/subs/mix.exs#L32

@sean-clayton
Copy link

@sean-clayton sean-clayton commented Apr 13, 2018

UPDATE

I got it all working!!


So I followed the steps and everything worked out in the end except... my app doesn't load. I'm completely new to AWS/any kind of devops stuff and am just an application developer.

Here is my project URL: https://gitlab.com/project-bourbon/mash

and here is my public url for the project: http://ec2-54-165-95-3.compute-1.amazonaws.com/

If there's anything glaringly obvious it'd be nice to know. I don't even know how to look at logs or anything :\

Here is my last deploy log as well: https://gitlab.com/project-bourbon/mash/-/jobs/62738365

Locally I can build production and provide the environment variables for my local db and it all works perfectly running from a docker container, so I know it's not that.

@joaquimadraz
Copy link
Owner Author

@joaquimadraz joaquimadraz commented Apr 13, 2018

@sean-clayton I'm glad that you were able to do it!

I was checking you project's source code it has something to do with the connection with the database. Do you want to explain a bit the problem you faced?

Regarding the logs, what I do for now is accessing the EC2 instance, grab the docker container logs.

  1. ssh to the EC2 instance
  2. run docker ps to check the containers that are running
CONTAINER ID IMAGE COMMAND ... PORTS
YourContainerID subs_web:latest subs_web/bin/subs... ... tcp, 0.0.0.0:443->443 ...
  1. Grab the container id (YourContainerID) and run docker logs YourContainerID
05:16:51.223 request_id=ar2f8mj2kg6se42lo5sabmgfl0cn6gi5 [info] Sent 200 in 168µs
05:59:35.608 request_id=mqgv4mqcvfgql16jgv9prnaptjce98e8 [info] GET /
05:59:35.608 request_id=mqgv4mqcvfgql16jgv9prnaptjce98e8 [info] Sent 200 in 173µs
06:35:06.920 request_id=5ge7814chchnn91qvvnpuhn75opulo8a [info] HEAD /
06:35:06.920 request_id=5ge7814chchnn91qvvnpuhn75opulo8a [info] Sent 200 in 144µs

This is not ideal but it's a good starting point to check if something went wrong on boot.

I'm working on logging everything to AWS CloudWatch. Would that be interesting for you?

@sean-clayton
Copy link

@sean-clayton sean-clayton commented Apr 13, 2018

@joaquimadraz Yup, I dug a little bit and forgot that I had ssh access to my EC2 instance 😅 Quickly got access to logs after that. And about CloudWatch: That would definitely interest me. I'm kinda surprised there isn't a step on that in the wizard.

And about the problem I had... I'm not 100% sure why it started working. I think it was because I wasn't passing the DATABASE_URL var as an environment variable so Phoenix couldn't see it. Not 100% sure about that since I wasn't able to look at logs when that was occurring. I then passed down my DB settings in the docker-compose.yml file and it seemed to work after that. Normally, Phoenix would still start even if no DB connection could be made (meaning I could still see my routes and stuff, it would just fail on Repo things), but for me it wasn't even doing that. Then I remembered that I'm running migrations in the before script so I think that those kept failing and preventing Phoenix from even booting up, but these are all just hunches and suspicions.

My next steps that I'm going to try learning (as I said, completely new to this side of development):

  • See if WebSockets work and if they don't, try to fix it
  • Figure out how to SSL in AWS. Lots of acronyms now... (ACM, EC2, ECS, ELB, etc)
  • Stick nginx in front of my ecs cluster to hold onto the certificate

Are these things "good practice" with AWS? Namely wondering about nginx there. I have no idea what the norm is on this field.


Update

Got an elastic load balancer with SSL support working :D

@blueangel0723
Copy link

@blueangel0723 blueangel0723 commented Apr 18, 2018

@joaquimadraz I have the trouble to deploy the Elixir to AWS. Can you help me, please?

@KalvinHom
Copy link

@KalvinHom KalvinHom commented Oct 3, 2018

@joaquimadraz Do you have any thoughts on adjusting the deploy.sh to not have the slight downtime between the the old service going down and the new one finishing starting up?

@joaquimadraz
Copy link
Owner Author

@joaquimadraz joaquimadraz commented Oct 4, 2018

Hey @KalvinHom. I didn’t have a change to play with it but you can accomplish that using AWS ALB (Application Load Balancer).
You can have an ALB pointing to a ECS cluster which allows to do blue/green deployments. After starting the new containers, the ALB will drain the connections from the old containers until all connections are for the new containers.
This is as far as I can help you. Haven’t set it up myself.

@stupidusernametoreportissues
Copy link

@stupidusernametoreportissues stupidusernametoreportissues commented Nov 27, 2018

Some minor issues.

What you'll need

  • docker
  • jq
  • awscli
  • ecs-cli
  1. git clone git@github.com:joaquimadraz/opensubs.io.git
  2. cd subs && git checkout 91c25e4
    clone creates directory opensubs.io not subs

Now in order for the files under priv/tasks to be compiled and available on the pre_hook_script we need to add the path to elixirc_paths on mix.exs:
could add more context such as apps/subs/mix.exs since previously edited root mix.exs

Great job with the commits on deployment branch. Made following along slight ambiguities like this much easier

just noticed oddness
https://github.com/joaquimadraz/opensubs.io/blob/deployment/config/deploy/shipit.sh
has
--region $AWS_ECS_REGION
which you corrected elsewhere

@srikanthkyatham
Copy link

@srikanthkyatham srikanthkyatham commented May 26, 2019

Hello,

Great article, seems pre_start_hook has been deprecated in favour of pre_start_hooks.
Link to changes https://hexdocs.pm/distillery/changelog.html#deprecations

In your blog post
set pre_start_hook: "rel/hooks/pre_start.sh" changes to
set pre_start_hooks: "rel/hooks"

Hope the future readers benefit from this.

@srikanthkyatham
Copy link

@srikanthkyatham srikanthkyatham commented May 26, 2019

Also one other change in the release_tasks.ex

{:ok, _ } = @repo.start_link(pool_size: 1) ->     {:ok, _ } = @repo.start_link(pool_size: 2) // minimum pool size requirement is 2, since ecto 3

Reference
http://blog.tap349.com/elixir/ecto/2018/11/15/ecto-upgrading-to-3.0/#increase-connection-pool-size-to-at-least-2

@siyomai
Copy link

@siyomai siyomai commented Jul 10, 2019

Hi @joaquimadraz,
What would be are changes if we are to use the latest version of distillery? Still figuring things out as far as AWS deployment is concerned. Thanks.

@arjun289
Copy link

@arjun289 arjun289 commented Sep 27, 2019

@siyomai you can consider using the latest version of elixir 1.9.1 which has a releases feature now.

@arjun289
Copy link

@arjun289 arjun289 commented Sep 27, 2019

@sean-clayton were you able to setup nginx with ssl?

@muslimecoin
Copy link

@muslimecoin muslimecoin commented Dec 18, 2019

how i can do it with multiple images in this project https://github.com/aviacommerce/avia can you give me a explanation on how to deploy it on aws

@chaitanyapi
Copy link

@chaitanyapi chaitanyapi commented Jun 5, 2020

Using all the steps followed, i am now able to start my app in EC2 which is having an RDS database. the phoenix endpoint is now my EC2 instance running on 4000 port. I have linked ports 4000 in my docker-compose file and then exposed 4000 port from my container. but still i am not able to connect to the application.
I have provided all IP access in the SG. however not sure what am i really missing.
any suggestion is highly appreciated.
Thank you.

@brundozer
Copy link

@brundozer brundozer commented Jul 6, 2020

@chaitanyapi I've just run in the same issue. Adding the port 4000 for incoming traffic on my ECS Allowed Ports security group fixed the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet