Enhanced Procfile support? empire.yml?

[ejholmes]

We've talked about this a little bit. Opening this up to keep a record of thoughts.

It would be nice to support a Procfile "upgrade" as an empire.yml file. This would potentially be formatted more closely to something like docker-compose.yml and allow for configuring things like:

1. Whether a dyno should get a load balancer attached.
2. Whether the load balancer should be exposed publicly or privately.
3. Healthcheck configurations.

An example of this format might look like this:

web:
  command: ./bin/web
  load_balancer:
    expose: external
    checks:
      - protocol: http
        path: /health
api:
  command: ./bin/api
  load_balancer:
    expose: internal
    checks:
      - protocol: tcp
worker:
  command: ./bin/worker

This would come with support for attaching a load balancer to any process.

[ejholmes]

It would be nice if, along with this, we support sidekiq containers, which ECS and Kubernetes support. A potential use case for this would be to have a sidekiq nginx container for the web process.

[public]

Any plans to support keeping this kind of configuration outside of the Docker image? Having to download many MB of Docker image just to extract a small text file makes the experience feel very inefficient sometimes :(

[ejholmes]

There's a couple of reasons we opted for extracting a Procfile:

1. It matches more closely to what Heroku does. Everything we have already has a Procfile defined.
2. It works properly with branches. Say you have a branch a that changes the command for the web process. If you swap between branch a and b, you don't have to change any configuration manually. Less room for error.

True, it is a little slower to deploy, but it hasn't been a major pain point for us yet. If we could support both, that would definitely be worth considering. Maybe if no Procfile is defined, we fallback to manual configuration?

[public]

Keeping the deployment configuration in sync with the repo sounds plausibly useful. One approach to this might be to copy docker-compose a bit like you described above expect with the addition of an image key that defines the image and tag to deploy?

[ejholmes]

Interesting idea, that could be a really nice alternative to extracting a Procfile. Right now, the /deploys endpoint takes a simple POST payload with the image to deploy:

{
  "image": "remind101/acme-inc:tag"
}

We could optionally extend this to allow the posting of something like a docker-compose.yml and it could use that to create the appropriate processes. The deployment tooling (tugboat in our case), could handle pulling out an extended Procfile using the GitHub api, which would be a lot faster than pulling a docker image.

Empire is still somewhat constrained to the idea of having a 1:1 relationship between an app and a docker image, so this will touch a lot of things, but sounds like a good idea.

I'll mull this over some more.

[public]

That 1:1 relationship is something that also makes Empire difficult for us to use for things like blue/green deployments right now so I suspect breaking that assumption might bring a lot more benefits :)

[corentone]

Hello everyone!
I would also be really interested in that feature!

I actually like the docker-compose support, mostly for its ability to select port and processes (I have a use case for more than one exposed process for a single image). It also seems more natural than extracting a Procfile from the image and run it :)

Is there any work started on this? Could I help?

[gileze33]

Would it be possible to deploy an app without an attached ELB with these changes?

We're using empire in production and loving it, but we're about to start moving our queue processors out of each service's API into a few new apps (they use rabbitmq) and these really don't need any ELBs (which carry a fair bit of monthly cost).

[phobologic]

@gileze33 - you should be able to do that currently. If your app Procfile doesn't specify any web processes, it shouldn't create an ELB. If this isn't the case, then I'd consider that a bug.

That said, I don't think that Empire removes the ELB if there is a web process in the Procfile, but the workers have been scaled to 0.

If you're re-using the same project with different app names (one for your web workers, one for your queue processors- I'm not sure I understand the use case here, so if this is the case I'd be curious why you are choosing to do that) but they all share the same Procfile, then it'll create an ELB currently.

Could that show up in the extended Procfile? Sure. We're still working on what we expect this might do. Hoping to have the first version of it written this quarter.

[ejholmes]

@gileze33 right, as @phobologic said the best way to do this is to just name the processes anything other than web. e.g.:

worker: ./bin/worker

If you've already deployed the app with a web process and want to remove the ELB, you can just emp destroy <app> then re-deploy it with the updated Procfile.

[corentone]

I just wanted to contribute to the proposed YAML with a couple extra points that could be helpful:

web:
  command: ./bin/web
  ports:
    - name: TCP_PORT #used for the ENVVAR
      clear_port: 65001
      ssl_port: 65002
    - name: HTTP_PORT #used for the ENVVAR
      clear_port: 80
      ssl_port: 443
  load_balancer:
    expose: external
    # the port there would be a portName
    port: TCP_PORT
    protocol: tcp
    checks:
      - protocol: http
        port: HTTP_PORT
        path: /health
api:
  command: ./bin/api
  load_balancer:
    expose: internal
    checks:
      - protocol: tcp
worker:
  command: ./bin/worker

First, allowing TCP and not only HTTP. This requires to allow customization of the user visible port (1024 and above) for both plain and ssl.
Second thing would be to allow to expose multiple ports, so the ELB health check could be performed on the second port (HTTP?). It seems nice to perform a health check on the HTTP port of our app. Multiple ports would also allow multiple ELBs for the day AWS supports multiple ELBs per service.
Another thing that could be interesting is volume (not shown in my example). I'm not sure exactly how that would work but I could see someone willing to access more capabilities of ECS through the ease of empire.

[gileze33]

@ejholmes That's great news about the existing Procfile / no ELB support, I'll give that a go this week.

@corentone It looks like you're also looking to add references within the YAML - would it be worth prefixing these like $HTTP_PORT when you use a reference to make it explicit?

[corentone]

@gileze33 Good idea. Actually, we could use YAML anchors instead of a string prefix. That way we only have one spot in the code with the port name. Prefixing would work too, but still requires two locations and a specific type of "language".

We could pass either the entire struct or the port name only depending how we implement it.

web:
  command: ./bin/web
  ports:
    - &tcp_port
      name: TCP_PORT #used for the ENVVAR
      clear_port: 65001
      ssl_port: 65002
    - &http_port
      name: HTTP_PORT #used for the ENVVAR
      clear_port: 80
      ssl_port: 443
  load_balancer:
    expose: external
    # the port there would be a portName
    port: << *tcp_port
    protocol: tcp
    checks:
      - protocol: http
        port: << *http_port
        path: /health

or

web:
  command: ./bin/web
  ports:
    - name: &tcp_port TCP_PORT 
      clear_port: 65001
      ssl_port: 65002
    - name: &http_port HTTP_PORT 
      clear_port: 80
      ssl_port: 443
  load_balancer:
    expose: external
    # the port there would be a portName
    port: *tcp_port
    protocol: tcp
    checks:
      - protocol: http
        port: *http_port
        path: /health

[mwildehahn]

I came across a use case where i want to expose multpile ports (which I believe fits with the Procfile defined by @corentone: http://docs.locust.io/en/latest/running-locust-distributed.html).

tldr; to run distributed load tests i need to run a master with port 8089 open for a web interface and TCP ports 5557 & 5557 + 1 (5557 is customizable) for the minion processes to connect to.

[ejholmes]

@mhahn there's currently an ECS limitation where we can only attach a single ELB to an ECS service. We'll have support for attaching ELB's to any process (not just "web") soon, so you could just define a different process with the same command.

[ejholmes]

I think with the addition of ports to the extended Procfile, we can go ahead and close this. This will be in the 0.12.0 release of Empire. Yay!

Doc updates are in #1006

Defining health checks is not yet supported.