STORY: Migrate away from Skyliner

[nickstenning]

Context

Unfortunately, Skyliner is shutting down. While our infrastructure will keep running, our ability to deploy will go away on 15 July.

Story

We need to migrate our application configuration and deployment pipeline to a different platform. We should aim to do this by the end of June in order to give ourselves a couple of weeks of buffer before Skyliner shuts down.

Discussion

Skyliner currently provides the following for us:

1. automated builds of our application into a Docker image from the hypothesis/h git repository.
2. automated and manual deployments of the build artifact into a QA environment
3. manual promotion of the latest build deployed to QA into a production environment
4. secure storage of application configuration
5. audit logs of what was deployed when, including Slack notifications of deployments
6. visual representation of deploy diff (both code and configuration) at deploy time
7. manual configuration of capacity (by choice of number and type of EC2 instances)
8. a fast "blue-green" deployment, in which new revisions of the application are brought up all at once, healthchecked, added to a load balancer, and then the old revision removed.

We don't necessarily (and perhaps cannot) find an economical alternative which provides all of these things, but this is the baseline on which to evaluate possibilities.

There are a number of alternatives to Skyliner. Here are some, along with the pros and cons of each:

AWS CodeStar

CodeStar bills itself as a "unified user interface, enabling you to...set up your entire continuous delivery toolchain in minutes." It's a UI that ties together a whole panoply of preexisting AWS products, including:

• CodeCommit (VCS hosting)
• CodeBuild (build job service)
• CodePipeline (continuous delivery pipeline management)
• CodeDeploy (deployment to running instances)

as well as a bunch of other deployment options. There a blog post that does a reasonable job of explaining what a CodeStar project looks like.

• ➕ Has preexisting templates for a number of different deployment options.
• ➖ Seems to require code is hosted using CodeCommit. See also this support thread.
• ➖ Manages own IAM user users (we already have IAM users for our developers).
• ➖ Not available in our current region, us-west-1.

Convox

Convox is open-source software that makes it possible to run your own Heroku-like cloud in your AWS account. There's a short demo video that shows what using it is like for a developer.

You install a so-called "Rack" (i.e. PaaS infrastructure) into your AWS account, and then the convox CLI tool interacts with it to run applications on your private PaaS.

Convox uses a Docker Compose file to describe how to run your application, including backing services such as databases.

• ➕ Docker-based application description.
• ➕ Dev-prod parity made easier by allowing local execution of environment.
• ➕ Manages application configuration as environment variables (convox env).
• ➖ Coupling deployment environment to development tools: churn for developers as deployment environment changes.
• ➖ Either we pay $500/mo. for Convox to manage the Rack, or we manage it ourselves.
• ➖ Convox appears to use rolling releases, not blue-green deployment.
• ➖ We'd either work against Convox convention by continuing to use databases outside the PaaS, or we'd have to do some work to move databases into the PaaS.
• ➖ Convox appears to only support load balancing with classic ELBs, which would be a problem for our WebSocket service.

Heroku

Heroku is one of the oldest and most well established PaaS providers. They offer a git-based deployment workflow, and as of a few years ago, delivery pipelines.

• ➕ Effectively zero infrastructure management for us. No infrastructure to manage or think about for the application tier.
• ➕ Does support Docker-based deployments.
• ➕ Manages application configuration as environment variables.
• ➖ Heroku only have two public clouds, in us-east-1 and eu-west-1. That means we'd either have to migrate our database services, or accept a substantial latency penalty.
• ➖ Docker-based deployments are in beta.

AWS CodeDeploy

One of the pieces that is tied together by CodeStar into a more integrated product. It is essentially a software agent that runs on EC2 instances to coordinate code deployment, together with a web dashboard and set of APIs which can be used to orchestrate deployment.

• ➕ Supports a Skyliner-like blue-green deployment process in which new code revisions are deployed to brand new instances before the old instances are recycled.
• ➖ Doesn't manage application configuration.
• ➖ Application Load Balancers are not supported for blue-green deployment configurations.
• ➖ Seems to be heavily focused on file-based deployments which might pose some difficulties for Docker-image-based deployments.

AWS Elastic Beanstalk

EB is a PaaS-like offering from AWS. You give it a description of your application and it manages the resources to run it inside AWS. In this sense it's like Skyliner.

• ➕ Native support for deploying Docker containers.
• ➕ As of recently supports an "immutable" deployment mode that works vaguely like Skyliner's blue-green deployments (although it brings up a canary instance first, which may make things slower).
• ➕ Has support for true autoscaling environments (i.e. scale-up and scale-down based on network traffic, load, schedule, etc.) which may contribute to cost-savings given the periodic nature of our load.
• ➕ Supports ALBs, including for immutable deployments.
• ➕ Manages application configuration environment variables.
• ➖ Doesn't manage VPCs, subnets, or other infrastructure that Skyliner manages for us.
• ➖ Doesn't have a nice dashboard for managing deployments -- we'll need to cobble something together with Jenkins, probably.
• ➖ Doesn't appear to have native support for artifact "promotion" from one environment (QA) to another (prod). We will likely have to cobble this together ourselves.
• ➖ EB "worker" environments depend on SQS, so we won't be using them. This shouldn't be a problem as we already have a worker solution which runs in our Docker containers.

[nickstenning]

On the basis of my research so far, I have an opinion about where we should go from here, but I'm going to hold off on expressing that opinion until we get a chance to talk about all these options as a team.

[ddollar]

Hope I'm not out of line commenting here but I saw your issue linked in convox/rack. I can add a bit of detail about Convox:

• We are soon moving to a per-seat pricing model rather than the blanket price per month which should align a lot more closely with your team size and stage. Expect this in the next couple of months.

• You can easily use databases outside Convox, either by putting them in the same VPC or by using VPC peering to give access to a Rack. We have convox resources create for convenience but you can easily run your own databases manually or even create CloudFormation stacks that match our naming, parameter, and output conventions to have them show up in the convox CLI.

• Convox uses ELBs in TCP mode so Websockets are find. Convox itself uses Websockets to power convox exec and convox run.

I'd also encourage you to check out our next generation tools currently in beta: https://github.com/convox/praxis

[nickstenning]

Thanks David, that's all useful context to have. We'll certainly take it into consideration.

[mwcampbell]

Just a curious observer. Thanks for posting your notes here. Can you please comment here when you come to a decision?

[muralikrishnag10]

@nickstenning Sorry if this is not proper etiquette since it might come across as a sales pitch but
have you considered DuploCloud as an alternative?

https://portal.duplocloud.net
DuploCloud is currently used in large-scale production environments in companies like Zenefits, Automation Anywhere among others.

Here's a Demo: https://www.youtube.com/embed/rwWgrT2b0Bs

Skyliner seems very similar to DuploCloud. A key difference is that duplocloud starts with a tenant as a base underlying construct. DuploCloud also help create policies while integrating native AWS services access S3, DynamoDB...

You can do a free sign-up on the website if you like. It would be great if you could evaluate us as well before you make your decision.

best rgds
Murali

[duplodemo]

@nickstenning adding to Murali's comment. DuploCloud has an offering called DuploLive where you will be able to deploy apps on aws w/o even requiring an AWS account, u just need an email address. Thats the signup that will give you a quick comparison.

• Thanks,
  Venkat

[nickstenning]

A couple of people outside Hypothesis asked if I could update this issue once we made a decision and learnt about how we were going to replace Skyliner.

In the end, we've gone with a deployment pipeline based around AWS Elastic Beanstalk. I'll detail some of the main points that led us to this decision below, but I'll first note that other teams making decisions for a new system may well find that some of the other options listed above suit them better. We wanted to minimise risk by changing as little as possible about our production infrastructure and deployment pipeline while moving away from Skyliner. With Elastic Beanstalk:

• We could use a deployment model very similar to Skyliner's. Specifically, we run a single application Docker container on each EC2 instance, and we replace the entire environment with new EC2 instances on each deployment.

  EB supports several different models for deployments: various flavours of rolling updates, blue-green deployments where you swap out entire environments, and -- our choice -- "immutable" updates where new infrastructure is spun up for each deployment before the old infrastructure is thrown away.

  "Immutable" environment updates may be slower than other deployment options, particularly given that EB insists on spinning up a single instance to healthcheck before bringing up the rest, but they minimise risk associated with configuration drift, and crucially they make it very easy to apply OS security upgrades to the fleet: we can simply run a deployment.

• EB has mature CLI tooling for triggering deployments programmatically (from our Jenkins build server).

  EB actually has its own special CLI distinct from the main AWS CLI tools. We don't use this, as it assumes that EB configuration lives in the same repository as the source code, which doesn't map that nicely to a Docker-based deployment model.

  We build all our applications into Docker images, test the images, and then push them to the public Docker Hub, which makes it really straightforward to deploy our applications on Elastic Beanstalk. All we have to do is upload a bare JSON file (the Dockerrun.aws.json file) to S3 and create an EB application version based on that.

• We could use Application Load Balancers to front our applications. We know about ALBs, we're familiar with their configuration, and they integrate well with the TLS certificate management tools in AWS.

One of the significant downsides of Elastic Beanstalk, perversely enough, is its flexibility. There are an enormous number of knobs and switches to twiddle, and this can be paralysing when all you want to do is get something up and running. As such, we've standardised the environments in which our applications run and the way they are deployed to those environments.

These "standards" take the form of a handful of short shell scripts, which wrap the AWS CLI tools, and a shared library of Jenkins Pipeline tools which make creating a build pipeline for a new application relatively straightforward. If anyone wants any help understanding how these pieces fit together to make a passable replacement for Skyliner, feel free to hit me up on Twitter.