No instructions for deploying to BOSH on AWS

[camelpunch]

I've tried to convert the BOSH Lite instructions, but end up lost because some stubs assume BOSH Lite. Some instructions for deploying a Diego CF to BOSH on AWS would make me happy.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/100895292.

[Amit-PivotalLabs]

Hey @camelpunch,

You're right, the manifest generation should be better documented in general, and in particular as it pertains to other infrastructures. In the mean time, here's what the BOSH-Lite instructions are, and I'll explain what needs to change:

cd ~/workspace/diego-release
./scripts/generate-deployment-manifest \
    ~/deployments/bosh-lite/director.yml \
    manifest-generation/bosh-lite-stubs/property-overrides.yml \
    manifest-generation/bosh-lite-stubs/instance-count-overrides.yml \
    manifest-generation/bosh-lite-stubs/persistent-disk-overrides.yml \
    manifest-generation/bosh-lite-stubs/iaas-settings.yml \
    manifest-generation/bosh-lite-stubs/additional-jobs.yml \
    ~/deployments/bosh-lite

The first argument to the script is just a stub with the director UUID:

---
director_uuid: YOUR_DIRECTOR_UUID

The second is a property overrides stub. You can see a full list of overridable properties with this command:

grep '(( property_over' manifest-generation/diego.yml | cut -d'(' -f3- | cut -d' ' -f2 | sort | uniq

Your manifest will "compile" with out overriding any properties, but there are some that you need to override for your cluster to work, and it's impossible to have a sane default for them because it depends a lot on things like your networking setup. Probably the simplest thing you can do to start for this stub is something like this:

property_overrides:
  etcd:
    require_ssl: false
  executor:
    allow_privileged: true
  garden-linux:
    allow_networks: ["10.10.0.2/32"] # This is the DNS for your AWS VPC; it depends on your CIDR range.
    btrfs_store_size_mb: 25000
  skip_cert_verify: true

This is similar to how it's deployed in the test environment, but a few key things will be missing:

• syslog forwarding to a service like papertrail
• encrypted traffic between etcd clients and servers, as well as peer-to-peer traffic amongst etcd servers
• the receptor will not register a route with the router, so you can't do things like point xray at it
• I don't know whether SSH access to your containers will work

The third, fourth, and sixth arguments allow you to override instance counts for jobs, persistent disk settings for jobs (specifically etcd), and colocate additional templates from additional releases. You probably don't need to modify these to get started, and can just use the relevant BOSH-Lite stubs.

The final (seventh) argument is a directory where you need to have a file called cf.yml which is your deployment manifest for the cf that you're deploying your diego alongside.

The third argument is an IaaS settings stub. Here's a quick way to see what it requires:

grep '(( iaas_settings' manifest-generation/diego.yml | cut -d'(' -f3- | cut -d' ' -f2 | sort | uniq

Here's an example that works assuming your VPC has been set up accordingly (with some names/IDs you'll have to change to match your IaaS configuration):

iaas_settings:
  compilation_cloud_properties:
    availability_zone: us-east-1b
    instance_type: c3.large
  resource_pool_cloud_properties:
  - cloud_properties:
      availability_zone: us-east-1b
      elbs:
      - NAME_OF_ELB_FOR_DIEGO_CONTAINER_SSH_ACCESS
    name: access_z1
  - cloud_properties:
      availability_zone: us-east-1c
      elbs:
      - NAME_OF_ELB_FOR_DIEGO_CONTAINER_SSH_ACCESS
    name: access_z2
  - cloud_properties:
      availability_zone: us-east-1d
      elbs:
      - NAME_OF_ELB_FOR_DIEGO_CONTAINER_SSH_ACCESS
    name: access_z3
  - cloud_properties:
      availability_zone: us-east-1b
    name: brain_z1
  - cloud_properties:
      availability_zone: us-east-1c
    name: brain_z2
  - cloud_properties:
      availability_zone: us-east-1d
    name: brain_z3
  - cloud_properties:
      availability_zone: us-east-1b
    name: cc_bridge_z1
  - cloud_properties:
      availability_zone: us-east-1c
    name: cc_bridge_z2
  - cloud_properties:
      availability_zone: us-east-1d
    name: cc_bridge_z3
  - cloud_properties:
      availability_zone: us-east-1b
      ephemeral_disk:
        size: 174080
        type: gp2
    name: cell_z1
  - cloud_properties:
      availability_zone: us-east-1c
      ephemeral_disk:
        size: 174080
        type: gp2
    name: cell_z2
  - cloud_properties:
      availability_zone: us-east-1d
      ephemeral_disk:
        size: 174080
        type: gp2
    name: cell_z3
  - cloud_properties:
      availability_zone: us-east-1b
    name: database_z1
  - cloud_properties:
      availability_zone: us-east-1c
    name: database_z2
  - cloud_properties:
      availability_zone: us-east-1d
    name: database_z3
  - cloud_properties:
      availability_zone: us-east-1b
    name: route_emitter_z1
  - cloud_properties:
      availability_zone: us-east-1c
    name: route_emitter_z2
  - cloud_properties:
      availability_zone: us-east-1d
    name: route_emitter_z3
  stemcell:
    name: bosh-aws-xen-hvm-ubuntu-trusty-go_agent
    version: latest
  subnet_configs:
  - name: diego1
    subnets:
    - cloud_properties:
        security_groups:
        - SOME_SECURITY_GROUP
        subnet: SOME_SUBNET_ID
      dns:
      - 10.10.0.2
      gateway: 10.10.50.1
      name: default_unused
      range: 10.10.50.0/25
      reserved:
      - 10.10.50.2 - 10.10.50.9
      static:
      - 10.10.50.10 - 10.10.50.31
  - name: diego2
    subnets:
    - cloud_properties:
        security_groups:
        - SOME_SECURITY_GROUP
        subnet: SOME_OTHER_SUBNET_ID
      dns:
      - 10.10.0.2
      gateway: 10.10.114.1
      name: default_unused
      range: 10.10.114.0/25
      reserved:
      - 10.10.114.2 - 10.10.114.9
      static:
      - 10.10.114.10 - 10.10.114.31
  - name: diego3
    subnets:
    - cloud_properties:
        security_groups:
        - SOME_SECURITY_GROUP
        subnet: YET_ANOTHER_SUBNET_ID
      dns:
      - 10.10.0.2
      gateway: 10.10.178.1
      name: default_unused
      range: 10.10.178.0/25
      reserved:
      - 10.10.178.2 - 10.10.178.9
      static:
      - 10.10.178.10 - 10.10.178.31

The hardest part is setting up AWS with the correct VPC, CIDRs, subnets, ELBs, security groups, and AZs. However, you may be able to piggyback off of an existing setup for CF on AWS if you have one. Especially if you're using a tool like CloudFormation to automate setting up your AWS environment, it might not be that bad.

[camelpunch]

Thanks @Amit-PivotalLabs! I do need to have the receptor register a route with the gorouter, as I'm doing this just to test that https://github.com/pivotal-cf-experimental/diegotaskscheduler (submits tasks and monitors them) is possible to deploy in a CF instance.

Great detail though, I'll have a play with this.

[Amit-PivotalLabs]

Combine this with your property overrides:

property_overrides:
  receptor:
    cors_enabled: true # not sure if this is needed in general, or some special thing for xray
    domain_names: ["receptor.YOUR_DOMAIN"] # subdomain can be anything, not necessarily "receptor"
    register_with_router: true
    username: RECEPTOR_BASIC_AUTH_UN
    password: RECEPTOR_BASIC_AUTH_PW

[camelpunch]

Cheers. I was hoping to get away with YOUR_DOMAIN being an IP. I guess I don't know it up front, though?

[Amit-PivotalLabs]

There's numerous options here.

On one end of the spectrum, you're deploying alongside a full CF, and you have an ELB balancing traffic in front of your routers with wildcard DNS pointing at your ELB. In that case, you use the domain from your wildcard record. This is the type of solution you'd use in a production setting.

On the other end, you could just spin up a jumpbox behind the firewall and work from there, and target the IP of one of the access VMs (where the receptor processes live). If you want to know up front, you could configure one of the access VMs to use a static IP, and make sure to allocate one for it in your deployment manifest that falls within the appropriate CIDR and doesn't clash with any other static IPs (e.g. the ones used for the database jobs).

There's all sorts of options in the middle, depending on how/whether you have DNS, an ELB, any HA Proxies, static IPs for the gorouters, etc.

[emalm]

Hi, @camelpunch,

Yes, we'd like to improve documentation about the AWS-specific information to supply for a Diego deployment. As @Amit-PivotalLabs pointed out, this information is primarily concentrated in the iaas-settings.yml stub, although depending on the desired deployment configuration you may also want to set values in persistent-disk-overrides.yml as well. I've added https://www.pivotaltracker.com/story/show/100909610 to address this, and I expect we'll get to it soon as part of polishing Diego's documentation for BOSH operators.

Thanks,
Eric, CF Runtime Diego PM

[camelpunch]

Great. I'll close this then. Sorry for going silent on it: it turned out not to be worth the effort for what I was doing.