Proposal: untied generation from folder structures

[ArthurHlt]

Introduction

Cf-ops-automation is tied to folder structure which can be a bottleneck for adoption, each teams use is own structure to handle bosh/cf deployment.

This proposal intent also to:

• Have less code to handle deployments (this will remove a lot things did by inspecting structure)
• Be more flexible

Untied deployment declaration

In this project each deployment must have a deployment-dependencies.yml which actually more describe deployment than dependencies.

We can use this file as a more generic solution to describe all deployments and where to find them, this file could be at the root of repository like travis can do with .travis.yml or any other CI system.

Proposal of the new format:

1. Rename this file by .deployments.yml and place is in the root folder
2. Deployment declaration could be have a new key called path to know where the stubs and templates are for the deployment.
3. Name could be not the first value in the yaml file, key name can be added.
4. Remove enable-cf-app.yml file, instead prefer use an enum called type which can have 2 possible values bosh (for bosh deployment) or cf for (for cloud foundry app)
5. Add sites key which contains list of sites available for deployment (e.g.: clouwatt, openwatt, sophia ...)
6. Add types key inside each site declared by sites which contains list of type of deployment (e.g.: prod, preprod, dev ...)
7. (optional) deployment for cloud foundry app could have multiple cf targets, a targets could be added for this usecase.

Here what can look the manifest:

deployments:
- name: micro-bosh
  type: bosh
  cli_version: v2
  path: path/to/my/deployment
  stemcells:
    bosh-openstack-kvm-ubuntu-trusty-go_agent: latest
  releases:
    route-registrar-boshrelease:
      base_location: https://bosh.io/d/github.com/
      repository: cloudfoundry-community/route-registrar-boshrelease    
    shield:
      base_location: https://bosh.io/d/github.com/
      repository: starkandwayne/shield-boshrelease        
    xxx_boshrelease:
      base_location: https://bosh.io/d/github.com/
      repository: xxx/yyyy
  errands:
    smoke_tests: ~
  sites:
  - name: cloudwatt
    types: [prod, dev]
- name: my-app
  type: bosh
  path: path/to/my/deployment
  targets:
  - cf_api_url: ~
    cf_username: ~
    cf_password: ~
    cf_organization: ~
    cf_space: ~

Use a manifest generation script instead of folder structure in a deployment

Actually, a deployment must always follow a structure and implicit manifest style (like using spruce).

Instead of oblige ops to follow this structure a simple shell script inside the deployment could be called.
Script should always receive the same input from concourse call and always the same output but you ops will be free to construct his final manifest.

Script name should always be called generate_manifest.sh to be after used by a CI systems.

As a side benefit by using a script let us be able to deploy in other CI than concourse.

Script input

$ ./generate_manifest.sh <sitename> <type>

• Sitename: define what is the site where we want to deploy. This can be cloudwatt, openwatt, sophia, bagnolet...
• Type: Define the type of deployment, this can be prod, preprod, qa, dev...

Those inputs let ops be able to generate manifest by these identifiers but he is not oblige to use it.
Input will be given by sites and types keys from .deployments.yml file by concourse.

Script ouput

The output should always return one or many absolute path to manifests to deploy for a deployment.

If there is more than one each path will be separate by a new line:

$ ./generate_manifest.sh cloudwatt prod
/path/to/my/first/manifest
/path/to/my/second/manifest

[poblin-orange]

Hello, thks for feedback and proposal.
Some thoughts:

• the main idea behind cf-ops-automation is standardizing complex bosh deployments (multiple directors, deployments, including cloudfoundry, services, and support tools), with convention over configuration / code orientation.
• as often (see bosh and cf), an opiniated approach limits somehow the freedom. Basically, there's nothing in cf-ops-automation you couldnt do with native concourse. Directory and naming is the convention we took to provide a descriptive automated deployment framework for production. Also, we wanted to have a strong separation between "reuseable aspects" and site specific secrets.

Im not sure the flexibility gained by removing central aspects of the cf-ops-automation makes sense, as the layer on top of concourse/bosh might be too thin to provide value

[ArthurHlt]

For your first point, it will be still the case.

Directory and naming convention is a too restrictive approach, this does not permit to others to use cf-ops-automation in their own environment without pain.

Even with this separation user still need, at some point, to make a link between his deployments and secrets for his site and environment. Sure, separate templating from stubs are interesting but there is a lack to those links easily, this proposal tried to find a way to do it.

It will be still automated, still separate, will still have a convention. it's just moving convention to another point to not be tied to a directory structure.

This is proposal and it's need to be refined to find a compromise.

[poblin-orange]

As illustrated by the usage of cf-deployment, bosh-deployment, the naming convention is quite convenient and can be adapted with symlinks.
Wrt secrets management, large scale credhub usage and operators reduce the need to generate any manifest.
The "site" dimension is out of scope of core COA. As git is the main source of truth, one is free to add any site/qa-integration-production workflow on top of different COA platform instance.