New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simpler times - deployment manifests designed for human BOSH operators #237

merged 8 commits into from Dec 12, 2017


None yet
6 participants

drnic commented Nov 27, 2017


The goal of this WIP PR is to bring kubo-deployment to the people who already have a BOSH & perhaps CF. Right now it offers two exciting deployment stories:

  • "just works" without any variables

    bosh deploy manifests/cfcr.yml
  • CF routing

    bosh deploy kubo-deployment/manifests/cfcr.yml \
      -o kubo-deployment/manifests/ops-files/cf-routing.yml \
      -l <(./kubo-deployment/manifests/helper/

NOTE: due to the continued development of master branch and this PR rebasing against it, I've had to fork this PR into two stands.

  • To deploy kubo/0.9.0:

    git clone -b stable-0.9.0
  • To deploy kubo/latest (this PR against master branch):

    git clone -b cfcr-simpler-times

The PR also adds sanity tests to operator files against the base manifest using bin/run_opsfile_test (thanks to !!)

screen shot 2017-11-29 at 5 49 16 pm

This PR is not yet feature parity/full documentation/full upgrade of existing wrapper scripts. Since the PR might not be merged without CFCR Core team having written it themselves, I'll leave it for now.

I do love how simple the commands above are now.


This comment has been minimized.

cfdreddbot commented Nov 27, 2017

Hey drnic!

Thanks for submitting this pull request! I'm here to inform the recipients of the pull request that you and the commit authors have already signed the CLA.


This comment has been minimized.

cf-gitbot commented Nov 27, 2017

We have created an issue in Pivotal Tracker to manage this:

The labels on this github issue will be updated when the story is started.

- alias: trusty
os: ubuntu-trusty
version: "3468.5"
version: 3468.latest

This comment has been minimized.


iainsproat Nov 29, 2017


The CI pipeline is failing due to the following error:
Error: No stemcell found at ''.


This comment has been minimized.


drnic commented Dec 1, 2017

Due to the major refactoring of properties + links in kubo-release master branch, I've decided that this PR will assume kubo-release master branch only. I've got a new branch stable-0.9.0 for manifests/operators for kub/0.9.0.

I'd like kubo-deployment to have a develop -> release-candidate -> master branch system so that master will in future always be manifests for a stable final release


This comment has been minimized.


drnic commented Dec 1, 2017


I've squashed the 30+ commits into a single commit with a nice commit message.


This comment has been minimized.


drnic commented Dec 7, 2017

I tried to rebase against master but...

CONFLICT (content): Merge conflict in src/kubo-deployment-tests/resources/environments/test_gcp_with_service_key/director.yml
Auto-merging src/kubo-deployment-tests/resources/environments/test_gcp_with_cf_and_service_key/director.yml
Auto-merging src/kubo-deployment-tests/resources/environments/test_gcp/director.yml
Auto-merging src/kubo-deployment-tests/resources/environments/test_aws_cf/director.yml
Auto-merging src/kubo-deployment-tests/resources/environments/test_aws/director.yml
Auto-merging src/kubo-deployment-tests/generate_manifest_test.go
CONFLICT (content): Merge conflict in src/kubo-deployment-tests/generate_manifest_test.go
CONFLICT (modify/delete): manifests/ops-files/worker-haproxy.yml deleted in HEAD and modified in simpler times: one lovely base manifest plus PR tests. Version simpler times: one lovely base manifest plus PR tests of manifests/ops-files/worker-haproxy.yml left in tree.
CONFLICT (rename/delete): manifests/ops-files/worker-haproxy-vsphere.yml deleted in HEAD and renamed to manifests/ops-files/iaas/vsphere/worker-haproxy.yml in simpler times: one lovely base manifest plus PR tests. Version simpler times: one lovely base manifest plus PR tests of manifests/ops-files/iaas/vsphere/worker-haproxy.yml left in tree.
Auto-merging manifests/ops-files/iaas/vsphere/master-static-ip.yml
CONFLICT (rename/delete): manifests/ops-files/k8s-haproxy-static-ips-vsphere.yml deleted in HEAD and renamed to manifests/ops-files/iaas/vsphere/cloud-config/k8s-haproxy-static-ips.yml in simpler times: one lovely base manifest plus PR tests. Version simpler times: one lovely base manifest plus PR tests of manifests/ops-files/iaas/vsphere/cloud-config/k8s-haproxy-static-ips.yml left in tree.
CONFLICT (rename/delete): manifests/ops-files/worker-haproxy-openstack.yml deleted in HEAD and renamed to manifests/ops-files/iaas/openstack/worker-haproxy.yml in simpler times: one lovely base manifest plus PR tests. Version simpler times: one lovely base manifest plus PR tests of manifests/ops-files/iaas/openstack/worker-haproxy.yml left in tree.
Auto-merging manifests/ops-files/iaas/aws/cloud-provider.yml
Auto-merging manifests/ops-files/addons-spec.yml
CONFLICT (content): Merge conflict in manifests/ops-files/addons-spec.yml
Auto-merging manifests/cfcr.yml
CONFLICT (content): Merge conflict in manifests/cfcr.yml
Auto-merging configurations/generic/project-config.yml
Auto-merging bin/lib/deploy_utils
CONFLICT (content): Merge conflict in bin/lib/deploy_utils
Auto-merging bin/generate_cloud_config
CONFLICT (content): Merge conflict in bin/generate_cloud_config


a) I appreciate seeing some of the changes in master branch
b) I am trying to do PRs to kubo-release + this branch to add lovely BOSH links between kubo-release -> routing-release/cloud_controller_ng from cf-deployment

drnic and others added some commits Dec 1, 2017

simpler times: one lovely base manifest plus PR tests
Operators with an existing BOSH (and perhaps a Cloud Foundry) can now
easily deploy CFCR/Kubo.

  bosh -d cfcr deploy manifests/cfcr.yml

The base manifest now has zero user-provided variables. It "just works".

It has also been renamed from kubo.yml to cfcr.yml.

The available operator files are listed and documented in manifests/

The base manifest assumes a slightly different cloud-config from the
wrapper scripts, kubo-ci, and configurations/docs/ folders. The base manifest
subscribes to the set of `vm_types` that cf-deployment subscribes to.

The wrapper scripts + kubo-ci assume that the base manifest does have several
user-provided variables. For compatibility, the manifests/ops-files/misc/bootstrap.yml
file re-exposes these variables. All wrapper scripts have been updated so
that users of the public documentation/bootstrapping process will not be
negatively affected. Similarly kubo-ci is happy.

Also new is `bin/run_opsfile_test` - a sanity-test harness borrowed from
cf-deployment that checks that all operator files are in manifests/,
and that they are all being `bosh int cfcr.yml -o file.yml` validated. This
script can be added to early stages of CI, especially for the PR pipeline to
help users know they need to add documentation + sanity tests for new
operator files. Such a CI job should be public.

A new operator file `cf-routing-links.yml` is a WIP with upstream kubo-release
and cf-deployment to share Cloud Foundry APIs and credentials via BOSH links.
Service account handling extracted and remove HAProxy

Signed-off-by: Ben Chapman <>

Reviewed, and running through pipeline. Approved in principle

BenChapman and others added some commits Dec 11, 2017

Remove Elastic Search example

Signed-off-by: Iain Sproat <>
Add descriptiojn for AWS LB ops-file

Signed-off-by: Ben Chapman <>
Add test for service key ops files
- Move the ops files to their proper places
- Add them to README
- Fix the addons-spec test


Signed-off-by: Iain Sproat <>
Add ops-file tests to CI tests

Signed-off-by: Akshay Mankar <>
Use `bosh-cli` instead of `bosh`
Because CI has it like that


Signed-off-by: Iain Sproat <>
Use latest stemcell in default deployment.yml
- Bump stemcell version in bootstrap.yml


Signed-off-by: Akshay Mankar <>

@akshaymankar akshaymankar merged commit c3bc623 into cloudfoundry-incubator:master Dec 12, 2017

@cf-gitbot cf-gitbot added in progress and removed in progress labels Dec 12, 2017


This comment has been minimized.


akshaymankar commented Dec 12, 2017

Finally, we are done with this!! 🕺
Only thing that we thought was missing was tests for But we left that out as of now. @drnic, if you could write those tests that would be awesome. Thanks again for the PR!


This comment has been minimized.


drnic commented Dec 13, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment