Skip to content
Cloud Foundry MySQL Release
Shell HTML Ruby Go
Latest commit 4457480 May 23, 2016 @njbennett njbennett committed with acrmp Don't append `quota-enforcer` user to IgnoredUsers
- The intent of the existing template is to ensure that the
`quota-enforcer` user is always ignored by itself
- However the quota-enforcer process always appends `quota-enforcer` to
the list of users to ignore.
- Doing it a second time here means that the generated SQL query mentions the
`quota-enforcer` user twice. This is functionally fine, but unnecessary.

[#119045503]

Signed-off-by: Andrew Crump <acrump@pivotal.io>
Failed to load latest commit information.
.final_builds Final release version v26 Feb 24, 2016
config Bump mariadb to 10.0.24 and galera to 25.3.14 Apr 19, 2016
docs Document quota-enforcer ignored_users property May 23, 2016
git-hooks Dyamically update package specs pre-commit for Go projects Jan 30, 2015
jobs Don't append `quota-enforcer` user to IgnoredUsers May 23, 2016
manifest-generation Add ignored_users to example property overrides May 23, 2016
packages Extract cluster-health-logger to new repo/package [#119423815] May 19, 2016
releases Final release version v26 Feb 24, 2016
scripts Add docs about deploying and cofiguring consul with the proxy May 19, 2016
src SST is disabled by default May 23, 2016
.envrc Remove requirement that tests are run from a specific directory. Apr 5, 2015
.gitignore Add script to test sample stubs. Apr 15, 2015
.gitmodules Extract cluster-health-logger to new repo/package [#119423815] May 19, 2016
.ruby-version Use ruby 2.3 in cf-mysql-release May 16, 2016
.travis.yml Bump src/github.com/cloudfoundry-incubator/switchboard May 5, 2015
CONTRIBUTING.md Add git branch workflow to CONTRIBUTING doc May 26, 2015
LICENSE Add initial LICENSE and NOTICE files. Oct 24, 2013
NOTICE Update NOTICE Dec 4, 2013
README.md Specify job overrides necessary for Consul May 19, 2016
commit-with-submodule-log bump cf-mysql-acceptance-tests Apr 21, 2016
submodule-log bump cf-mysql-acceptance-tests Apr 21, 2016
update Make 'update' script fail with valid error message Nov 23, 2015

README.md

Cloud Foundry MySQL Service

Table of contents

Components

Downloading a Stable Release

Development

Release notes & known issues

Deploying

Registering the Service Broker

Security Groups

Smoke Tests and Acceptance Tests

Deregistering the Service Broker

CI

Components

A BOSH release of a MySQL database-as-a-service for Cloud Foundry using MariaDB Galera Cluster and a v2 Service Broker.

ComponentDescription
CF MySQL Broker Advertises the MySQL service and plans. Creates and deletes MySQL databases and credentials (bindings) at the request of Cloud Foundry's Cloud Controller.
MySQL Server MariaDB 10.0.24; database instances are hosted on the servers.
Proxy Switchboard; proxies to MySQL, severing connections on MySQL node failure.

Proxy

Traffic to the MySQL cluster is routed through one or more proxy nodes. The current proxy implementation is Switchboard. This proxy acts as an intermediary between the client and the MySQL server, providing failover between MySQL nodes. The number of nodes is configured by the proxy job instance count in the deployment manifest.

NOTE: If the number of proxy nodes is set to zero, apps will be bound to the IP address of the first MySQL node in the cluster. If that IP address should change for any reason (e.g. loss of a VM) or a proxy was subsequently added, one would need to re-bind all apps to the IP address of the new node.

For more details see the proxy documentation.

Dashboard

A user-facing service dashboard is provided by the service broker that displays storage utilization information for each service instance. The dashboard is accessible by users via Single Sign-On (SSO) once authenticated with Cloud Foundry. The dashboard URL can be found by running cf service MY_SERVICE_INSTANCE.

Service authors interested in implementing a service dashboard accessible via SSO can follow documentation for Dashboard SSO.

Prerequisites

  1. SSO is initiated when a user navigates to the URL found in the dashboard_url field. This value is returned to cloud controller by the broker in response to a provision request, and is exposed in the cloud controller API for the service instance. A users client must expose this field as a link, or it can be obtained via curl (cf curl /v2/service_instances/:guid) and copied into a browser.

  2. SSO requires the following OAuth client to be configured in cf-release. This client is responsible for creating the OAuth client for the MySQL dashboard. Without this client configured in cf-release, the MySQL dashboard will not be accessible but the service will be otherwise functional. Registering the broker will display a warning to this effect.

    properties:
        uaa:
          clients:
            cc-service-dashboards:
              secret: cc-broker-secret
              scope: cloud_controller.write,openid,cloud_controller.read,cloud_controller_service_permissions.read
              authorities: clients.read,clients.write,clients.admin
              authorized-grant-types: client_credentials
    
  3. SSO was implemented in v169 of cf-release; if you are on an older version of cf-release you'll encounter an error when you register the service broker. If upgrading cf-release is not an option, try removing the following lines from the cf-mysql-release manifest and redeploy.

    dashboard_client:
      id: p-mysql
      secret: yoursecret

Implementation Notes

The following links show how this release implements Dashboard SSO integration.

  1. Update the broker catalog with the dashboard client properties
  2. Implement oauth workflow with the omniauth-uaa-oauth2 gem
  3. Use the cf-uaa-lib gem to get a valid access token and request permissions on the instance
  4. Before showing the user the dashboard, the broker checks to see if the user is logged-in and has permissions to view the usage details of the instance.

Broker Configuration

Require HTTPS when visiting Dashboard

The dashboard URL defaults to using the https scheme. This means any requests using http will automatically be redirected to https instead. To override this, you can change jobs.cf-mysql-broker_z1.ssl_enabled to false.

Keep in mind that changing the ssl_enabled setting for an existing broker will not update previously advertised dashboard URLs. Visiting the old URL may fail if you are using the SSO integration, because the OAuth2 client registered with UAA will expect users to both come from and return to a URI using the scheme implied by the ssl_enabled setting.

Note: If using https, the broker must be reached through an SSL termination proxy. Connecting to the broker directly on https will result in a port 443: Connection refused error.

Trust Self-Signed SSL Certificates

By default, the broker will not trust a self-signed SSL certificate when communicating with cf-release. To trust self-signed SSL certificates, you can change jobs.cf-mysql-broker_z1.skip_ssl_validation to true.

Downloading a Stable Release

Final releases are designed for public use, and are tagged with a version number of the form "v". The releases and corresponding release notes can be found on github.

Development

See our contributing docs for instructions on how to make a pull request.

This BOSH release doubles as a $GOPATH. It will automatically be set up for you if you have direnv installed.

# fetch release repo
mkdir -p ~/workspace
cd ~/workspace
git clone https://github.com/cloudfoundry/cf-mysql-release.git
cd cf-mysql-release/

# switch to develop branch (not master!)
git checkout develop

# automate $GOPATH and $PATH setup
direnv allow

# initialize and sync submodules
./update

If you do not wish to use direnv, you can simply source the .envrc file in the root of the release repo. You may manually need to update your $GOPATH and $PATH variables as you switch in and out of the directory.

Release Notes & Known Issues

For release notes and known issues, see the release wiki.

Deploying

Prerequisites

Overview

  1. Upload Stemcell
  2. Upload Release
  3. Create Infrastructure
  4. Deployment Components
  5. Create Manifest and Deploy

After installation, the MySQL service will be visible in the Services Marketplace; using the CLI, run cf marketplace.

Upload Stemcell

The latest final release expects the Ubuntu Trusty (14.04) go_agent stemcell version 2859 by default. Older stemcells are not recommended. Stemcells can be downloaded from http://bosh.io/stemcells; choose the appropriate stemcell for your infrastructure (vsphere esxi or aws hvm).

Upload Release

You can use a pre-built final release or build a dev release from any of the branches described in Getting the Code.

Final releases are stable releases created periodically for completed features. They also contain pre-compiled packages, which makes deployment much faster. To deploy the latest final release, simply check out the master branch. This will contain the latest final release and accompanying materials to generate a manifest. If you would like to deploy an earlier final release, use git checkout <tag> to obtain both the release and corresponding manifest generation materials. It's important that the manifest generation materials are consistent with the release.

If you'd like to deploy the latest code, build a release yourself from the develop branch.

Upload a pre-built final BOSH release

Run the upload command, referencing the latest config file in the releases directory.

  $ cd ~/workspace/cf-mysql-release
  $ git checkout master
  $ ./update
  $ bosh upload release releases/cf-mysql-<N>.yml

If deploying an older final release than the latest, check out the tag for the desired version; this is necessary for generating a manifest that matches the code you're deploying.

  $ cd ~/workspace/cf-mysql-release
  $ git checkout v<N>
  $ ./update
  $ bosh upload release releases/cf-mysql-<N>.yml

Create and upload a BOSH Release:

  1. Checkout one of the branches described in Getting the Code. Build a BOSH development release.

    $ cd ~/workspace/cf-mysql-release
    $ git checkout release-candidate
    $ ./update
    $ bosh create release
    

    When prompted to name the release, call it cf-mysql.

  2. Upload the release to your bosh environment:

    $ bosh upload release
    

Create Infrastructure

Note: No infrastructure changes are required to deploy to bosh-lite

Define subnets

Prior to deployment, the operator should define three subnets via their infrastructure provider. The MySQL release is designed to be deployed across three subnets to ensure availability in the event of a subnet failure. During installation, a fourth subnet is required for compilation vms. The sample_aws_stub.yml demonstrates how these subnets can be configured on AWS across multiple availability zones.

Create load balancer

In order to route requests to both proxies, the operator should create a load balancer. Manifest changes required to configure a load balancer can be found in the proxy documentation. Once a load balancer is configured, the brokers will hand out the address of the load balancer rather than the IP of the first proxy. Currently, load balancing requests across both proxies can increase the possibility of deadlocks. See the routing documentation for more information. To avoid this problem, configure the load balancer to route requests to the second proxy only in the event of a failure.

  • Note: When using an Elastic Load Balancer (ELB) on Amazon, make sure to create the ELB in the same VPC as your cf-mysql deployment
  • Note: For all load balancers, take special care to configure health checks to use the health_port of the proxies (default 1936). Do not configure the load balancer to use port 3306.

Deployment Components

Database nodes

There are three mysql jobs (mysql_z1, mysql_z2, mysql_z3) which should be deployed with one instance each. Each of these instances will reside in separate subnets as described in the previous section. The number of mysql nodes should always be odd, with a minimum count of three, to avoid split-brain. When the failed node comes back online, it will automatically rejoin the cluster and sync data from one of the healthy nodes. Note: Due to our bootstrapping procedure, if you are bringing up a cluster for the first time, there must be a database node in the first subnet.

Proxy nodes

There are two proxy jobs (proxy_z1, proxy_z2), which should be deployed with one instance each to different subnets. The second proxy is intended to be used in a failover capacity. In the event the first proxy fails, the second proxy will still be able to route requests to the mysql nodes.

Broker nodes

There are also two broker jobs (cf-mysql-broker_z1, cf-mysql-broker_z2) which should be deployed with one instance each to different subnets. The brokers each register a route with the router, which load balances requests across the brokers.

Create Manifest and Deploy

Deploy on BOSH-lite

To use the provided manifest stubs you will need a version of bosh-lite that supports multiple containers on the same subnet (bosh-lite version 9000.50.0+).

  1. Run the following script to generate a working manifest for a bosh-lite on your local machine:

    $ ./scripts/generate-bosh-lite-manifest
    
  2. Run the following command to deploy cf-mysql to the bosh-lite (assumes release has already been uploaded):

    $ bosh deploy
    
  3. To edit the manifest:

    $ bosh edit deployment
    

Deploy on AWS or vSphere

Copy sample stubs and fill in values

We have provided example stubs to serve as a starting point. Copy these stubs to your config repository, and fill in the REPLACE_WITH text with values for your environment.

For example, the following files can be used for an AWS deployment:

$ cp cf-mysql-release/manifest-generation/examples/aws/iaas-settings.yml \
    cf-mysql-release/manifest-generation/examples/property-overrides.yml \
    cf-mysql-release/manifest-generation/examples/cf-stub.yml \
    <YOUR_CONFIG_REPO>/cf-mysql/

Additional example stubs can be found under cf-mysql-release/manifest-generation/examples/. These include:

Fill in the CF stub

The script must obtain some configuration options from the CF deployment (e.g. CF Admin credentials). Edit the copied cf-stub.yml from the previous section to include values from your CF manifest. Existing admin users can be found under properties.uaa.scim.users in the CF manifest. This admin user must have the cloud_controller.admin UAA permission.

Note: This change to the CF manifest is temporary while we investigate better methods for sharing properties across deployments.

Generate AWS or vSphere manifest

Run the ./scripts/generate-deployment-manifest with the stubs you created in the preceeding steps.

Usage:
  Mandatory arguments:
    -c CF Manifest
    -p Property overrides stub file
    -i Infrastructure settings stub file
  Optional arguments
    -n Instance count overrides stub file (single node, 3 node)
    -v Release versions stub file
    -j Manifest stub file for additional jobs

For example:

$ ./scripts/generate-deployment-manifest \
  -c <YOUR_CONFIG_REPO>/cf-manifest.yml \
  -p <YOUR_CONFIG_REPO>/cf-mysql/property-overrides.yml \
  -i <YOUR_CONFIG_REPO>/cf-mysql/iaas-settings.yml \
  > cf-mysql.yml
Deploying

Once the manifest has been generated, do the following to deploy cf-mysql on your environment.

  1. Deploy CF-MySQL (assumes release has already been uploaded):

    $ bosh deploy
    
  2. To edit the manifest:

    $ bosh edit deployment
    

Deployment Manifest Properties

Manifest properties are described in the spec file for each job; see jobs.

You can find your director_uuid by running bosh status --uuid.

The MariaDB cluster nodes are configured by default with 100GB of persistent disk. This can be configured in your stub or manifest using disk_pools.mysql-persistent-disk.disk_size, however your deployment will fail if this is less than 3GB; we recommend allocating 10GB at a minimum.

Registering the Service Broker

BOSH errand

BOSH errands were introduced in version 2366 of the BOSH CLI, BOSH Director, and stemcells.

$ bosh run errand broker-registrar

Note: the broker-registrar errand will fail if the broker has already been registered, and the broker name does not match the manifest property jobs.broker-registrar.properties.broker.name. Use the cf rename-service-broker CLI command to change the broker name to match the manifest property then this errand will succeed.

Manually

  1. First register the broker using the cf CLI. You must be logged in as an admin.

    $ cf create-service-broker p-mysql BROKER_USERNAME BROKER_PASSWORD URL
    

    BROKER_USERNAME and BROKER_PASSWORD are the credentials Cloud Foundry will use to authenticate when making API calls to the service broker. Use the values for manifest properties jobs.cf-mysql-broker_z1.properties.auth_username and jobs.cf-mysql-broker_z1.properties.auth_password.

    URL specifies where the Cloud Controller will access the MySQL broker. Use the value of the manifest property jobs.cf-mysql-broker_z1.properties.external_host. By default, this value is set to p-mysql.<properties.domain> (in spiff: "p-mysql." .properties.domain).

    For more information, see Managing Service Brokers.

  2. Then make the service plan public.

Security Groups

Note: adding additional security groups for cf-mysql is not required on bosh-lites running cf-release v212 or later.

Since cf-release v175, applications by default cannot to connect to IP addresses on the private network. This prevents applications from connecting to the MySQL service. To enable access to the service, create a new security group for the IP configured in your manifest for the property jobs.cf-mysql-broker_z1.mysql_node.host.

  1. Add the rule to a file in the following json format; multiple rules are supported.

    [
          {
              "destination": "10.10.163.1-10.10.163.255",
              "protocol": "all"
          },
          {
              "destination": "10.10.164.1-10.10.164.255",
              "protocol": "all"
          },
          {
              "destination": "10.10.165.1-10.10.165.255",
              "protocol": "all"
          }
      ]
    
  2. Create a security group from the rule file.

    $ cf create-security-group p-mysql rule.json
  • Enable the rule for all apps $ cf bind-running-security-group p-mysql

Security group changes are only applied to new application containers; existing apps must be restarted.

Smoke Tests and Acceptance Tests

The smoke tests are a subset of the acceptance tests, useful for verifying a deployment. The acceptance tests are for developers to validate changes to the MySQL Release. These tests can be run manually or from a BOSH errand. For details on running these tests manually, see Acceptance Tests.

The MySQL Release contains an "acceptance-tests" job which is deployed as a BOSH errand. The errand can then be run to verify the deployment. A deployment manifest generated with the provided spiff templates will include this job. The errand can be configured to run either the smoke tests (default) or the acceptance tests.

Running Smoke Tests via BOSH errand

To run the MySQL Release Smoke tests you will need:

  • a running CF instance
  • credentials for a CF Admin user
  • a deployed MySQL Release with the broker registered and the plan made public

Run the smoke tests via bosh errand as follows:

$ bosh run errand acceptance-tests

Modifying values under jobs.acceptance-tests.properties may be required. Configuration options can be found in the job spec.

De-registering the Service Broker

The following commands are destructive and are intended to be run in conjuction with deleting your BOSH deployment.

BOSH errand

BOSH errands were introduced in version 2366 of the BOSH CLI, BOSH Director, and stemcells.

This errand runs the two commands listed in the manual section below from a BOSH-deployed VM.

$ bosh run errand broker-deregistrar

Manually

Run the following:

$ cf purge-service-offering p-mysql
$ cf delete-service-broker p-mysql

Deployment Resources

The service is configured to have a small footprint out of the box. These resources are sufficient for development, but may be insufficient for production workloads. If the service appears to be performing poorly, redeploying with increased resources may improve performance. See deployment resources for further details.

Something went wrong with that request. Please try again.