Improve the names and descriptions of the database plan offerings so that they are clear and make sense

[ccostino]

The current names and descriptions for the database plans that are offered by our RDS broker don't make much sense to our customers, nor do they adequately reflect the actual service instance details they represent. For instance medium-gp-psql doesn't mean much to anyone (what does the gp mean, anyway? -- it's mean for general purpose, but that's beside the point), and the description of Dedicated higher workload medium RDS PostgreSQL DB instance isn't that helpful, either.

We ought to update these names and descriptions so they are clearer, use plain language, and accurately represent their respective service details.

Acceptance criteria:

• When a user runs cf marketplace -e aws-rds they should see a useful listing of all database plans we currently support that have clear names and useful descriptions.

---

Security considerations:

• None; any details we share are already publicly visible in the catalog-template.yml file.

Implementation sketch:

• Update the catalog-template.yml names and descriptions once we know what want to put

[markdboyd]

@rbogle Could we change the database plan names? Would that affect people currently using those plans?

[markdboyd]

One of the open questions for this ticket is what impact, if any, there would be to existing database services if we renamed the service plan used to create them. It's possible that making such a change could leave databases in an orphaned/unmanageable state, so we need to test the impact of this change before making such changes for our broker in production.

Testing plan:

• Update pipeline for aws-broker to deploy a new instance of the AWS broker (e.g. aws-broker-mboyd-dev) in the development environment
  • Deploying/creating a separate service broker will ensure that no existing services created by the actual AWS broker in development are impacted by testing.
• Create a database using an existing plan on the test broker
• Rename the plan used to create the database
• Use cf CLI to determine whether database with renamed plan can still be used or managed

Cleanup:

• Delete services created with aws-broker-mboyd-dev
• Disable broker aws-broker-mboyd-dev
• Delete aws-broker-mboyd-dev app
• Use terraform to delete RDS databases and S3 snapshots bucket created for aws-broker-mboyd-dev
• Delete cg-broker-aws-broker-mboyd-dev-policy IAM policy
• Delete cg-broker-aws-broker-mboyd-dev IAM user
• Delete aws-broker-mboyd-dev pipeline

[markdboyd]

I have verified that when you rename (change the actual name attribute) of a service plan, even one that is actively in use, CF has no issues with it.

See this database service which was created with the micro-psql plan for aws-broker-mboyd-dev broker:

$ cf services 
Getting services in org cloud-gov-operators / space mark.boyd as markdboyd...

name                        service   plan         bound apps   last operation     broker                 upgrade available
mboyd-dev-test-micro-psql   aws-rds   micro-psql                create succeeded   aws-broker-mboyd-dev

But after I changed that plan name from micro-psql to micro-psql-test, the service still appears and CF automatically updated the plan name:

$ cf services
Getting services in org cloud-gov-operators / space mark.boyd as markdboyd...

name                        service   plan              bound apps   last operation     broker                 upgrade available
mboyd-dev-test-micro-psql   aws-rds   micro-psql-test                create succeeded   aws-broker-mboyd-dev

Thus, renaming service plans seems to be a safe operation even if they are currently in use.

As an aside, to make the aws-broker CI pipeline run without issues, you need to update the name in pipeline.ymland update the pipeline with set-pipeline before changing the plan name in catalog-template.yml and pushing a commit.

[markdboyd]

I created an initial PR to update the description and metadata of our RDS plans to hopefully be more clear:

#238

Other thoughts:

• We should probably remove our shared plans from the catalog, since they are deprecated Can't do this per @bengerman13's comment below
• Our medium-gp-psql* and large-gp-psql* plans are completely the same (both use db.m5.large instances). Should we deprecate one or the other? I assume not given the disruption such a change would incur. And perhaps we want to preserve the medium vs large distinction because while they may use the same instance type currently, that could change?
• I'd like to remove gp from the plan names because it has no obvious meaning (one of the original motivations for this created ticket). And the testing above proved that renaming plans has no ill effects. The real question is what to rename the plans. We could do something like:
  • medium-gp-psql -> large-psql
  • large-gp-psql -> ??? (Can't rename to xlarge because xlarge-sql would get used below)
  • xlarge-gp-psql -> xlarge-psql

Feedback from @cloud-gov/platform-ops on the above points would be welcome.

[bengerman13]

We should probably remove our shared plans from the catalog, since they are deprecated

doing this breaks our ability to modify or delete them. We have to wait until we've migrated/deleted all the instances

[bengerman13]

I did a little poking at this - my hunch is that non-gp used to be backed by t-series instances, and gp by m-series, which would've been much more notable back when t-series did not default to unlimited bursting.

I think it makes sense to build our plan around no longer differentiating those, so have something like:

• small
• medium
• large
• xlarge
• medium-redundant
• large-redundant
• xlarge-redundant

to line those up with instance types, we should consider:

• what plans are in use today (so we can minimize the service instances we need to migrate)
• what features are not available on lower instance classes (this is part of how we got into this mess; there was some size/version combination that didn't support disk encryption, which we need to support)

Once we've got that planned out, we can probably migrate users behind the scenes with zero downtime if we're careful. We still need to follow our deprecation policy, though, to make sure users can update any tooling they have that relies on plan names

[markdboyd]

Mapping of plans to RDS instance classes

Plan(s)	Instance type
micro-psql, micro-psql-redundant	db.t3.micro
small-psql, small-psql-redundant	db.t3.small
medium-psql, medium-psql-redundant	db.t3.medium
medium-gp-psql, medium-gp-psql-redundant	db.m5.large
large-gp-psql, large-gp-psql-redundant	db.m5.large
xlarge-gp-psql, xlarge-gp-psql-redundant	db.m5.xlarge
small-mysql, small-mysql-redundant	db.t2.small
medium-mysql, medium-mysql-redundant	db.t2.medium
medium-gp-mysql, medium-gp-mysql-redundant	db.m5.large
large-gp-mysql, large-gp-mysql-redundant	db.m5.large
xlarge-gp-mysql, xlarge-gp-mysql-redundant	db.m5.xlarge
medium-oracle-se2	db.t3.medium
large-gp-sqlserver-se	db.m5.large

Instance type differences

We are using DB instances from the db.t2, db.t3, and db.m5 families. The db.t2 and db.t3 families offer burstable performance, meaning they have capped performance with the ability to burst to full CPU usage for short periods based on CPU credits. These instances are really only ideal for apps with very low traffic (development/testing) or apps that truly do have small, short spikes in traffic.

The db.m5 instances have fixed performance based on the compute/memory resources defined for the instance type. Thus, these instances have more stable, predictable performance and are good for most applications, hence why they were tagged as "gp" or general purpose in our service plans.

Encryption

All of our currently supported database instance types support database encryption.

[bengerman13]

note that T3 instances are always burstable (for money that makes them effectively the same cost as an equivalent m5)

Amazon RDS T3 instances are configured for Unlimited mode, which means they can burst beyond the baseline over a 24-hour window for an additional charge.

[markdboyd]

I think the most natural transition plan would be to delete our medium-gp-psql* plans and medium-gp-mysql* plans and then rename the rest of the plans follows:

• large-gp-psql, large-gp-psql-redundant -> large-psql, large-psql-redundant
• xlarge-gp-psql, xlarge-gp-psql-redundant -> xlarge-psql, xlarge-psql-redundant
• large-gp-mysql, large-gp-mysql-redundant -> large-mysql, large-mysql-redundant
• xlarge-gp-mysql, xlarge-gp-mysql-redundant -> xlarge-mysql, xlarge-mysql-redundant

However, according to the results generated by this script for our production CF environment, we have:

• 12 medium-gp-psql database services which are bound to 29 apps total
• 5 medium-gp-psql-redundant database services which are bound to 6 apps total
• 5 medium-gp-mysql database services which are bound to 15 apps total
• 5 medium-gp-mysql-redundant database services which are bound to 5 apps total

It seems like it would be a big effort/impact to even considering deleting those instances or migrating them to a new instance class, so I'm inclined to say that it wouldn't be a worthwhile effort.

[bengerman13]

migrating them under the hood would not be hard, but we'd need to tell our users about the name change

[rbogle]

As I recall this ticket came about as we updated instance classes on the plans to move off of deprecating ones (t2, m3/4) notably t2 availability is very low and can take up to an hour to become available, and gen 5 got rid of ‘medium’ instance classes so the distinction between our medium and large plans effectively disappeared. The T class was largely intended to be used for low cost sandbox plans. Ms mapped more logically size to performance, but less so now. I think our naming should map to some notion of tiers of performance /capability vs ‘size’ which frees us to stay flexible with the underlying AWS changes in Instance classes and architectures. Hopefully some of that provides useful context!

On Mon, May 16, 2022 at 5:54 AM Ben Berry ***@***.***> wrote: migrating them <https://cli.cloudfoundry.org/en-US/v6/migrate-service-instances.html> under the hood would not be hard, but we'd need to tell our users about the name change — Reply to this email directly, view it on GitHub <#199 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3B5QPHVLB7VHAYMFSJ7QLVKJVSJANCNFSM5JWVBBAA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Thank You! Rian Bogle ( He/Him ) Platform Engineer @ Cloud.gov

[markdboyd]

@bengerman13 It looks like migrate-service-instances is only available on the CF CLI v6. We could run the CF CLI v6 just to use it, I suppose.

But even so, wouldn't we have to run the command for every database service that we wanted to upgrade? We could script it, I suppose, though we would have to be very very careful with that.

All of which again makes me wonder if the juice is worth the squeeze in eliminating service plans.

[markdboyd]

I'd also appreciate others input on this conversation:

#238 (comment)

[markdboyd]

After discussing with the team in standup today and with the users in our office hours, we decided:

• We are going to update our RDS website page with a reference to the AWS docs on maximum connections for choosing the right database plan size
• We are not going to do any further plan renaming or deletion given the low effort to value ratio of that work

[markdboyd]

PR to add guidance to our RDS documentation page:

cloud-gov/cg-site#2117

[markdboyd]

All PRs merged and cleanup complete. Closing.