feat(redshift): manage database users and tables via cdk (#15931)

This feature allows users to manage Redshift database resources, such as users, tables, and grants, within their CDK application. Because these resources do not have CloudFormation handlers, this feature leverages custom resources and the Amazon Redshift Data API for creation and modification.

The generic construct for this type of resource is `DatabaseQuery`. This construct provides the base functionality required for interacting with Redshift database resources, including configuring administrator credentials, creating a custom resource handler, and granting necessary IAM permissions. The custom resource handler code contains utility functions for executing query statements against the Redshift database.

Specific resources that use the `DatabaseQuery` construct, such as `User` and `Table` are responsible for providing the following to `DatabaseQuery`: generic database configuration properties, specific configuration properties that will get passed to the custom resource handler (eg., `username` for `User`). Specific resources are also responsible for writing the lifecycle-management code within the handler. In general, this consists of: configuration extraction (eg., pulling `username` from the `AWSLambda.CloudFormationCustomResourceEvent` passed to the handler) and one method for each lifecycle event (create, update, delete) that queries the database using calls to the generic utility function.

Users have a fairly simple lifecycle that allows them to be created, deleted, and updated when a secret containing a password is updated (secret rotation has not been implemented yet). Because of #9815, the custom resource provider queries Secrets Manager in order to access the password.

Tables have a more complicated lifecycle because we want to allow columns to be added to the table without resource replacement, as well as ensuring that dropped columns do not lose data. For these reasons, we generate a unique name per-deployment when the table name is requested to be generated by the end user. We also notify create a new table (using a new generated name) if a column is to be dropped and let CFN lifecycle rules dictate whether the old table should be removed or kept.

User privileges on tables are implemented via the `UserTablePrivileges` construct. This construct is located in the `private` directory to ensure that it is not exported for direct public use. This means that user privileges must be managed through the `Table.grant` method or the `User.addTablePrivileges` method. Thus, each `User` will have at most one `UserTablePrivileges` construct to manage its privileges. This is to avoid a situation where privileges could be erroneously removed when the same privilege is managed from two different CDK applications. For more details, see the README, under "Granting Privileges".

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

packages/@aws-cdk/aws-redshift/README.md

@@ -26,15 +26,16 @@
 To set up a Redshift cluster, define a `Cluster`. It will be launched in a VPC.
 You can specify a VPC, otherwise one will be created. The nodes are always launched in private subnets and are encrypted by default.
 
-``` typescript
-import redshift = require('@aws-cdk/aws-redshift');
-...
-const cluster = new redshift.Cluster(this, 'Redshift', {
-    masterUser: {
-      masterUsername: 'admin',
-    },
-    vpc
-  });
+```ts
+import * as ec2 from '@aws-cdk/aws-ec2';
+
+const vpc = new ec2.Vpc(this, 'Vpc');
+const cluster = new Cluster(this, 'Redshift', {
+  masterUser: {
+    masterUsername: 'admin',
+  },
+  vpc
+});
 ```
 
 By default, the master password will be generated and stored in AWS Secrets Manager.
@@ -49,30 +50,198 @@ Depending on your use case, you can make the cluster publicly accessible with th
 To control who can access the cluster, use the `.connections` attribute. Redshift Clusters have
 a default port, so you don't need to specify the port:
 
-```ts
-cluster.connections.allowFromAnyIpv4('Open to the world');
+```ts fixture=cluster
+cluster.connections.allowDefaultPortFromAnyIpv4('Open to the world');
 ```
 
 The endpoint to access your database cluster will be available as the `.clusterEndpoint` attribute:
 
-```ts
+```ts fixture=cluster
 cluster.clusterEndpoint.socketAddress;   // "HOSTNAME:PORT"
 ```
 
 ## Rotating credentials
 
 When the master password is generated and stored in AWS Secrets Manager, it can be rotated automatically:
 
-```ts
+```ts fixture=cluster
 cluster.addRotationSingleUser(); // Will rotate automatically after 30 days
 ```
 
 The multi user rotation scheme is also available:
 
-```ts
+```ts fixture=cluster
+import * as secretsmanager from '@aws-cdk/aws-secretsmanager';
+
 cluster.addRotationMultiUser('MyUser', {
-  secret: myImportedSecret
+  secret: secretsmanager.Secret.fromSecretNameV2(this, 'Imported Secret', 'my-secret'),
+});
+```
+
+## Database Resources
+
+This module allows for the creation of non-CloudFormation database resources such as users
+and tables. This allows you to manage identities, permissions, and stateful resources
+within your Redshift cluster from your CDK application.
+
+Because these resources are not available in CloudFormation, this library leverages
+[custom
+resources](https://docs.aws.amazon.com/cdk/api/latest/docs/custom-resources-readme.html)
+to manage them. In addition to the IAM permissions required to make Redshift service
+calls, the execution role for the custom resource handler requires database credentials to
+create resources within the cluster.
+
+These database credentials can be supplied explicitly through the `adminUser` properties
+of the various database resource constructs. Alternatively, the credentials can be
+automatically pulled from the Redshift cluster's default administrator
+credentials. However, this option is only available if the password for the credentials
+was generated by the CDK application (ie., no value vas provided for [the `masterPassword`
+property](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-redshift.Login.html#masterpasswordspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan)
+of
+[`Cluster.masterUser`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-redshift.Cluster.html#masteruserspan-classapi-icon-api-icon-experimental-titlethis-api-element-is-experimental-it-may-change-without-noticespan)).
+
+### Creating Users
+
+Create a user within a Redshift cluster database by instantiating a `User` construct. This
+will generate a username and password, store the credentials in a [AWS Secrets Manager
+`Secret`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-secretsmanager.Secret.html),
+and make a query to the Redshift cluster to create a new database user with the
+credentials.
+
+```ts fixture=cluster
+new User(this, 'User', {
+  cluster: cluster,
+  databaseName: 'databaseName',
+});
+```
+
+By default, the user credentials are encrypted with your AWS account's default Secrets
+Manager encryption key. You can specify the encryption key used for this purpose by
+supplying a key in the `encryptionKey` property.
+
+```ts fixture=cluster
+import * as kms from '@aws-cdk/aws-kms';
+
+const encryptionKey = new kms.Key(this, 'Key');
+new User(this, 'User', {
+  encryptionKey: encryptionKey,
+  cluster: cluster,
+  databaseName: 'databaseName',
+});
+```
+
+By default, a username is automatically generated from the user construct ID and its path
+in the construct tree. You can specify a particular username by providing a value for the
+`username` property. Usernames must be valid identifiers; see: [Names and
+identifiers](https://docs.aws.amazon.com/redshift/latest/dg/r_names.html) in the *Amazon
+Redshift Database Developer Guide*.
+
+```ts fixture=cluster
+new User(this, 'User', {
+  username: 'myuser',
+  cluster: cluster,
+  databaseName: 'databaseName',
+});
+```
+
+The user password is generated by AWS Secrets Manager using the default configuration
+found in
+[`secretsmanager.SecretStringGenerator`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-secretsmanager.SecretStringGenerator.html),
+except with password length `30` and some SQL-incompliant characters excluded. The
+plaintext for the password will never be present in the CDK application; instead, a
+[CloudFormation Dynamic
+Reference](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html)
+will be used wherever the password value is required.
+
+### Creating Tables
+
+Create a table within a Redshift cluster database by instantiating a `Table`
+construct. This will make a query to the Redshift cluster to create a new database table
+with the supplied schema.
+
+```ts fixture=cluster
+new Table(this, 'Table', {
+  tableColumns: [{ name: 'col1', dataType: 'varchar(4)' }, { name: 'col2', dataType: 'float' }],
+  cluster: cluster,
+  databaseName: 'databaseName',
+});
+```
+
+### Granting Privileges
+
+You can give a user privileges to perform certain actions on a table by using the
+`Table.grant()` method.
+
+```ts fixture=cluster
+const user = new User(this, 'User', {
+  cluster: cluster,
+  databaseName: 'databaseName',
+});
+const table = new Table(this, 'Table', {
+  tableColumns: [{ name: 'col1', dataType: 'varchar(4)' }, { name: 'col2', dataType: 'float' }],
+  cluster: cluster,
+  databaseName: 'databaseName',
+});
+
+table.grant(user, TableAction.DROP, TableAction.SELECT);
+```
+
+Take care when managing privileges via the CDK, as attempting to manage a user's
+privileges on the same table in multiple CDK applications could lead to accidentally
+overriding these permissions. Consider the following two CDK applications which both refer
+to the same user and table. In application 1, the resources are created and the user is
+given `INSERT` permissions on the table:
+
+```ts fixture=cluster
+const databaseName = 'databaseName';
+const username = 'myuser'
+const tableName = 'mytable'
+
+const user = new User(this, 'User', {
+  username: username,
+  cluster: cluster,
+  databaseName: databaseName,
+});
+const table = new Table(this, 'Table', {
+  tableColumns: [{ name: 'col1', dataType: 'varchar(4)' }, { name: 'col2', dataType: 'float' }],
+  cluster: cluster,
+  databaseName: databaseName,
+});
+table.grant(user, TableAction.INSERT);
+```
+
+In application 2, the resources are imported and the user is given `INSERT` permissions on
+the table:
+
+```ts fixture=cluster
+const databaseName = 'databaseName';
+const username = 'myuser'
+const tableName = 'mytable'
+
+const user = User.fromUserAttributes(this, 'User', {
+  username: username,
+  password: SecretValue.plainText('NOT_FOR_PRODUCTION'),
+  cluster: cluster,
+  databaseName: databaseName,
+});
+const table = Table.fromTableAttributes(this, 'Table', {
+  tableName: tableName,
+  tableColumns: [{ name: 'col1', dataType: 'varchar(4)' }, { name: 'col2', dataType: 'float' }],
+  cluster: cluster,
+  databaseName: 'databaseName',
 });
+table.grant(user, TableAction.INSERT);
 ```
 
-This module is part of the [AWS Cloud Development Kit](https://github.com/aws/aws-cdk) project.
+Both applications attempt to grant the user the appropriate privilege on the table by
+submitting a `GRANT USER` SQL query to the Redshift cluster. Note that the latter of these
+two calls will have no effect since the user has already been granted the privilege.
+
+Now, if application 1 were to remove the call to `grant`, a `REVOKE USER` SQL query is
+submitted to the Redshift cluster. In general, application 1 does not know that
+application 2 has also granted this permission and thus cannot decide not to issue the
+revocation. This leads to the undesirable state where application 2 still contains the
+call to `grant` but the user does not have the specified permission.
+
+Note that this does not occur when duplicate privileges are granted within the same
+application, as such privileges are de-duplicated before any SQL query is submitted.

packages/@aws-cdk/aws-redshift/lib/database-options.ts

@@ -0,0 +1,26 @@
+import * as secretsmanager from '@aws-cdk/aws-secretsmanager';
+import { ICluster } from './cluster';
+
+/**
+ * Properties for accessing a Redshift database
+ */
+export interface DatabaseOptions {
+  /**
+   * The cluster containing the database.
+   */
+  readonly cluster: ICluster;
+
+  /**
+   * The name of the database.
+   */
+  readonly databaseName: string;
+
+  /**
+   * The secret containing credentials to a Redshift user with administrator privileges.
+   *
+   * Secret JSON schema: `{ username: string; password: string }`.
+   *
+   * @default - the admin secret is taken from the cluster
+   */
+  readonly adminUser?: secretsmanager.ISecret;
+}

packages/@aws-cdk/aws-redshift/lib/index.ts

@@ -1,8 +1,11 @@
 export * from './cluster';
 export * from './parameter-group';
+export * from './database-options';
 export * from './database-secret';
 export * from './endpoint';
 export * from './subnet-group';
+export * from './table';
+export * from './user';
 
 // AWS::Redshift CloudFormation Resources:
 export * from './redshift.generated';

packages/@aws-cdk/aws-redshift/lib/private/database-query-provider/handler-name.ts

@@ -0,0 +1,5 @@
+export enum HandlerName {
+  User = 'user',
+  Table = 'table',
+  UserTablePrivileges = 'user-table-privileges',
+}

packages/@aws-cdk/aws-redshift/lib/private/database-query-provider/index.ts

@@ -0,0 +1,20 @@
+/* eslint-disable-next-line import/no-unresolved */
+import * as AWSLambda from 'aws-lambda';
+import { HandlerName } from './handler-name';
+import { handler as managePrivileges } from './privileges';
+import { handler as manageTable } from './table';
+import { handler as manageUser } from './user';
+
+const HANDLERS: { [key in HandlerName]: ((props: any, event: AWSLambda.CloudFormationCustomResourceEvent) => Promise<any>) } = {
+  [HandlerName.Table]: manageTable,
+  [HandlerName.User]: manageUser,
+  [HandlerName.UserTablePrivileges]: managePrivileges,
+};
+
+export async function handler(event: AWSLambda.CloudFormationCustomResourceEvent) {
+  const subHandler = HANDLERS[event.ResourceProperties.handler as HandlerName];
+  if (!subHandler) {
+    throw new Error(`Requested handler ${event.ResourceProperties.handler} is not in supported set: ${JSON.stringify(Object.keys(HANDLERS))}`);
+  }
+  return subHandler(event.ResourceProperties, event);
+}

packages/@aws-cdk/aws-redshift/lib/private/database-query-provider/privileges.ts

@@ -0,0 +1,70 @@
+/* eslint-disable-next-line import/no-unresolved */
+import * as AWSLambda from 'aws-lambda';
+import { TablePrivilege, UserTablePrivilegesHandlerProps } from '../handler-props';
+import { ClusterProps, executeStatement, makePhysicalId } from './util';
+
+export async function handler(props: UserTablePrivilegesHandlerProps & ClusterProps, event: AWSLambda.CloudFormationCustomResourceEvent) {
+  const username = props.username;
+  const tablePrivileges = props.tablePrivileges;
+  const clusterProps = props;
+
+  if (event.RequestType === 'Create') {
+    await grantPrivileges(username, tablePrivileges, clusterProps);
+    return { PhysicalResourceId: makePhysicalId(username, clusterProps, event.RequestId) };
+  } else if (event.RequestType === 'Delete') {
+    await revokePrivileges(username, tablePrivileges, clusterProps);
+    return;
+  } else if (event.RequestType === 'Update') {
+    const { replace } = await updatePrivileges(
+      username,
+      tablePrivileges,
+      clusterProps,
+      event.OldResourceProperties as UserTablePrivilegesHandlerProps & ClusterProps,
+    );
+    const physicalId = replace ? makePhysicalId(username, clusterProps, event.RequestId) : event.PhysicalResourceId;
+    return { PhysicalResourceId: physicalId };
+  } else {
+    /* eslint-disable-next-line dot-notation */
+    throw new Error(`Unrecognized event type: ${event['RequestType']}`);
+  }
+}
+
+async function revokePrivileges(username: string, tablePrivileges: TablePrivilege[], clusterProps: ClusterProps) {
+  await Promise.all(tablePrivileges.map(({ tableName, actions }) => {
+    return executeStatement(`REVOKE ${actions.join(', ')} ON ${tableName} FROM ${username}`, clusterProps);
+  }));
+}
+
+async function grantPrivileges(username: string, tablePrivileges: TablePrivilege[], clusterProps: ClusterProps) {
+  await Promise.all(tablePrivileges.map(({ tableName, actions }) => {
+    return executeStatement(`GRANT ${actions.join(', ')} ON ${tableName} TO ${username}`, clusterProps);
+  }));
+}
+
+async function updatePrivileges(
+  username: string,
+  tablePrivileges: TablePrivilege[],
+  clusterProps: ClusterProps,
+  oldResourceProperties: UserTablePrivilegesHandlerProps & ClusterProps,
+): Promise<{ replace: boolean }> {
+  const oldClusterProps = oldResourceProperties;
+  if (clusterProps.clusterName !== oldClusterProps.clusterName || clusterProps.databaseName !== oldClusterProps.databaseName) {
+    await grantPrivileges(username, tablePrivileges, clusterProps);
+    return { replace: true };
+  }
+
+  const oldUsername = oldResourceProperties.username;
+  if (oldUsername !== username) {
+    await grantPrivileges(username, tablePrivileges, clusterProps);
+    return { replace: true };
+  }
+
+  const oldTablePrivileges = oldResourceProperties.tablePrivileges;
+  if (oldTablePrivileges !== tablePrivileges) {
+    await revokePrivileges(username, oldTablePrivileges, clusterProps);
+    await grantPrivileges(username, tablePrivileges, clusterProps);
+    return { replace: false };
+  }
+
+  return { replace: false };
+}