New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot create Aurora Serverless cluster using RDS Construct #929
Comments
Thanks for reporting. As a workaround, you should be able to use property overrides |
Ah sweet, thanks! |
I set this up for our usage, it's ugly, but it may be of use to someone who needs a "quick and dirty" version of Serverless Aurora: Please ignore the lack of comments, and possibly broken code, I removed some proprietary information. import {Connections, ISecurityGroup, IVpc, Port, SecurityGroup, SubnetSelection} from "@aws-cdk/aws-ec2";
import {
CfnDBCluster,
CfnDBSubnetGroup,
DatabaseSecret,
Endpoint,
SecretRotation,
SecretRotationApplication,
SecretRotationOptions
} from "@aws-cdk/aws-rds";
import {AttachmentTargetType, ISecretAttachmentTarget, SecretAttachmentTargetProps, SecretTargetAttachment} from "@aws-cdk/aws-secretsmanager";
import {Construct, RemovalPolicy, Token} from "@aws-cdk/core";
export interface ServerlessAuroraProps {
readonly vpc: IVpc;
readonly subnets: SubnetSelection;
readonly clusterName: string;
readonly masterUsername?: string;
readonly securityGroup?: ISecurityGroup;
readonly secretRotationApplication?: SecretRotationApplication;
readonly maxCapacity: number;
}
export class ServerlessAurora extends Construct implements ISecretAttachmentTarget {
public securityGroupId: string;
public clusterIdentifier: string;
public clusterEndpoint: Endpoint;
public secret: SecretTargetAttachment;
public connections: Connections;
public vpc: IVpc;
public vpcSubnets: SubnetSelection;
public secretRotationApplication: SecretRotationApplication;
public securityGroup: ISecurityGroup;
constructor(scope: Construct, id: string, private props: ServerlessAuroraProps) {
super(scope, id);
this.vpc = props.vpc;
this.vpcSubnets = props.subnets;
this.secretRotationApplication = props.secretRotationApplication || SecretRotationApplication.MYSQL_ROTATION_SINGLE_USER;
const secret = new DatabaseSecret(this, "MasterUserSecret", {
username: props.masterUsername || "root",
});
const securityGroup = props.securityGroup || new SecurityGroup(this, "DatabaseSecurityGroup", {
allowAllOutbound: true,
description: `DB Cluster (${props.clusterName}) security group`,
vpc: props.vpc
});
this.securityGroup = securityGroup;
this.securityGroupId = securityGroup.securityGroupId;
const cluster = new CfnDBCluster(this, "DatabaseCluster", {
engine: "aurora",
engineMode: "serverless",
engineVersion: "5.6",
dbClusterIdentifier: props.clusterName,
masterUsername: secret.secretValueFromJson("username").toString(),
masterUserPassword: secret.secretValueFromJson("password").toString(),
dbSubnetGroupName: new CfnDBSubnetGroup(this, "db-subnet-group", {
dbSubnetGroupDescription: `${props.clusterName} database cluster subnet group`,
subnetIds: props.vpc.selectSubnets(props.subnets).subnetIds
}).ref,
vpcSecurityGroupIds: [securityGroup.securityGroupId],
storageEncrypted: true,
// Maximum here is 35 days
backupRetentionPeriod: 35,
scalingConfiguration: {
autoPause: true,
secondsUntilAutoPause: 300,
minCapacity: 1,
maxCapacity: props.maxCapacity
}
});
cluster.applyRemovalPolicy(RemovalPolicy.DESTROY, {applyToUpdateReplacePolicy: true});
this.clusterIdentifier = cluster.ref;
// create a number token that represents the port of the cluster
const portAttribute = Token.asNumber(cluster.attrEndpointPort);
this.clusterEndpoint = new Endpoint(cluster.attrEndpointAddress, portAttribute);
if (secret) {
this.secret = secret.addTargetAttachment('AttachedSecret', {target: this});
}
const defaultPort = Port.tcp(this.clusterEndpoint.port);
this.connections = new Connections({securityGroups: [securityGroup], defaultPort});
// This is currently causing errors when deploying, since it uses a SAM template under the hood
// Error: "Received malformed response from transform AWS::Serverless-2016-10-31"
// It also adds in a warning from the CDK:
// "This stack is using the deprecated `templateOptions.transform` property. Consider switching to `templateOptions.transforms`."
// Which has notified *all* of our DevOps when we upgraded.
// this.addRotationSingleUser("Rotation");
}
/**
* Adds the single user rotation of the master password to this cluster.
*/
public addRotationSingleUser(id: string, options?: SecretRotationOptions): SecretRotation {
if (!this.secret) {
throw new Error('Cannot add single user rotation for a cluster without secret.');
}
return new SecretRotation(this, id, {
secret: this.secret,
application: this.secretRotationApplication,
vpc: this.vpc,
vpcSubnets: this.vpcSubnets,
target: this,
automaticallyAfter: options ? options.automaticallyAfter : undefined,
});
}
public asSecretAttachmentTarget(): SecretAttachmentTargetProps {
return {
targetId: this.clusterIdentifier,
targetType: AttachmentTargetType.CLUSTER
};
}
} |
No. The DatabaseCluster doesn't expose the |
I see, would it be preferable to add a "Serverless" Database Prop or instead change DatabaseClusterProps to support "Serverless" configurations? |
I think creating a separate resource is probably the best option considering most of the options on the current cluster type don't quite apply. E.g only MySQL and Postgres are available, and you can't call lambdas from stored procedures or queries, etc. Serverless also has the "Web API" feature that may be useful to turn on from some stacks. (Though, I'm not sure that it's currently supported in Cfn) |
This link doesn't seem to work. Can you give an example of how to do this? |
I also dont see that link working. Can you please an example of property overrides? |
This doc shows how to do property overrides. |
any hopes for a release that allows this? |
@jogold is this a matter of adding an |
I think that it's more complicated than that and that it requires a separate class. |
I would use this |
I built on some of the work by @ApocDev and @StevenAskwith, in case it helps anyone else here: const vpc = new Vpc(this, 'Vpc', {maxAzs: 2});
const securityGroup = new SecurityGroup(this, 'AuroraSecurityGroup', {
vpc,
allowAllOutbound: true,
});
const secret = new DatabaseSecret(this, 'AuroraSecret', {
username: 'root',
});
const cluster = new CfnDBCluster(this, 'AuroraCluster', {
engine: 'aurora',
engineVersion: '5.6.10a',
engineMode: 'serverless',
masterUsername: secret.secretValueFromJson('username').toString(),
masterUserPassword: secret.secretValueFromJson('password').toString(),
vpcSecurityGroupIds: [securityGroup.securityGroupId],
deletionProtection: false,
enableHttpEndpoint: true,
storageEncrypted: true,
backupRetentionPeriod: 7,
dbSubnetGroupName: new CfnDBSubnetGroup(this, 'AuroraSubnetGroup', {
dbSubnetGroupDescription: 'AuroraSubnetGroup',
subnetIds: vpc.selectSubnets({subnetType: SubnetType.PRIVATE}).subnetIds,
}).ref,
scalingConfiguration: {
autoPause: true,
minCapacity: 1,
maxCapacity: 16,
secondsUntilAutoPause: 300,
},
}); |
Working construct with CDK 1.55.0 import * as cdk from '@aws-cdk/core';
import * as rds from '@aws-cdk/aws-rds';
import * as secretsmanager from '@aws-cdk/aws-secretsmanager';
import * as ec2 from '@aws-cdk/aws-ec2';
import * as kms from '@aws-cdk/aws-kms';
export interface AuroraServerlessProps {
readonly engine: rds.IClusterEngine;
readonly clusterIdentifier?: string;
readonly parameterGroup?: rds.ParameterGroup;
readonly vpc: ec2.IVpc;
readonly vpcSubnets: ec2.SubnetSelection;
readonly securityGroups?: ec2.ISecurityGroup[];
readonly masterUsername?: string;
readonly defaultDatabaseName?: string;
readonly backup?: rds.BackupProps;
readonly preferredMaintenanceWindow?: string;
readonly storageEncryptionKey?: kms.IKey;
readonly deletionProtection?: boolean;
readonly removalPolicy?: cdk.RemovalPolicy;
readonly scalingConfig?: rds.CfnDBCluster.ScalingConfigurationProperty;
readonly enableDataApi?: boolean;
}
// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-rds-dbcluster.html
// https://madabout.cloud/2019/09/01/aws-data-api-for-amazon-aurora-serverless/
// https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-rds/lib/cluster.ts
// https://github.com/ysku/aurora-serverless-example/blob/master/lib/database.ts
export class AuroraServerless extends cdk.Resource implements ec2.IConnectable, secretsmanager.ISecretAttachmentTarget {
/**
* Identifier of the cluster
*/
public readonly clusterIdentifier: string;
/**
* ARN of the cluster
*/
public readonly clusterArn: string;
/**
* The endpoint to use for read/write operations
*/
public readonly clusterEndpoint: rds.Endpoint;
/**
* Access to the network connections
*/
public readonly connections: ec2.Connections;
/**
* Security group identifier of this database
*/
public readonly securityGroups: ec2.ISecurityGroup[];
/**
* The secret attached to this cluster
*/
public readonly secret?: secretsmanager.ISecret;
private readonly secretRotationApplication: secretsmanager.SecretRotationApplication;
/**
* The VPC where the DB subnet group is created.
*/
private readonly vpc: ec2.IVpc;
/**
* The subnets used by the DB subnet group.
*/
private readonly vpcSubnets: ec2.SubnetSelection;
constructor(scope: cdk.Construct, id: string, props: AuroraServerlessProps) {
super(scope, id);
const {
clusterIdentifier,
engine,
parameterGroup,
storageEncryptionKey,
backup,
preferredMaintenanceWindow,
deletionProtection,
removalPolicy,
defaultDatabaseName,
vpc,
vpcSubnets,
scalingConfig,
enableDataApi,
masterUsername,
} = props;
this.vpc = vpc;
this.vpcSubnets = vpcSubnets;
// DB subnet group
const { subnetIds } = vpc.selectSubnets(vpcSubnets);
const dbSubnetGroup = new rds.CfnDBSubnetGroup(this, 'DbSubnetGroup', {
dbSubnetGroupDescription: `CloudFormation managed DB subnet group.`,
subnetIds,
});
// DB security group
const securityGroups = props.securityGroups ?? [
new ec2.SecurityGroup(this, 'DbSecurityGroup', {
vpc: vpc,
allowAllOutbound: false,
}),
];
this.securityGroups = securityGroups;
// DB secret
const secret = new rds.DatabaseSecret(this, 'DbSecret', {
username: masterUsername || 'root',
});
this.secretRotationApplication = engine.singleUserRotationApplication;
// bind the engine to the Cluster
const clusterEngineBindConfig = engine.bindToCluster(this, {
parameterGroup: parameterGroup,
});
const clusterParameterGroup = parameterGroup ?? clusterEngineBindConfig.parameterGroup;
const clusterParameterGroupConfig = clusterParameterGroup?.bindToCluster({});
// DB cluster
const cluster = new rds.CfnDBCluster(this, 'DbCluster', {
// basic
engine: engine.engineType,
engineVersion: engine.engineVersion?.fullVersion,
engineMode: 'serverless',
dbClusterIdentifier: clusterIdentifier,
dbSubnetGroupName: dbSubnetGroup.ref,
vpcSecurityGroupIds: securityGroups.map((sg) => sg.securityGroupId),
// port: clusterEngineBindConfig.port, // Aurora Serverless always runs on default port
dbClusterParameterGroupName: clusterParameterGroupConfig?.parameterGroupName,
// associatedRoles: undefined,
// admin
masterUsername: secret.secretValueFromJson('username').toString(),
masterUserPassword: secret.secretValueFromJson('password').toString(),
backupRetentionPeriod: backup?.retention.toDays(),
preferredBackupWindow: backup?.preferredWindow,
preferredMaintenanceWindow: preferredMaintenanceWindow,
databaseName: defaultDatabaseName,
deletionProtection: deletionProtection,
// encryption
kmsKeyId: storageEncryptionKey?.keyArn,
// storageEncrypted: storageEncryptionKey ? true : encrypted, // Aurora Serverless is always encrypted
// serverless config
enableHttpEndpoint: enableDataApi,
scalingConfiguration: scalingConfig,
});
// Default deletion policy for AWS:RDS:DBCluster is SNAPSHOT
// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-deletionpolicy.html
if (removalPolicy) {
cluster.applyRemovalPolicy(removalPolicy, {
applyToUpdateReplacePolicy: true,
});
} else {
// The CFN default doesn't cover UpdateReplacePolicy. Fix that here.
cluster.cfnOptions.updateReplacePolicy = cdk.CfnDeletionPolicy.SNAPSHOT;
}
this.clusterIdentifier = cluster.ref;
this.secret = secret.attach(this);
this.clusterEndpoint = new rds.Endpoint(cluster.attrEndpointAddress, cdk.Token.asNumber(cluster.attrEndpointPort));
const defaultPort = ec2.Port.tcp(this.clusterEndpoint.port);
this.connections = new ec2.Connections({ securityGroups, defaultPort });
const { region, account } = this.stack;
this.clusterArn = `arn:aws:rds:${region}:${account}:cluster:${this.clusterIdentifier}`;
}
/**
* Adds the single user rotation of the master password to this cluster.
*/
// https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L542
public addRotationSingleUser(automaticallyAfter?: cdk.Duration): secretsmanager.SecretRotation {
if (!this.secret) {
throw new Error('Cannot add single user rotation for a cluster without secret.');
}
const id = 'RotationSingleUser';
const existing = this.node.tryFindChild(id);
if (existing) {
throw new Error('A single user rotation was already added to this cluster.');
}
return new secretsmanager.SecretRotation(this, id, {
secret: this.secret,
automaticallyAfter,
application: this.secretRotationApplication,
vpc: this.vpc,
vpcSubnets: this.vpcSubnets,
target: this,
});
}
public asSecretAttachmentTarget(): secretsmanager.SecretAttachmentTargetProps {
return {
targetType: secretsmanager.AttachmentTargetType.RDS_DB_CLUSTER,
targetId: this.clusterIdentifier,
};
}
/*
// https://github.com/aws/aws-cdk/blob/26a69b1b090b49505f69ef2879b68d2382ea27ec/packages/%40aws-cdk/aws-rds/lib/cluster.ts#L566
public addRotationMultiUser(id: string, options: rds.RotationMultiUserOptions): secretsmanager.SecretRotation {
if (!this.secret) {
throw new Error('Cannot add multi user rotation for a cluster without secret.');
}
return new secretsmanager.SecretRotation(this, id, {
secret: options.secret,
masterSecret: this.secret,
automaticallyAfter: options.automaticallyAfter,
application: this.multiUserRotationApplication,
vpc: this.vpc,
vpcSubnets: this.vpcSubnets,
target: this,
});
}
*/
} |
this article saved my day |
Thanks @asterikx for sharing your construct, saved a lot of time! |
Adds a new construct to specify Aurora Serverless clusters This is largely a stripped down version of `Cluster` as there are many properties within clusters that do not apply to Aurora Serverless. Some of the notable exclusions are: * backup windows * maintenance windows * associated roles (as S3 import/exports are not supported) * iam database authentication * exporting cloudwatch log exports * storage encryption is always true in Aurora serverless Added: * Scaling options which only apply to Aurora serverless and cannot be used for provisioned clusters * Enum `AuroraCapacityUnit` to specify the provisioned capacity * Enable http endpoint for a serverless cluster Closes #929 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
For those finding this issue closed after a time and looking for what to do now: An L2 construct Have fun. |
This still seems problematic (or at least not clearly documented). For instance, Also, why are Lastly, it seems impossible to create a serverless cluster without NAT gateways, which are not required. If I use isolated subnets for the
But if I create private subnets in my VPC definition, NAT gateways are automatically created. Setting Expand to see definition const secret = new rds.DatabaseSecret(stack, 'RDSSecret', {
secretName: 'dev/rds',
username: process.env.APP_NAME,
});
const subnetGroup = new rds.SubnetGroup(stack, 'RDSSubnetGroup', {
subnetGroupName: 'rds-subnet-group',
vpc: props.vpc,
description: 'Private subnet group for DBs',
});
const db = new rds.ServerlessCluster(stack, 'Database', {
backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 7),
clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
credentials: {
username: process.env.APP_NAME,
secret,
},
subnetGroup,
defaultDatabaseName: process.env.APP_NAME,
deletionProtection: stack.stage === 'prod',
enableDataApi: true,
engine: {
engineType: 'aurora-postgresql',
engineVersion: {
majorVersion: '10',
fullVersion: '10.7',
},
// Why are these rotation values needed?
multiUserRotationApplication:
secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_MULTI_USER,
singleUserRotationApplication:
secretsmanager.SecretRotationApplication.POSTGRES_ROTATION_SINGLE_USER,
supportedLogTypes: ['what is this? not documented'],
// What is this?
bindToCluster({}, {}) {},
},
scaling: {
autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
minCapacity: rds.AuroraCapacityUnit.ACU_1,
maxCapacity:
stack.stage === 'prod'
? rds.AuroraCapacityUnit.ACU_8
: rds.AuroraCapacityUnit.ACU_1,
},
}); |
Related: #15929 |
I made some progress on this, but there are still some questionable things going on. Here's my definition of my serverless cluster using const db = new rds.CfnDBCluster(stack, 'RDSDatabase', {
backupRetentionPeriod: stack.stage === 'prod' ? 30 : 1,
databaseName: process.env.APP_NAME,
dbClusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
enableHttpEndpoint: true,
engine: 'aurora-postgresql',
engineMode: 'serverless',
engineVersion: '10.14',
masterUsername: secret.secretValueFromJson('username').toString(),
masterUserPassword: secret.secretValueFromJson('password').toString(),
vpcSecurityGroupIds: [dbSecGroup.securityGroupId],
scalingConfiguration: {
autoPause: stack.stage !== 'prod',
maxCapacity: stack.stage === 'prod' ? 8 : 2,
minCapacity: 2,
secondsUntilAutoPause: 3600,
},
}); Here's the const db = new rds.ServerlessCluster(stack, 'RDSDatabase', {
backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 1),
clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
credentials: rds.Credentials.fromSecret(secret),
defaultDatabaseName: process.env.APP_NAME,
enableDataApi: true,
engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
scaling: {
autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
maxCapacity:
stack.stage === 'prod'
? rds.AuroraCapacityUnit.ACU_8
: rds.AuroraCapacityUnit.ACU_2,
minCapacity: rds.AuroraCapacityUnit.ACU_2,
},
vpc: ec2.Vpc.fromLookup(stack, 'MainVpc', { isDefault: true }),
securityGroups: [dbSecGroup],
}); I get this error when trying to deploy: Yet using |
|
@ffxsam the reply by @nikita-sheremet-clearscale is spot on. Do you have any other questions/problems with getting this working? |
@skinny85 No sir! This wound up working fine for me: (expand for code) const secret = new rds.DatabaseSecret(stack, 'RDSSecret', {
secretName: 'dev/rds',
username: process.env.APP_NAME,
});
const mainVpc = ec2.Vpc.fromLookup(stack, 'MainVpc', { isDefault: true });
const dbSecGroup = new ec2.SecurityGroup(stack, 'RDSSecurityGroup', {
description: 'Access to DBs in VPC',
securityGroupName: 'database-access',
vpc: mainVpc,
allowAllOutbound: true,
});
dbSecGroup.addIngressRule(dbSecGroup, ec2.Port.tcp(5432));
const db = new rds.ServerlessCluster(stack, 'RDSDatabase', {
backupRetention: Duration.days(stack.stage === 'prod' ? 30 : 1),
clusterIdentifier: `${process.env.APP_NAME}-${stack.stage}`,
credentials: rds.Credentials.fromSecret(secret),
defaultDatabaseName: process.env.APP_NAME,
enableDataApi: true,
engine: rds.DatabaseClusterEngine.auroraPostgres({
version: rds.AuroraPostgresEngineVersion.VER_10_14,
}),
scaling: {
autoPause: Duration.hours(stack.stage === 'prod' ? 0 : 1),
maxCapacity:
stack.stage === 'prod'
? rds.AuroraCapacityUnit.ACU_8
: rds.AuroraCapacityUnit.ACU_2,
minCapacity: rds.AuroraCapacityUnit.ACU_2,
},
vpc: mainVpc,
vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
securityGroups: [dbSecGroup],
}); |
There doesn't seem to be a way to create an Aurora Serverless database cluster using the RDS Construct.
EngineMode
is available in the underlying cfn but not in the Construct library.Target Framework: netcoreapp2.1
Amazon.CDK: 0.12.0
Amazon.CDK.AWS.RDS: 0.12.0
The text was updated successfully, but these errors were encountered: