Amazon Managed Streaming for Apache Kafka (MSK). Securely stream data with a fully managed, highly available Apache Kafka service.
For detailed information, check out our Operator Guide for this bundle.
Our bundles aren't intended to be used locally, outside of testing. Instead, our bundles are designed to be configured, connected, deployed and monitored in the Massdriver platform.
Bundles are the basic building blocks of infrastructure, applications, and architectures in Massdriver. Read more here.
Form input parameters for configuring a bundle for deployment.
View
brokers_per_zone
(number): Number of brokers per availability zone. Must be one of:[1, 2, 3, 4]
. Default:1
.instance_type
(string): Type of instance for brokers. Must be one of:['kafka.t3.small', 'kafka.m5.large', 'kafka.m5.xlarge', 'kafka.m5.2xlarge', 'kafka.m5.4xlarge', 'kafka.m5.8xlarge', 'kafka.m5.12xlarge', 'kafka.m5.16xlarge', 'kafka.m5.24xlarge']
.kafka_version
(string): The version of Kafka to run. Must be one of:['2.8.1', '2.8.0', '2.7.2', '2.7.0', '2.6.3', '2.6.2', '2.6.1', '2.6.0', '2.5.1', '2.4.1.1', '2.3.1', '2.2.1']
. Default:2.6.2
.logging_method
(string): Method used to store kafka logs. Must be one of:['cloudwatch', 's3']
. Default:cloudwatch
.volume_size
(integer): EBS volume size for message storage in GiB (between 1 and 16384). Minimum:1
. Maximum:16384
.
{
"__name": "Staging",
"brokers_per_zone": 1,
"instance_type": "kafka.t3.small",
"kafka_version": "2.6.2",
"logging_method": "cloudwatch",
"volume_size": 100
}
{
"__name": "Production",
"brokers_per_zone": 1,
"instance_type": "kafka.m5.2xlarge",
"kafka_version": "2.6.2",
"logging_method": "s3",
"volume_size": 500
}
Connections from other bundles that this bundle depends on.
View
-
aws_authentication
(object): . Cannot contain additional properties.data
(object)-
arn
(string): Amazon Resource Name.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
external_id
(string): An external ID is a piece of data that can be passed to the AssumeRole API of the Security Token Service (STS). You can then use the external ID in the condition element in a role's trust policy, allowing the role to be assumed only when a certain value is present in the external ID.
-
specs
(object)aws
(object): .-
region
(string): AWS Region to provision in.Examples:
"us-west-2"
-
-
vpc
(object): . Cannot contain additional properties.data
(object)infrastructure
(object)-
arn
(string): Amazon Resource Name.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
cidr
(string)Examples:
"10.100.0.0/16"
"192.24.12.0/22"
-
internal_subnets
(array)-
Items (object): AWS VCP Subnet.
-
arn
(string): Amazon Resource Name.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
aws_zone
(string): AWS Availability Zone.Examples:
-
cidr
(string)Examples:
"10.100.0.0/16"
"192.24.12.0/22"
Examples:
-
-
-
private_subnets
(array)-
Items (object): AWS VCP Subnet.
-
arn
(string): Amazon Resource Name.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
aws_zone
(string): AWS Availability Zone.Examples:
-
cidr
(string)Examples:
"10.100.0.0/16"
"192.24.12.0/22"
Examples:
-
-
-
public_subnets
(array)-
Items (object): AWS VCP Subnet.
-
arn
(string): Amazon Resource Name.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
aws_zone
(string): AWS Availability Zone.Examples:
-
cidr
(string)Examples:
"10.100.0.0/16"
"192.24.12.0/22"
Examples:
-
-
-
specs
(object)aws
(object): .-
region
(string): AWS Region to provision in.Examples:
"us-west-2"
-
Resources created by this bundle that can be connected to other bundles.
View
authentication
(object): Highly scalable event bus managed by AWS. Cannot contain additional properties.data
(object): Cannot contain additional properties.-
authentication
(object): Zookeeper and bootstrap broker connection strings. Cannot contain additional properties.brokers_connection_string
(string)zookeeper_connection_string
(string)
-
infrastructure
- One of
- AWS Infrastructure ARNobject: Minimal AWS Infrastructure Config. Cannot contain additional properties.
-
arn
(string): Amazon Resource Name.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
- AWS Infrastructure ARNobject: Minimal AWS Infrastructure Config. Cannot contain additional properties.
- One of
-
security
- One of
- AWS Security informationobject: Informs downstream services of network and/or IAM policies. Cannot contain additional properties.
-
iam
(object): IAM Policies. Cannot contain additional properties.^[a-z]+[a-z_]*[a-z]+$
(object)-
policy_arn
(string): AWS IAM policy ARN.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
-
identity
(object): For instances where IAM policies must be attached to a role attached to an AWS resource, for instance AWS Eventbridge to Firehose, this attribute should be used to allow the downstream to attach it's policies (Firehose) directly to the IAM role created by the upstream (Eventbridge). It is important to remember that connections in massdriver are one way, this scheme perserves the dependency relationship while allowing bundles to control the lifecycles of resources under it's management. Cannot contain additional properties.-
role_arn
(string): ARN for this resources IAM Role.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
-
network
(object): AWS security group rules to inform downstream services of ports to open for communication. Cannot contain additional properties.^[a-z-]+$
(object)-
arn
(string): Amazon Resource Name.Examples:
"arn:aws:rds::ACCOUNT_NUMBER:db/prod"
"arn:aws:ec2::ACCOUNT_NUMBER:vpc/vpc-foo"
-
port
(integer): Port number. Minimum:0
. Maximum:65535
. -
protocol
(string): Must be one of:['tcp', 'udp']
.
-
-
- AWS Security informationobject: Informs downstream services of network and/or IAM policies. Cannot contain additional properties.
- One of
-
specs
(object)-
aws
(object): .-
region
(string): AWS Region to provision in.Examples:
"us-west-2"
-
-
azure
(object): .region
(string): Select the Azure region you'd like to provision your resources in.
-
gcp
(object): .-
project
(string) -
region
(string): The GCP region to provision resources in.Examples:
"us-east1"
"us-east4"
"us-west1"
"us-west2"
"us-west3"
"us-west4"
"us-central1"
-
-
kafka
(object): Informs downstream bundles of Kafka specific data. Cannot contain additional properties.version
(string): Currently deployed Kafka version.
-
Did we miss something? Please submit an issue to report any bugs or request additional features.
Note: Massdriver bundles are intended to be tightly use-case scoped, intention-based, reusable pieces of IaC for use in the Massdriver platform. For this reason, major feature additions that broaden the scope of an existing bundle are likely to be rejected by the community.
Still want to get involved? First check out our contribution guidelines.
If your use-case isn't covered by this bundle, you can still get involved! Massdriver is designed to be an extensible platform. Fork this bundle, or create your own bundle from scratch!
Questions? Concerns? Adulations? We'd love to hear from you!
Please connect with us!