Additional AWS managed resources

[infinitecompute]

What problem are you facing?

We are gathering community feedback to help us prioritize development of additional AWS managed services and maturing of existing service implementations.

• What are the most important AWS services for you? Please share your service list in the comments.
• What are your use cases? This will help us understand how to best support your situation.
• Would you be interested in contributing? If so, in which capacity? This could take the form of usage and early feedback, code contributions, and improved documentation.

Crossplane currently supports 58+ AWS API types, see https://doc.crds.dev/github.com/crossplane/provider-aws.

Please drop us a comment with a list of the most important AWS services for your use cases.

How could Crossplane help solve your problem?

We will be prioritizing updates and additional services in Crossplane based on feedback.

Related Issues

• VCP, Subnet, SecurityGroup and InternetGateway at v1beta1 move network resources to v1beta1 #182
• IAM from v1alpha3 -> v1beta1 AWS IAM resources to v1beta1 #100
• DBSubnet Group -> v1beta1 DBSubnet Group from Alpha to Beta #165
• DynamoDB Table Add AWS DynamoDB API types and controllers  #147
• More IAM types - Expanding IAM support #151
  • IAMRole, IAMRolePolicyAttachment -> v1beta1 move IAMRole and IAMRolePolicyAttachment to v1beta1 #141
  • IAMUser IAMUser #194, IAMUserPolicyAttachment IAMUserPolicyAttachment #196
  • IAMGroup, IAMGroupUserMembership, IAMGroupPolicyAttachment Added AWS IAMGroup with UserMembership, and PolicyAttachment #249
  • IAMUserPolicyAttachment refers to IAMPolicy Update IAMUserPolicyAttachment to refer IAMPolicy #231, PR Policy ARN referencers for IAMUserPolicyAttachment and IAMRolePolicyAttachment #257
• DNS: Route 53 Add Route53 DNS Support #172
• Certificate Manager AWS Certificate Manager #171
• SQS Add Simple Queue Service  #170
• ELB Elastic Load Balancer support #180
• SNS Simple Notification Service support #175
• RouteTable to v1beta1 AWS networking and VPC resources to v1beta1 #145
• S3 bucket API types and controllers to v1beta1 quality #331
• ElastiCache Add ElastiCache support #168, PR cache: add cachecluster resource #189
• ECR support Support for AWS Elastic Container Registry (ECR) #307
• S3 Bucket Policy to v1beta1 #391
• IAM User Access Key v1alpha1 #403
• Add API Gateway as a managed resource Add API Gateway as a managed resource #235
• ACK code generation of native Crossplane AWS resources and provider-aws see note below
  • see Crossplane Community Day #2 talk by @jaypipes (ACK lead) and @muvaf (Crossplane Maintainer) on Accelerating Crossplane AWS Provider coverage with ACK code generation that shows ACK generated Crossplane provider resources.

Up Next

• Add AWS ElasticSearch Service as managed resource Add AWS ElasticSearch Service as managed resource #238
• Add AWS EMR Cluster as a managed resource Add AWS EMR Cluser as a managed resource #239
• Lambda Add AWS Lambda Function as a custom resource #234
• DocumentDB (with MongoDB compatibility) AWS DocumentDB Support #268
• Cache Subnet Group CacheSubnetGroup #169, Implement CacheSubnetGroup resource for ReplicationGroup to be created in a VPC #95
• Add AWS Lambda Function as a custom resource Add AWS Lambda Function as a custom resource #234
• Add CloudFront as a managed resource Add CloudFront as a managed resource #236
• Add Auto Scaling Group as a managed resource Add Auto Scaling Group as a managed resource #237

Parked for Design Review

• Kinesis

[janwillies]

It might be interesting to watch what AWS is up to: https://github.com/aws/aws-service-operator-k8s

[negz]

@infinitecompute Should we move this issue to the stack-aws repo? We typically try to keep cloud-provider-aligned issues in their cloud-provider-stack repos.

[jbw976]

Thank makes the most sense to me @negz, i'll transfer now. Then I'll tweet that link to get some more community visibility on this call for opinions

[muvaf]

Elasticache provisioned by Crossplane cannot live in a VPC right now due to missing managed resource CacheSubnetGroup. It's a fairly simple resource as it's basically a logical grouping of subnets. We could consider implementing it. See #95 for details.

[infinitecompute]

Noted @muvaf , I'll add this dependency to the prioritization. thanks!

[muvaf]

RDS from alpha to beta

@infinitecompute RDS is already v1beta1

[janwillies]

We are a user of DocumentDB, it would be great to see this supported: #268

[nicorikken]

We are looking into ECR support. Our use-case (as discussed on Slack):

Currently we have deployed our own container registry within the Kubernetes cluster. We'd like to move to ECR to have a managed solution and also increase security by having seperate registries's. However, this means we need to migrate in the order of a 100 container registries's to separate ECR registries. But apart from that initial migration, teams ideally can create container registries just like they can create github repo's. Most are familiar with Kubernetes, but less so with AWS. So having ECR support in Crossplane could ease the creation of cluster-specific registries for teams, rather than having to learn the AWS UI. Also team-specific resources are now all defined in Kubernetes, so it would be nice to tie the ECR creation in to that templating (managed through ArgoCD Gitops).
To be honest, I'm not yet sure what would be easier: 1) clear team instructions how to use the AWS Console, 2) Some Cloudformation solution, 3) Managing from Kubernetes through Crossplane.

I'm still not sure if Crossplane is the right solution for our use-case, that might take until October this year. If it does, we might contribute code to implement this feature.

[prasek]

We are a user of DocumentDB, it would be great to see this supported: #268
@janwillies added to the list ☝️

We are looking into ECR support. Our use-case (as discussed on Slack) ...
@nicorikken - looks like @muvaf created #307 for this and it's added to the list

Also note that we're looking at code generation options for the AWS provider (e.g. crossplane/crossplane#262) so we'll be getting support for a lot more cloud services shortly. 🚀

[prasek]

We are looking into ECR support. Our use-case (as discussed on Slack):

@nicorikken - @enderv added ECR support in #337, let us know if you run into any issues, thx!

https://doc.crds.dev/github.com/crossplane/provider-aws/ecr.aws.crossplane.io/Repository/v1alpha1

[prasek]

@krishchow thanks for your contributions adding support for:

• S3 Bucket to v1beta1 #331
• S3 Bucket Policy support #289
• Referencer for SubnetGroup AWS ElasticCache #314
• Add ARN to AtProvider for SNS Topic #348

plus a few more on the way:

• S3 Bucket Policy to v1beta1 #391
• IAM User Access Key v1alpha1 #403

nice work! 🚀

[prasek]

Excited to see the joint work between @jaypipes, @muvaf, and @kasey to adapt the AWS ACK codegen pipeline to emit a native Crossplane provider-aws, so you can compose even more AWS cloud services using Crossplane:

https://twitter.com/jaypipes/status/1320714029917229064

Hi Jan! Yep, @muvaf has been digging in to the ACK code generator and running with a proof-of-concept hack that I whipped up a few weeks ago. I'm looking forward to reviewing his PRs this week and going further down the road of code re-use and sharing.

aws-controllers-k8s/community#313 (comment)

ACK's mission is to facilitate the most Kubernetes-native way for users to interact with AWS managed services via the Kubernetes API and configuration language. While Crossplane does enable Kubernetes users to create infrastructure resources using the Kubernetes API/language, Crossplane has a much broader mission of enabling cross-cloud-provider workflows and multi-provider infrastructure needs.

Crossplane and ACK contributors are actually collaborating with each other, as we view the two projects as complementary, not competitive. In fact, I've been noodling around some ideas of using the ack-generate CLI tool to output Go code that follows the Crossplane object model/interfaces -- something that would eventually allow Crossplane's AWS cloud provider code to be replaced with code generated from ACK.

To follow along checkout the following: 🚀

• https://github.com/jaypipes/aws-controllers-k8s/tree/crossplane
• Additional fields with defaults to generate.Config object aws-controllers-k8s/community#443
• Crossplane controller code generation using ACK pipeline aws-controllers-k8s/community#449
• AWS Go SDK v1 connection method for ACK-generated resources #401
• 12 new API Gateway managed resources generated by ACK #416 - 12 new generated API Gateway resources
• Generated ECR Repository API & Controller #409 - generated ECR resource & controller

[janwillies]

#407 (fargate) and #405 (secrets manager)

[lvalerio]

What are the most important AWS services for you?
#234 ( Lambda Function )

What are your use cases?
Event-driven programming. Need to deploy a Lambda function to run on events generated from an s3 bucket

Would you be interested in contributing? If so, in which capacity?
Code contributions and improved documentation

[muvaf]

Hi @lvalerio If you're interested in contributing #234 , you can take a look at our Provider Development Guide to get started. Feel free to join our Slack, too. Happy to help you along 🙂

[muvaf]

Here is the list of resources that we'll implement using AWS code generation pipeline:

• Generate SNS services using ACK #451 SNS missing resources
• Implement all KMS services #509 KMS
• Implement Amazon MQ service #510 Amazon MQ
• Implement missing RDS resources #511 RDS missing resources
• Implement missing Route53 resources #512 Route53 missing resources
• Implement SQS services #513 SQS missing resources
• Generate SageMaker Services using ACK #458 SageMaker

I believe this issue has been a long-running one and we may never be able to close it if we keep adding. So, I plan to close it after this list of issues are closed.

[dradetsky]

One service which my org will need & which is not yet mentioned here is EventBridge. Unfortunately, I'm not super-familiar with how we're currently using EventBridge, so I can't give a ton of useful info on our use-case (I should probably find out, huh?) but I believe it's basically

• There are several serverless apps which have their own EventBus's.
• Some serverless apps use EventBridge rules as a kind of crond-for-lambda.

Basically, there will probably be more of these kinds of rules added, and we'd like to provide a (much better) interface for app devs to define & use these rules.

[DWSR]

Hey all, we're investigating how to create an SQS-triggered Lambda function using Crossplane. It appears that creating the event source mapping between the SQS Queue and the Lambda function is currently unsupported in Crossplane based on prior art from Terraform.

This use case is super important to us because we need to be able to use Lambda to transform messages placed on an SQS queue (we're unable to use any other offering for $reasons).

[mikebollandajw]

We are also looking for Cloudwatch Event Rules / EventBridge to SNS (#1150)

[jrake-revelant]

Any chance the code-generation will be picked up again? It looks like only one of #149 (comment) got worked on. It would be great to know if we should count on more work of ACK being re-used. Thanks!

[haarchri]

please Open dedicated issues for new resources - we will close the ticket here

[xchan]

Here is the list of resources that we'll implement using AWS code generation pipeline:

• Generate SNS services using ACK #451 SNS missing resources
• Implement all KMS services #509 KMS
• Implement Amazon MQ service #510 Amazon MQ
• Implement missing RDS resources #511 RDS missing resources
• Implement missing Route53 resources #512 Route53 missing resources
• Implement SQS services #513 SQS missing resources
• Generate SageMaker Services using ACK #458 SageMaker

I believe this issue has been a long-running one and we may never be able to close it if we keep adding. So, I plan to close it after this list of issues are closed.

Any progress on SageMaker Services?