Integrate LocalStack for Developing AWS Applications Locally

[Blind-Striker]

This pull request initiates integrating fundamental AWS services into the Aspire. The current changes lay the groundwork for incorporating basic AWS services such as S3, SQS, and SNS.

The next phase of development will focus on integrating LocalStack support using localstack-dotnet-client, enabling local development and testing of these AWS integrations.

This PR is a work in progress and serves as a platform for discussion and feedback on the current implementation approach.

Fixes #874

(I'm open to any help and collaboration on this PR, including pair programming sessions. If you're interested or have insights to share, please DM me on Twitter or respond directly to this PR. )

Update

This PR is now focused solely on the integration of LocalStack and CloudFormation. For more information, please see the discussion below.

Ref: #975

[Blind-Striker]

@dotnet-policy-service agree

[davidfowl]

Thanks for jumping in and getting this started @Blind-Striker! Hopefully @normj will have some feedback as well.

These resources probably need to implement IResourceWithConnectionString so that they can be referenced by other projects using WithReference as part of orchestration.

[normj]

Thanks @Blind-Striker for getting the ball rolling. We are definitely interested in the Aspire project and I have been playing around with it to figure out what would be most useful.

With adding AWS resource identifies to the list of Resources in the AppHost I'm trying to understand how the component side in the applications projects would consume this information. For @davidfowl perspective our SDK service clients don't have any state to a particular resource and the identifiers are used at the individual operation basis. The SDK service clients are very RPC focused.

What I have been playing with is in the AppHost is having a host component that allows you to set the general configuration of the service clients, like the region and profile, with a default and a project specific override. It would also like to figure at a component level what would the be Aspire way of adding the service clients with the project pulling in those AppHost settings as well including the SDK with the Otel configuration.

[Blind-Striker]

I initially tried to directly mimic Azure resources in Aspire without fully understanding them, but I realized Azure and AWS SDKs approach things differently. As @normj mentioned, "SDK service clients don't have any state to a particular resource and the identifiers are used at the individual operation basis." Therefore, when defining AWS Aspire Resources, passing information like connection strings and URLs seems not necessary.

I shifted my focus to creating a working example. I started writing Aspire AWS Components using AWS SDKs. Here, I'm considering utilizing AWSSDK.Extensions.NETCore.Setup while creating Clients. Do you have any other suggestions @normj?

I'll also try to implement HealthCheck and Tracing, but I'm unsure about the ActivitySource implementation in AWS SDK.

Another topic is Provisioning. It seems Azure ResourceManager is equivalent of AWS CloudFormation in Azure. However, AWS SDKs can also create most resources. The best practice is probably to use AWS CDK for .NET, but initially, this might increase complexity. It may require wrapping CLI commands like "cdk deploy" and adding a node.js runtime to the project. So, starting with SDK-based resource creation seems reasonable.

I aim to create a sample Aspire project using AWS SNS, SQS, and S3 by next week and produce a working POC. Depending on @normj's feedback, I can continue or possibly hand over the project to the AWS .NET team, leaving development to the professionals. 😁

[RichiCoder1]

Another topic is Provisioning. It seems Azure ResourceManager is equivalent of AWS CloudFormation in Azure. However, AWS SDKs can also create most resources. The best practice is probably to use AWS CDK for .NET, but initially, this might increase complexity. It may require wrapping CLI commands like "cdk deploy" and adding a node.js runtime to the project. So, starting with SDK-based resource creation seems reasonable.

This is an interesting question. I think the closest analog to the current Azure experience (azd) would actually be either raw CloudFormation and/or AWS's Copilot (https://aws.github.io/copilot-cli/).

Supporting the .NET CDK would be an interesting exercise though, since the whole point of .NET Aspire is to do configuration over code.

[normj]

For provisioning I would probably go with plain CloudFormation. CDK is great but I wouldn't want to add the complexity of a Node dependency. At least not for application resource provisioning. For deployment that is a different story but lets not balloon this conversation with deployment.

You could create everything via the SDK but that makes the tear down hard. Having the application resources in a CloudFormation stack makes it easy to tear down all of the resources.

[Blind-Striker]

@davidfowl Up to now; two things have caught my attention:

• The project doesn't use nuget.org; instead, several different NuGet sources have been added. When I tried to restore, nuget couldn't find the AWS.S3 and AWSSDK.Extensions.NETCore.Setup libraries, which might indicate a synchronization issue. Temporarily, I've added nuget.org to the nuget.config for local development.
• Under src/Components/Common, two classes are used for Azure components: AzureComponent and HealthChecksExtensions. These aren't inside a C# project but are included to csproj of Azure components. This has been done to avoid adding an extra dependency to NuGet packages probably. I'll try to follow a similar approach.

[danmoseley]

The project doesn't use nuget.org; instead, several different NuGet sources have been added. When I tried to restore, nuget couldn't find the AWS.S3 and AWSSDK.Extensions.NETCore.Setup libraries, which might indicate a synchronization issue. Temporarily, I've added nuget.org to the nuget.config for local development.

It's policy that our product builds don't depend on nuget.org directly but rather on a mirror. It just means that when we need something, we mirror it. I kicked off a job to mirror the latest AWSSDK.S3 and AWSSDK.Extensions.NETCore.Setup and it should take a few minutes.

If you need other things or specific versions just let us know. This step is probably going to show up regularly in this repo by its nature, but as you say, you can unblock yourself by temporarily adding the nuget.org feed.

[davidfowl]

I aim to create a sample Aspire project using AWS SNS, SQS, and S3 by next week and produce a working POC. Depending on @normj's feedback, I can continue or possibly hand over the project to the AWS .NET team, leaving development to the professionals. 😁

This would be super useful. It would help clarify the differences so we can design the components and app model APIs. I'm intrigued by the conversation so far, thanks for kicking the tires here!

[Blind-Striker]

Thanks, @davidfowl, for the support! I hope those tires don't decide to kick back 😄

[Blind-Striker]

I'm nearly done with the first version of the POC. Probably, By the end of today, I'll have completed an API that can write to and read from S3 and can work both on both AWS and Localstack, along with an Aspire Host, under samples. I familiarized myself with Aspire better, and I'll share the insights I've gathered once the POC is finished.

[eerhardt]

The pattern we've been following is having a "component" library for each individual service. So I would imagine there would be one for S3, one for SQS, etc.

[eerhardt]

Check out #825 which is a good example of all the necessary steps to adding a full component.

[Blind-Striker]

Thanks for the information @eerhardt, I'll fix it in the next iteration.

[mmillican]

As a .NET + AWS guy, I'm happy to help test out. I will try to look through this PR more in depth over the weekend as well.

[davidfowl]

Is this needed?

[Blind-Striker]

Hey @davidfowl, I'm writing a detailed report on what I did so far. I'll answer your question

[Blind-Striker]

LocalStack Integration (@davidfowl ):
When using LocalStack, we need to run a LocalStack container before provisioning. I'm currently unsure how to do this. My next step will involve creating a ContainerResource for LocalStack. For now, I've added a docker-compose.yml file; running it first brings up LocalStack, followed by running Aws.AppHost to observe the creation of an S3 bucket.

I'll fix it in the next iteration. I'm trying to figure out how I can run a docker container in the provisioning stage.

[davidfowl]

If local stack run as a container that has to run before your code runs, then we need to add a new life cycle event after everything starts.

[normj]

My overall concerns are

• The AWS X resources don't really do anything. They are just names of resources to create with the provisioner.
• I don't think in the provisioning we would want to create the resources not tied to a CloudFormation stack. It would be a hard manual clean up process when you want to tear things down.
• This is using hard coded resource names so multiple developers would run into a lot of collisions reusing the same resources.

My limited understanding of Azure is it looks like all of the provisioned resources are in a resource group. That way @davidfowl could have one resource group and @eerhardt could have a separate resource group with no collisions to worry about. When either one of them are done they could delete their resource group to tear everything down.

In AWS a CloudFormation stack is the tool we have to handle this. A major difference with CloudFormation is the best practice is to let CloudFormation generate the unique names so multiple developers or environments can use the same account with no collisions.

My rough vision is users would define the logical resource names they want to use in the AppHost. Provisioning will create all of the application resources under a CloudFormation stack and then provide to the applications the mapping between the logical and physical name/ids.

There are challenges in that we would need the dev loop cycle to be quick dealing with the CloudFormation stack. Somehow detect that there has been no changes to the stack's template since it last ran and then skip provisioning process. Also we would need to come to some convention how to present the logical to physical names to the application.

[Blind-Striker]

With the latest commits, now we have a functioning POC 🚀

• CloudFormation output support added to AWS Constructs.
• AwsProvisioner now sets related properties for resources after creating the CloudFormation stack with the help of provisioners.
• Access info properties from these resources are injected into the example application by Aspire as connection strings.

Although we now have a working POC, this PR is far from production-ready. There are pending TODOs, and the Provisioning library requires refactoring. I've learned much about Aspire internals along the way and would approach the design differently with my current knowledge. For instance, the relationship between Constructs and Resources and setting resource parameters could follow a much cleaner and more efficient method. However, the POC has achieved its goal.

Areas Where I Need Help:

• Still encountering the CS8002 error for LocalStack libraries. @danmoseley @eerhardt
• Uncertain if AWS dependencies have been added to mirror NuGet sources. @danmoseley
• Need assistance with launching the LocalStack container before provisioning. @davidfowl
• General reviews and suggestions would be greatly appreciated. @normj

Plans for the Next Iteration:

• Focus on documentation, refactoring, and making the code more readable, maintainable
• Focus on the Aspire.AWS.* project: I used the LocalStack library directly in Aws.UserService for the POC and bypassed components for now.
• Although ConnectionStrings are currently functional, they don't align with the AWS resources usage scenarios. I plan to add AWS-specific WithReference methods to ResourceBuilderExtensions. Also, I'm thinking about adding suitable option objects on the component side.
• Currently, S3, SNS, and SQS are being created. In the next iteration, I'll add DynamoDB support and a console application to process items in SQS, completing the POC.

How to Run the POC:

First, you need to run docker-compose.yml in samples/aws/. This brings up and configures the LocalStack container. I found this workaround until the issue of running the container before provisioning was resolved.

• Run the Aws.AppHost project.

After the dashboard appears, you can use the following commands to see the AWS resources created on LocalStack:

awslocal s3api list-buckets
awslocal sns list-topics
awslocal sqs list-queues
awslocal cloudformation describe-stacks --stack-name AspireAwsSampleStack

awslocal is a wrapper around the aws command line for use with LocalStack, developed by LocalStack team.

• Open the Swagger of Aws.UserService from the dashboard.

You can send the following payload:

JSON:

{
  "Name": "Deniz İrgin",
  "Email": "deniz@deniz.com",
  "ImageName": "profile.jpg"
}

CURL:

curl -X 'POST' \
  'http://localhost:5099/profile' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "Name": "Deniz İrgin",
  "Email": "deniz@deniz.com",
  "ImageName": "profile.jpg"
}'

At this stage, Aws.UserService will upload an image to S3, then send the Profile model to an SNS topic, which will be queued in a subscribed SQS Queue. I embedded an image in base64 encoded to a constant variable, I use that image for uploading to S3.

• Observe AWS resources;

Since there's no application listening to the SQS currently, you can check the number of items in the Queue and dequeue them individually with the following commands:

awslocal sqs get-queue-attributes --queue-url "http://sqs.eu-central-1.localhost.localstack.cloud:4566/000000000000/AspireAwsSampleStack-ProfilesQueue-a2da3b22" --attribute-names All

awslocal sqs receive-message --queue-url http://sqs.eu-central-1.localhost.localstack.cloud:4566/000000000000/AspireAwsSampleStack-ProfilesQueue-a2da3b22

To see the uploaded image:

awslocal s3api list-objects --bucket "aspireawssamplestack-profilepicturesbucket-d703f1c8"

And that's it. 🙌

The past few days have been quite intense for me. I plan to slow down but will continue working on this PR. I can't thank you enough for your ideas and support throughout this process 🙏

[normj]

Here are some quick comments but I still need to go through in depth.

We need to separate LocalStack from AWS into different packages. LocalStack is a separate product by a separate company and Aspire must not give the impression that LocalStack is something owned or maintained by AWS. This is nothing against LocalStack and I recognized it is a popular product we just need to avoid confusion of who owns and maintains LocalStack. LocalStack parts should be in a package like Aspire.Hosting.LocalStack.AWS where LocalStack comes first so as not giving the impression it is a sub product of AWS.

You might want to look at my recent PR #975 for integrating DynamoDB local. It has to deal with similar problems that you probably have with Localstack with spinning up the emulator container and have SDK service clients connect to it. In particular where I'm setting the AWS_ENDPOINT_URL_DYNAMODB environment variable DynamoDB local's endpoint in hosting for the service clients to pick up. The service specific endpoint environment variables is fairly new. There are things in I call in the PR I still need in Aspire to make it better for launching the container.

For Aspire I'm not set on using AWSSDK.Extensions.NETCore.Setup. That library has gotten a bit dated as I wrote it when .NET Core was still in beta. Plus that library is just about pulling in configuration and DI. But following Aspire's goal of opinated best practice I think we would want to have the service client automatically enrolled with OTel via the OpenTelemetry.Instrumentation.AWS package.

Service client injection is the next thing I plan on playing around with if you want to hold off on this.

[normj]

I'm really worried about the approach of provisioning. I can appreciate having everything in typed classes but having another layer of abstraction that is manually maintain will be impossible to keep up with. That is why we have to rely so much on code generation. I'm sure the Aspire team do not want to sign themselves up with daily releases like we have to do with the SDK. We need a solution that allows users to take advantages of new AWS features without manual updates required here.

What if instead of a single AWS provisioner package we have different provisioners depending on the provisioning engine. For example we could have Aspire.Hosting.AWS.Provisioning.CloudFormation which uses a CloudFormation template in the AppHost project. Then we could have Aspire.Hosting.AWS.Provisioning.CDK which takes a reference to a CDK project in the solution. That would also pave a path for Terraform or Pulumi users as well.

Also, and no knock to the Azure Provisioner, but my feeling this is the time for the Aspire project to experiment and we don't need to attempt to follow the same pattern done there.

[Blind-Striker]

We need to separate LocalStack from AWS into different packages. LocalStack is a separate product by a separate company and Aspire must not give the impression that LocalStack is something owned or maintained by AWS. This is nothing against LocalStack and I recognized it is a popular product we just need to avoid confusion of who owns and maintains LocalStack. LocalStack parts should be in a package like Aspire.Hosting.LocalStack.AWS where LocalStack comes first so as not giving the impression it is a sub product of AWS.

I understand and fully agree with these concerns. The primary purpose of LocalStack is to ensure compatibility with official AWS SDKs, CDKs, and CLI commands. Essentially, what LocalStack.NET does is configure official AWS SDK for .NET Clients to work with LocalStack containers, basically changing their configuration to target a LocalStack container.

What if instead of a single AWS provisioner package we have different provisioners depending on the provisioning engine. For example we could have Aspire.Hosting.AWS.Provisioning.CloudFormation which uses a CloudFormation template in the AppHost project. Then we could have Aspire.Hosting.AWS.Provisioning.CDK which takes a reference to a CDK project in the solution. That would also pave a path for Terraform or Pulumi users as well.

This is an excellent idea and has broadened my perspective. However, whether we use CloudFormation, CDK, or even support Terraform or Pulumi, every solution using official AWS APIs will inherently work with LocalStack by default. To this end, I'm considering the Aspire.Hosting.LocalStack.Provisioner library to act more like a wrapper, an extension for other AWS provisioners, configuring them to target LocalStack and initializing a LocalStack container before provisioning.

Similarly, for future client libraries like Aspire.AWS.S3, Aspire.AWS.SNS, etc., the Aspire.LocalStack component should act as an extension to configure registered AWS Clients for LocalStack. This is precisely what LocalStack.NET project does.

I acknowledge that the title of this PR may currently be misleading, and I plan to revise it to reflect its scope and objectives more accurately. Thank you, @normj, for your insightful comments. They have helped clarify the direction of this PR. I am open to feedback on the proposed use of Aspire.Hosting.AWS.Provisioning.CloudFormation and Aspire.Hosting.AWS.Provisioning.CDK and look forward to implementing @vlesierse's promising solution.

Thank you once again for your valuable input. 🙏

[davidfowl]

Also, and no knock to the Azure Provisioner, but my feeling this is the time for the Aspire project to experiment and we don't need to attempt to follow the same pattern done there.

Agree. No need to copy it and in fact, we've been discussing this same problem (the scaling problem) with the azure sdk team. We want to ideally bridge into a model where aspire doesn't need to model all of azure 😄. Still lots of work to do in this space and ideally, we'd have something that scales from dev/test scenarios to prod (in terms of what options you can set)

[vlesierse]

What we could do is to create a separate project which contains the AWS CDK stack(s) and constructs and basically describes the infrastructure. When those stack(s) get deployed, we can read back the ARNs, connection strings and secrets. The AppHost will be then be used to tight the application configuration and infrastructure together while keep them separated and don't blend/create different abstraction layers for the same resources.

This approach would work the same for CloudFormation, Terraform and other IaC solutions.

Currently I'm experimenting with some different things, like deploying the projects to ECS or Lambda, but I need some time to finish this and wrap it in a PR.

[ppittle]

we've been discussing this same problem (the scaling problem) with the azure sdk team. We want to ideally bridge into a model where aspire doesn't need to model all of azure 😄. Still lots of work to do in this space and ideally, we'd have something that scales from dev/test scenarios to prod (in terms of what options you can set)

@davidfowl - is there a Discussion on this or other way to discuss/follow what the team is thinking in this area?

[attilah]

This is an awesome contribution to Aspire @Blind-Striker thank you!

Regarding the deployment part, is that far fetched to not to integrate CFN, CDK or any other deployment technology, but generate the manifest file with all the required resource configuration as a result and then build LocalStack, CFN, CDK, WingLang, whatever deployment one would like?

[davidfowl]

OK @normj made a new PR. Lets figure out how we can make the local stack experience work with the native AWS experience.

[davidfowl]

@Blind-Striker is there still interest in localstack in aspire?

[Blind-Striker]

Hi @davidfowl, I remain committed to the project and follow @normj's progress. Closing this pull request and proceeding with Norm's new one would be best. I still need the AWS dependencies to be added to the project and a base for incorporating LocalStack. Norm and I had a brief discussion before the start of the year, where we decided on this approach. But I'm open to suggestions

[davidfowl]

@karolz-ms is it possible to represent this today with dcp?

[davidfowl]

Ping @karolz-ms @danegsta

[davidfowl]

@Blind-Striker I think we have a pretty good model shaping up here with the latest changes being made to aspire and @normj's changes as well. The first thing we want to do is rebuild this PR on top of what @normj has done in the cloudformation branch. Ideally we would be able to use the same APIs to execute cloud formation templates against local stack (it's insane that this works 😄).

I don't know if the AWS client as it exists in Norm's PR works with local stack, but if it does it would be a great way to show an end to end based on cloud formation.

Then from there, we can weave in CDK constructs and higher level AWS primitives (probably starting with S3?).

[Blind-Striker]

Hey @davidfowl ,

Thanks for the headsup 🙏

First thing on Monday, I'll check @normj 's PR and need to get in sync with both latest development in Aspire and Norm's have done. But I'm not sure if we can proceed from this PR

At the company I'm working with, we heavily utilize LocalStack, particularly since our entire IaC pipeline relies on AWS CDK, which generates CloudFormation templates. Thus, we've successfully set up numerous AWS resources in LocalStack without encountering any issues, aside from some edge cases. I'm especially confident that we won't have any troubles with fundamental services like S3, SQS, and SNS.

[normj]

I don't know if the AWS client as it exists in Norm's PR works with local stack, but if it does it would be a great way to show an end to end based on cloud formation.

My assumptions is this should work doing something like this. We will probably want some extensions methods to make this more seamless. And figure out how to get the ServiceURL from the LocalStack instance that was just spun up via the AppHost.

var cloudFormationClient = new AmazonCloudFormationClient(new AmazonCloudFormationConfig
{
    ServiceURL = "http://localhost:3001" // Where ever LocalStack is running
});

var awsResources = builder.AddAWSCloudFormationTemplate("AspireSampleDevResources", "app-resources.template")
                        .WithParameter("DefaultVisibilityTimeout", "30")
                        .WithReference(cloudFormationClient);

[Blind-Striker]

Hi @davidfowl,

Due to my workload, it looks like I won't be able to work on this PR for another 1-2 weeks. From what I understand, you want to introduce LocalStack support as soon as possible, with or without me. How should we proceed in this situation? If you have a deadline in mind and my involvement is delaying the feature release, I'm open to contributing by reviewing your PR or supporting in any other way you find helpful. Besides that, as I mentioned, I am committed to the project and can continue working on the PR at my own pace. However, I don't want to be a roadblock to the feature's progress.

[davidfowl]

Maybe someone else could pick it up? We're pretty crammed at the moment for the next preview (which is going to be our last preview with big features). It's fine to delay for a future release. The most important thing at the moment is to find issues that we can fix with the model that may prevent this from working. That's why these spikes are great.

If there's anyone else that can take a look at this it would be great.

[davidfowl]

@Blind-Striker any updates? We have main opened up for an 8.1 release. We can continue to iterate on this again once you have time.