Add Multi repo support

[dosper7]

In several companies that use microservices have their own cloud platform teams and normally this "translates" into having a service per git repos.

In this cases how to use .NET Aspire? Meaning, different projects from different repos? It's not common to have multiple services within the same repo (unless you have a monorepo like Google 😆)

What is your suggestion?

[davidfowl]

Clone multiple repos locally, create an Uber solution and add the apphost to it, wiring up all of the projects required.

That's where I would start.

it you don’t want to pull and run the other repos locally, then I’m not sure exactly what the workflow would be (point to something already running).

[DamianEdwards]

Some patterns I imagine might emerge:

• git sub-modules for service repos and a parent repo with the .sln file and AppHost project, if you want to build and run all services locally
• Services published as container images that are then composed into an AppHost project as containers, if you don't want to build the services locally but you do want to run them locally
• Dev-instances of services deployed to dedicated environments and/or locally that are then composed into an AppHost project as HTTP endpoints, if you don't want to build or run the services locally

[bbrandt]

I believe we would need to publish our service defaults as a NuGet package, published to our own private NuGet server (Azure Artifacts in our case). Then projects throughout all the different repos could add a reference to this service defaults NuGet package.

[contione]

using NuGet might complicate debugging and hinder quick updates.

[DGAISmith]

Although this is true, how often would you realistically want to change/debug the service defaults? And when you do - when we want to debug the code delivered via nuget package, we just take a local clone of the repo and swap the nuget reference out for a project reference in the solution so it's fairly simple to get a local setup ready for debugging. So it is a complication, but a small one and the benefit of sharing the defaults via nuget outweighs it.

[scabana]

Always dreamed of having a primitive that would look something like

<ProjectReference Include="../Foo/Foo.csproj">
  <FallbackPackageReference Include="Foo" Version="1.0.0" />
</ProjectReference>

Although, I do also think it would make things much harder to have a mental model of what's going on, have good build reproducibility.

Maybe something like this:

<PackageReference Include="Foo" Version="1.0.0" ProjectReference="../Foo/Foo.csproj" />

And a parameter like dotnet run --use-local-projects. But I digress, back to the main topic here:

With that said, having the service defaults as a project reference for smaller codebases makes sense, but I feel it doesn't grow well into a large system of composed services. Also, you may not want to push a fundamental service default change across the whole system at once since those changes tend to have impact. The solution of moving it to a NuGet enables the gradual rollout of those changes across a system while re-using knowledge, behaviors and expectations of devs towards NuGet upgrades in their system.

[DamianEdwards]

I believe we would need to publish our service defaults as a NuGet package, published to our own private NuGet server (Azure Artifacts in our case). Then projects throughout all the different repos could add a reference to this service defaults NuGet package.

Yep, or you could put the ServiceDefaults in a sub-module and include it that way. There are lots of ways for folks to share code and assets among teams and I don't think we want to be too prescriptive about it. We should ensure we're flexible enough to support different patterns.

[artyom-p]

@DamianEdwards would be nice to have some samples on how to proceed with multi-repo scenarios. It is definitely a very cool and useful tool, but a huge part of microservice solutions are split across different repos, and in this case its not that obvious how to utilize it in local environment.

[dosper7]

I also don't understand how the deployment story is in this scenario, where we "run" our k8s cluster with a custom cloud platform team.

For instance, we use github + tekton pipelines + argo CD to deploy to k8s using multiple repos (one per service), all things k8s related are generated/managed by the platform (I've worked in previous companies with similar setup bu using different tools like gitlab as an example)

How Aspire can be helpful in this cases?

[davidfowl]

@artyom-p

@DamianEdwards would be nice to have some samples on how to proceed with multi-repo scenarios. It is definitely a very cool and useful tool, but a huge part of microservice solutions are split across different repos, and in this case its not that obvious how to utilize it in local environment.

Are you cloning everyone's repositories and running them today?

[davidfowl]

@dosper7

I also don't understand how the deployment story is in this scenario, where we "run" our k8s cluster with a custom cloud platform team.

It sounds like you don't need to use the deployment part of aspire because you have already figured it out! It's still great for local development, using components, and service discovery!

[artyom-p]

@davidfowl Yes, we clone all the repositories locally and debug service communication having multiple studio instances opened. Its doable, but its a pain.

[davidfowl]

@artyom-p What does the pain free solution look like? What would you ideal experience be?

[artyom-p]

@davidfowl Let's say we have 2 microservices that need to communicate with each other, they live in their own git repositories.

Each microservice can have its own or shared dependencies, for example:
ServiceA depends on ServiceB, Redis and MongoDB
ServiceB depends on MongoDB

There are cases when you want to debug only one service, but you still need another one to be running with all its dependencies, in this case, we can just somehow reference that second service and make it run, I think there was something similar in Tye before if I'm not mistaken.

The second case is more complex - when you want to debug both services. I'm not sure if it's doable without referencing projects of ServiceB in ServiceA solution.

[DGAISmith]

We also have the same multi-repo approach here, and this is where Tye was really strong for local development. A tye.yaml file could be relatively pathed to target x services, all cloned locally. Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire looks so much better in many ways to Tye, and I can see it's trying to achieve different things, but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

[davidfowl]

There are cases when you want to debug only one service, but you still need another one to be running with all its dependencies, in this case, we can just somehow reference that second service and make it run, I think there was something similar in Tye before if I'm not mistaken.

You can do a similar thing with the app host. Clone multiple repositories and create a project file that references individual projects. Maybe we need to enable referencing app hosts themselves 🤔.

Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire does require .NET 8 and above.

but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

Can you show me your tye setup?

[scabana]

You can do a similar thing with the app host. Clone multiple repositories and create a project file that references individual projects. Maybe we need to enable referencing app hosts themselves 🤔.

That could be a solution! In our case, we have git repositories per service in a system. We expect to do the following:

• For a specific service:
  • Run/Debug locally
  • Run tests in an isolated way, having its dependencies handled through Testcontainers
  • Generate a GitOps (argocd) application yaml (trying to keep it simple)
• For a group of service, or a system:
  • Generate a graph of dependencies across services
  • Stop deployment (which are independent per service) if a dependency is broken/missing

I have the impression we could generate a strongly typed dll/nuget that would expose another app host's resources which would enable us to have compile time errors. This would come with a challenge for handling circular dependencies, generated constants representing the resources from another app host could be enough.

I'd love to be able to use Aspire here. We have a few hundreds of services, thinking about having them all cloned with the right version is not practical. I'd rather have Renovate / Dependabot update nuget refrences and have build pipelines validate the integrity of the system.

[davidfowl]

What are you trying to accomplish though? Is the goal to run with another team's dependencies?

[scabana]

The end goal is running services with their dependencies in production for sure! With that said, when services are ran locally, they either have mocks for dependencies or they act a one of the services within an actual deployment (far from ideal since you can get traffic from other people on that system). One of the reason to interconnect the services from different repositories/teams is to have a build time check that all interconnections between the service are valid, ie: Foo service that required Bar service's url should fail at build time if the Bar service has not been declared in the system. In a distributed system, it would also enable having a shared redis cache deployment while keeping the ownership of that part outside of the team's boundary (for example, a platform engineering team could expose their resources and service teams could consume those in a safe way, not just a magic configuration key that comes from who knows where).

[DGAISmith]

Many thanks for the reply, really enjoying the Conf this year.

Tye was also useable as a completely external tool, no changes were required to any service, and they could also target a variety of different dotnet versions.

Aspire does require .NET 8 and above.

You say that, and of course you're totally correct if you want to use everything in the Aspire stack (and who am I to argue with the mad professors who came up with this amazing stuff), but I have been able to hack something together today from the starter application template using AddExecutable():

var builder = DistributedApplication.CreateBuilder(args);

//var apiservice = builder.AddProject<Projects.AspireApp2_ApiService>("apiservice");
var apiservicePort = 5323;
var apiserviceScheme = "http";
var apiserviceHostname = "localhost";

var apiservice = builder
    .AddExecutable("apiservice", "dotnet", @"..\AspireApp2.ApiService6", "run", $"--urls={apiserviceScheme}://{apiserviceHostname}:{apiservicePort}")
    .WithServiceBinding(apiservicePort, apiserviceScheme)
    .WithOtlpExporter();

builder.AddProject<Projects.AspireApp2_Web>("webfrontend");
    // .WithReference(apiservice);

builder.Build().Run();

In this case, ApiService6 is a dotnet6 api, inside which I've added OpenTelemetry support based on the code in the ServiceDefaults project. You do then end up with traces and (partial) metrics in the Asipre dashboard. So this is potentially useful-ish (I should try this with func.exe as well to see if we can spin up Functions in a decent way while Function support isn't quite ready yet). Service discovery is obvs broken here, and Aspire Components are presumably off the table too.

This serves two purposes though - allows some use of older dotnet applications, and also allows targeting of different repos outside of the solution the Aspire AppHost lives in. So it falls back to a similar kind of experience to Tye in these respects.
Probably not what you had in mind when you designed Aspire of course, but it's really nice to get some of the new dashboard experience with older projects.

but being able to use Tye as a non-invasive external local dev orchestrator remains a huge benefit and time saver for multi-repo scenarios.

Can you show me your tye setup?

Sure, we have a lot of tye files, each targeting a specific surface area of our microservices (which are all in individual repos). Redacted example below:

name: Portal Backend Frontend
extensions:
- name: zipkin
- name: seq
services:
- name: Portal-UI
  executable: cmd
  args: '/c npm start'
  workingDirectory: ./MyCompany.Portal/src/Portal
  bindings:
    - host: localhost
      port: 80
      protocol: https
- name: Identity-Api
  project: ./MyCompany.IdentityApi/src/MyCompany.IdentityApi/MyCompany.IdentityApi.csproj
  bindings:  
   - port: 5000
     protocol: https	  
- name: Portal-Api
  project: ./MyCompany.PortalApi/src/MyCompany.PortalApi/MyCompany.PortalApi.csproj
  bindings:  
   - port: 19081
     protocol: http
- name: DataPull-Function
  azureFunction: ./MyCompany.Functions/src/MyCompany.DataPull.Functions
  bindings:
  - port: 7071
    protocol: http      
- name: DataMapper-Function
  azureFunction: ./MyCompany.Functions/src/MyCompany.DataMapper.Functions
  bindings:
  - port: 7072
    protocol: http      

Key thing here is that we haven't made any code changes to any of these services, and they're spread over 4 different repos.

[DamianEdwards]

@artyom-p I've opened dotnet/aspire-samples#35 to track adding some samples of multi-repo orchestration from an AppHost project.

[davidfowl]

Still missing lots of details. Is the apphost running somewhere that app this team A and B are able to "register" with the app host? Is each developer supposed to do this or does this happen once. Is the app deployed or is it running on a developer's machine?

Is team B running team A's code? How does that work?

[luizbon]

Still missing lots of details. Is the apphost running somewhere that app this team A and B are able to "register" with the app host? Is each developer supposed to do this or does this happen once. Is the app deployed or is it running on a developer's machine?

Is team B running team A's code? How does that work?

My idea was more of a concept, not an actual definition.
But based on your questions, here is what I think.

AppHost is a service that can run locally or in a centralised environment.
If running in a central place, each team can connect to the same host and execute the registration of each service. This means that each team would have their services up and running for each other. This could be an integration testing environment.
Developers can hook their development versions to that host and work like dev containers. Since Aspire controls service discovery, it can also control routing to multiple instances and versions like a service mesh solution.

Another option would be to have the AppHost running locally. In this case, developers will need to register each other teams service.
For this, it could have two options: team A grabs the manifest from team B and vice-versa, just like an OpenAPI document. Then, register in their local AppHost and create a mock server. (Another feature Aspire can provide)
The second option would be to download a version of the other team's app via a package/docker image or the code itself. And proceed with regular service registration.

Each approach will depend on how development teams like to work, but the important thing is the stack will not impose how teams should work.
Aspire wants to be opinionated, but having the flexibility enables current solutions to integrate with Aspire and use the opinionated version for green field solutions.

[davidfowl]

I've added #1071 so far to make it a bit easier to this (#829 (comment)). Paths are relative to the app host project to make it easy to specify relative paths to the project. This should enable more experimentation.

[bastiennoel93]

Hi @davidfowl ,
I use the latest Aspire builds with your feature.
But I get the following error when I run the project with my external project reference:

Have you ever encountered this problem? Can you tell me more please ?

[davidfowl]

My idea was more of a concept, not an actual definition.

I hear you, but I think we need to explore something in the realm of the feature set of the app host. The app host is never deployed, it's a local orchestrator only. It can run executables, containers, projects, cloud and reference remote resources (deployed elsewhere). The dashboard is also an OTLP server so telemetry is pushed to it with a single env variable config. With those primitives, it should be possible to build various workflows with these primitives.

We're still working through a couple of issues like enabling docker networking and making it easier to reference things within containers (so you can run your another team's container images for e.g.). Hopefully that'll be enough for you all to experiment and come back with feedback about what works and what doesn't work.

AppHost is a service that can run locally or in a centralised environment.
If running in a central place, each team can connect to the same host and execute the registration of each service. This means that each team would have their services up and running for each other. This could be an integration testing environment.
Developers can hook their development versions to that host and work like dev containers. Since Aspire controls service discovery, it can also control routing to multiple instances and versions like a service mesh solution.

I think we're pretty far away from this based on the current set of capabilities. That sad, maybe you could build something on top of these primitives to get the next level of understanding for what this workflow looks like.

[luizbon]

The app host is never deployed, it's a local orchestrator only.

That makes complete sense.
I want aware of this premise. Not sure if it is documented and I missed that.

Regarding my idea, I don't think it is aligned with the project goals then, but also can be implemented as an extension on top of it.

[davidfowl]

Regarding my idea, I don't think it is aligned with the project goals then, but also can be implemented as an extension on top of it.

Maybe consider other interesting ways to connect services like using an external service discovery system (dapr, consul) and the app host to bridge local and remote development.

Maybe you can use a shared configuration server (something like azure app config in the azure case) as a way to store service discovery information across teams.

The problem is the code needs to be executed (that means access to another teams' code or assets), or it needs to be running remotely.

[atrauzzi]

👋 -- Hello! Chiming in late here, but I was thinking: Why not make it possible to make a reference a git repo URL? With something like that, Aspire could clone and then either discover or be explicitly told what type of project is within for it to manage?

This would be a game changer, particularly for onboarding scenarios. I've followed a pattern of having a "home" repo that houses a docker-compose.yml definition that presumes the presence of other repositories in the past. Aspire is in a great position to improve on such a concept.

[dosper7]

@atrauzzi that seems good idea (just don't know about the execution part), but would definitely be helpful since the repos url is "always" there. It seems an approach like Go or Deno do with the "usings/imports".

Maybe something like this pseudo code:

var builder = DistributedApplication.CreateBuilder(args);
var apiservice = builder.AddGitProject("www.github.com/FooCompany/BaaRepoName").WithName("apiservice");

What do you think? Crazy stupid idea?

[davidfowl]

👋 -- Hello! Chiming in late here, but I was thinking: Why not make it possible to make a reference a git repo URL? With something like that, Aspire could clone and then either discover or be explicitly told what type of project is within for it to manage?

Because I don't think aspire should manage git repos. Pulling, updating, picking the right branch etc etc. That should be managed externally.

This would be a game changer, particularly for onboarding scenarios. I've followed a pattern of having a "home" repo that houses a docker-compose.yml definition that presumes the presence of other repositories in the past. Aspire is in a great position to improve on such a concept.

Right, I think it should be possible to have an aspire app host reference source anywhere but it shouldn't be responsible for the cloning and updating of the git repo.

[atrauzzi]

Yeah, I get what you're saying, it even makes sense. But I guess then it takes away some of that snappy convenience of being able to obtain the Aspire project itself and have things self-assemble.

If something like this were to exist, I don't think I'd want to see aspire go as far as branch management. Just clone and then put the developer back in the drivers seat, that's all.

[davidfowl]

If something like this were to exist, I don't think I'd want to see aspire go as far as branch management. Just clone and then put the developer back in the drivers seat, that's all.

That feels very incomplete IMO. I'm not sure we would want to ship a feature like that. I'm sure it could be built on top of our existing primitives, so if that's something interesting, somebody should build that and put up a nuget package.

[atrauzzi]

Oh! So that is to say, there's support for different source types to be added? And I guess then the rest of Aspire would (hopefully) pick up, assuming all went well?

[DamianEdwards]

The hosting primitives for DistributedApplicationBuilder are extensible, as long as any added resource eventually boils down to a container or an executable. So you could add overloads of AddProject() that accept git URLs or create a new AddGitProject() method that adds all the cloning logic but ultimately just adds normal project resources to the model.

[atrauzzi]

Love it. Something like .AddGithubProject("someoneororg/blabla") and then it would go cd ../../; mkdir "someoneororg/blabla"; cd "someoneororg/blabla"; git clone... (oversimplified!), and then do the regular add.

Although I agree it has an out-of-scopey flavour, I could see it being a very popular feature or extension. We'll see 🙂

[WhiteOlivierus]

I have made a little started on something like you mentioned. This is only for projects now, but since I used this workflow also for other systems when Project Tye was not yet dead, I think this could be nice to have.

https://github.com/Dutchskull/Aspire-Git

[davidfowl]

VERY COOL! Maybe you can model it as a specific resource:

var gitRepo = builder.AddGitRepository(...);

gitRepo.AddProject("api", ...);

We don't have a great place to build projects at the moment. I see you're doing it in the call to Add currently. Maybe it would be better to use a https://learn.microsoft.com/en-us/dotnet/api/aspire.hosting.lifecycle.idistributedapplicationlifecyclehook?view=dotnet-aspire-8.0

[WhiteOlivierus]

I will see what I can do. This was a proof of concept since I just started looking at Dotnet Aspire as replacement for my Project Tye setup. This will be expanded up on.

[davidfowl]

Agree! I love the experimentation; it'll help us design and build features into the product.

[bUKaneer]

Hey, I've been working on a project that I think may be useful for some here. Please feel free to send feedback and let me know what you think.

https://github.com/bUKaneer/white-label.starthere