Deploying to Kubernetes: A Retrospective

[Christopher Brown (cdbrown2018)]

In the beginning...

Two years ago, my team began a new project. The project began simple enough -- just a UI and an API. But we knew the project would soon grow, requiring distributed services that were a mix of custom code, publicly available containers, and even commercial-off-the-shelf (COTS) images from industry peers. .NET Aspire 9.1 had just been released, was all the rage, and a coworker strongly recommended it. "It's so cool, Chris. You can just hit F5 and all of your services spin up. You can even deploy straight to Kubernetes using Aspirate!" It was the perfect fit.

Now, it's July of 2026, the (now renamed) Aspire 13.5 release is around the corner, and first-party Kubernetes support is in preview, and seemingly gearing up for GA. Aspire has grown a lot, and so has our project. This is the story of bringing our project from a local development environment to a full-blown Kubernetes project. We'll discuss the things Aspire does well, the things it can improve on, and my hopes for the future of the platform.

Deployment Environment

Our deployment story consists of multiple isolated environments, each considered its own deployment. Each deployment is an isolated Kubernetes cluster with custom admission and tenancy policies. These policies impose certain restrictions on the deployment manifests that if improperly configured will results in failed deployment validation by the Kubernetes controller.

Some of the services we deploy are only relevant in deployed environments, such as GitOps controllers. We don't have those represented in our App Model because there is no natural way to do so. These types of services are typically configured at the manifest-level, and have environment-specific configuration that can be extracted into values files.

Deployment Time

Let's run through the process of publishing Kubernetes artifacts. We will discuss the successes and shortcomings of the process as it is currently implemented.

We will start the publishing process like any good Aspire project

aspire publish

This successfully creates a set of Helm files. At first glance, these are great (and they are). However, a closer look will reveal several problems. Lets run through a few.

1. Deployments powered by Containers do not have configurable images
2. Custom manifests are hard coded with values
3. Secret management

Deployments powered by Containers do not have configurable images

The thought process for the current implementation of Containers makes a lot of sense. The containers are not custom code, and are hosted somewhere accessible to the developer's machine, and presumably to the deployment environment. This assumption breaks when the deployment exists in an isolated network.

An image must be pushed to an OCI repository accessible to the isolated machine. This is probably not a concern of Aspire. The process for that is going to look different from company to company. However, Aspire can make the process easier by placing the container's image names in the same parameters block in the values file that contains the image details for ProjectResources and others.

Custom manifests are hard coded with values

Each manifest might have values that differ based on the environment. These values should be able to be extracted into a values file so they can be overridden per environment.

Our current solution for varied configurations is using --environment to specify a custom environment name and then using look-up tables to properly assign values. This solution is simple enough, but it poses several problems:

1. Unnecessary code to maintain
2. Requirement for creating distinct helm charts for each Deployment
3. The setup is rigid and requires modifying code if an environment is added, changed, or removed

These concerns could be mitigated by allowing custom resources to be configured through Helm charts. This presents an edge case in Aspire's current implementation. Custom resources can absolutely be configured through environment variables. However, these are not compute resources. These are resources whose only responsibility is to produce a properly-formatted .yaml file.

There is currently no mechanism within Aspire to say "This custom manifest has a property that is variable and should be configured through a values file."

Secret management

In its current state, we cannot use Aspire to run a full aspire deploy. A consequence of this is the necessity of persistent Helm files, including secrets. The current setup for secrets makes a lot of sense in the case that charts do not need to be source tracked or stored on disk. However, it requires storing secrets in plain-text if you want to be able to use the charts.

To avoid plain-text secrets, our clusters use an external secret manager that supports runtime injection. This completely replaces the Kubernetes secrets engine.

All in, the following manual modifications must be made to Helm charts:

1. Pull out all 3rd party images into the values file
2. Add environment specific values files (i.e., values/values-deployment-a.yaml, values/values-deployment-b.yaml, values/values-deployment-c.yaml)
3. Pull out hard-coded configuration from manifests such as service meshes and GitOps controllers and place them in each applicable values file
4. Reconfigure secrets to use our external secrets engine.

These changes alone are not the end of the world by any stretch. My concern is how this breaks the idealistic Aspire deployment pipeline. The necessity of manual changes makes it impossible to run through the entire deployment scenario. Some of this can be worked around (GitOps and service mesh configuration could be worked around by creating look-up tables, hard-coded environments, etc.). However, those solutions require a significant amount of custom code that could be avoided by persistent values files (i.e., source-tracked values files that Aspire pulls from during aspire deploy), or through better use of ParameterResources, allowing for configuration through more native .NET configuration methods such as appsettings.json.

To my understanding, the 1st point about 3rd party image tags cannot be worked around as of 13.4.

What went well

My team had its production time moved up significantly, forcing a rapid deployment on a compressed timeline. Aspire enabled the team to focus cleanly on fixing bugs and implementing final features without concern for the deployment story. We had confidence through previous deployments that Aspire would get us 90% of the way there, and that we could complete the last 10% in an afternoon. We were proven right. It took a single afternoon of modifications to get the charts ready for deployment.

Tangential to this story, Aspire has sped up our on-boarding time immensely. Very late into our project, we introduced a new developer. He was a complete new hire. We spent more time getting his machine set up with organizational tools than we did getting it set up to run the project. He was able to gather an understanding of our project in record time due to the dashboard's customization features, such as the ability to hide service URLs, add custom commands, etc.

Summary

In order of priority, I believe the following use-cases would bring Aspire to the next level in the Kubernetes deployment space

1. Support for parameterized container image names
2. Integration with values files for CRDs and other custom manifests (Hook ParameterResources up to Helm-templated values)
3. Persistent values files
4. Support for third-party secrets engines

I am very excited about Aspire -- what it can do now, and what it will be able to do in the future. I hope this post provides insight into Aspire's strengths and weaknesses in the field, and I hope I can continue to contribute to the community and to the project.

I would be very interested in hearing other people in the community's experience with Kubernetes deployments, and any tips or tricks you all may have gathered.

[Tj (tjwald)]

Thank you for sharing! We were thinking of adopting aspire for automatic helm chart publishing, but we do need to also solve the secrets - how did you solve that?

[David Negstad (danegsta)]

Would it be fair to categorize as a common theme that you want a publish once, deploy many architecture? It seems like that's a common theme for a lot of the headaches you ran into like wanting parameterized image references so that they could be overridden for each environment deployment, wanting your VirtualService entries to be configurable via values, or needing support for ArgoCD configuration via values file.

Obviously that's not the only takeaway (support for custom secrets providers), but I get a feeling that you were sort of running into a few different variations on a similar shaped problem.

[Tj (tjwald)]

David Negstad (@danegsta) Definitely! We publish one terraform template and helm chart, and then each have a values file(s) that turn on or off specific services or infrastructure per deployment.

Mostly my team does this due to staging having the latest bits that aren't in prod yet, but other teams literally have different product offerings in different regions / customer tenants, and the way to set them is tfvars and values.yaml.

[Christopher Brown (cdbrown2018)]

I think that's a very fair assessment. However, my dream is to be able to use aspire deploy --environment=DeploymentA to be able to go end-to-end without having to manually modify a helm file. To my understanding, that's the goal the Aspire team holds as well?

To that end, it would still be required to be able to configure container images using ParameterResource so that values could be pulled in from appsettings.DeploymentA.json or from any other configuration provider, including the parameter store that Aspire creates and maintains.

[Pedro Brown (pedrobrown)]

We recently ran a local experiment with an existing application that already had a Helm chart, similar to the migration scenario described in Discussion #18741. The goal was to model the Kubernetes resources in the AppHost and compare the output of aspire publish with the existing chart.

We updated the Aspire packages and modeled the application resources directly in C#:

var kubernetes = builder.AddKubernetesEnvironment("k8s")
    .WithContainerRegistry(containerRegistry)
    .WithHelm(helm =>
    {
        helm.WithNamespace("sample-namespace");
        helm.WithReleaseName("sample-release");
        helm.WithChartVersion("0.1.0");
    });

var database = builder.AddPostgres("database")
    .AddDatabase("appdb");

var migrations = builder.AddProject<Projects.Sample_Migrations>("migrations")
    .WithReference(database)
    .WaitFor(database)
    .PublishAsKubernetesService();

var api = builder.AddProject<Projects.Sample_Api>("api")
    .WithReference(database)
    .WithReference(migrations)
    .WaitForCompletion(migrations)
    .PublishAsKubernetesService();

We then used the Kubernetes resource callbacks to customize the generated resources.

For example, we added an image pull Secret to generated workloads:

static void AddImagePullSecret(KubernetesResource resource)
{
    var secret = new LocalObjectReferenceV1
    {
        Name = "registry-pull-secret"
    };

    switch (resource.Workload)
    {
        case Deployment deployment:
            deployment.Spec.Template.Spec.ImagePullSecrets.Add(secret);
            break;

        case StatefulSet statefulSet:
            statefulSet.Spec.Template.Spec.ImagePullSecrets.Add(secret);
            break;
    }
}

We also added a ConfigMap and mounted an imported file into a generated Deployment:

static void AddImportedConfiguration(KubernetesResource resource)
{
    resource.AdditionalResources.Add(new ConfigMap
    {
        Metadata = new ObjectMetaV1
        {
            Name = "identity-configuration"
        },
        Data =
        {
            ["configuration.json"] =
                File.ReadAllText("configuration.json")
        }
    });

    if (resource.Workload is not Deployment deployment)
    {
        return;
    }

    deployment.Spec.Template.Spec.Volumes.Add(new VolumeV1
    {
        Name = "identity-configuration",
        ConfigMap = new ConfigMapVolumeSourceV1
        {
            Name = "identity-configuration"
        }
    });

    var container = deployment.Spec.Template.Spec.Containers
        .First(container => container.Name == "identity");

    container.VolumeMounts.Add(new VolumeMountV1
    {
        Name = "identity-configuration",
        MountPath = "/app/configuration.json",
        SubPath = "configuration.json",
        ReadOnly = true
    });
}

For StatefulSets, we added a headless Service through AdditionalResources:

static void AddHeadlessService(
    KubernetesResource resource,
    string componentName,
    int port)
{
    if (resource.Workload is not StatefulSet statefulSet)
    {
        return;
    }

    var serviceName = $"{componentName}-headless";

    statefulSet.Spec.ServiceName = serviceName;

    resource.AdditionalResources.Add(new Service
    {
        Metadata = new ObjectMetaV1
        {
            Name = serviceName
        },
        Spec = new ServiceSpecV1
        {
            Type = "ClusterIP",
            ClusterIp = "None",
            Ports =
            {
                new ServicePortV1
                {
                    Name = "tcp",
                    Port = port,
                    TargetPort = port
                }
            }
        }
    });
}

These C# customizations allowed us to reproduce most of the existing Helm chart. The remaining mismatch was the migration workload.

The existing chart uses a one-shot Job, while Aspire generated the migration project as a Deployment:

Existing chart:  batch/v1 Job
Aspire output:   apps/v1 Deployment

The migration process applies EF Core migrations, performs application-specific seeding, and exits. Therefore, a Deployment is not semantically equivalent to a Job.

We also tested the EF Core migration bundle API:

var migrations = api
    .AddEFMigrations(
        "migrations",
        "Sample.Infrastructure.AppDbContext")
    .WithMigrationsProject(
        "../Sample.Infrastructure/Sample.Infrastructure.csproj")
    .WithReference(database)
    .PublishAsMigrationBundle(
        targetRuntime: "linux-x64",
        publishContainer: true);

This did not solve the problem because:

• the generated workload was still a Deployment;
• the bundle did not include the application's custom seeding behavior;
• the generated bundle caused Helm packaging problems because of its size.

The possible workaround is a custom pipeline step after publish-k8s:

builder.Pipeline.AddStep(
    "customize-migrations-workload",
    async context =>
    {
        // Locate the generated Helm output.
        // Remove the migration Deployment.
        // Write a batch/v1 Job.
        // Preserve the generated image, parameters,
        // environment variables, ConfigMaps, and Secrets.
    },
    dependsOn: ["publish-k8s"],
    requiredBy: [WellKnownPipelineSteps.Publish]);

From this experiment we learned that the AppHost model is expressive enough for many existing Kubernetes resources, but is missing a typed way to declare or customize workload kinds such as Job and CronJob. The same applies to externally managed Secrets.

In summary, the most useful additions would be:

• typed Job and CronJob resource support;
• C# customization of generated workload kinds;
• reusable APIs for ConfigMap imports and volume mounts;
• support for externally managed Secrets and Secret providers;
• preservation of generated parameters and configuration when customizing resources.

Hope this helps. I'm completely new to contributing to OS, so please forgive me if this comment isn't relevant to this discussion.

[Tj (tjwald)]

Is it not possible to change the deployment to a job via the PublishAsKubernetesService callback?

[Pedro Brown (pedrobrown)]

I tested this in a session with an agent (GPT-5.5 medium) last week, on Aspire 13.4.6-preview.

The callback can modify the generated KubernetesResource, but there is no built-in Job workload type. We tried replacing the generated workload with a custom Workload("batch/v1", "Job") implementation.

Although aspire publish then emitted:

apiVersion: batch/v1
kind: Job

the generated manifest contained only metadata and no valid spec, Pod template, container, or environment configuration.

We also tried using AddManifest, but that API was not available in the package surface actually referenced by our AppHost.

So the callback cannot currently convert the generated Deployment into a valid Kubernetes Job. The generated workload logic appears to support Deployment and StatefulSet shapes only.

Maybe I missed something?

[Tj (tjwald)]

No idea - just wondering - maybe submit an issue?