[Spike] improve azure container app deployments to avoid error-prone and confusing multiple revisions

[vicondoa]

• Make sure you've installed the latest version using instructions in the wiki

Output from azd version
Run azd version and copy and paste the output here:
azd version 1.5.0 (commit 012ae73)

Describe the bug
Azure Container Apps with Azure Developer CLI currently have a major limitation in how it handles deployments. Deployments of updated revisions occur in two phases: provision where the spec for the container app is changed and deployment where the image is updated. This creates a problem when developers revise their service and bicep configs, as it often has dependencies between its code in its Docker container, environment variables, resources, mounts, health probes, load balancers, and more. If these dependencies are not deployed in the right order, deployments can fail, and this can possibly create an outage because the container app can't start up.

azd deploy will only replace containers[0].image with the new image. This undocumented behavior is a huge limitation in the case where someone would like to run the same image in a sidecar or init container. A developer shouldn't need to have a code change made to azd anytime they come up with a new way an image should be included in a a container app spec.

The current setup is confusing. It's unclear what the special exists behavior is for in the templates. In a team of 10 people, most of us were really surprised to find out that the bicep configs for a container app don't actually determine what image should be used.

We mitigate this issue by adding a post-deploy hook to rollout a third revision to copy the image on containers[0] to all other containers where it's needed. A container app should not need three revisions per deployment. Each revision is an opportunity of incompatibility between bicep spec, revision 1, and revision 2 that is very error prone. We have burnt many hours of engineer time working around this.

To Reproduce

1. create a container app that has two containers, where you would like it to use the built image on both
2. run azd up
3. notice how only the first container has changed in the container app

Expected behavior

azd up should only produce one revision per service. developer should be able to specify container app spec and how the images are used all in one place rather than the undocumented behavior and exists hack in the template. Consider how kubernetes deployments work with kubectl apply.

Environment
Information on your environment:
* Azure container apps
* python

[Petermarcu]

@ellismg is some of this related to the getting started scenario where some people don't have ACR or containers already in a registry to point to at the time of provisioning the ACA infra?

[vhvb1989]

@Petermarcu , I think this is currently a per-design limitation on azd+aca. The azd+aca model considers only one container per service, however, aca can take an array of containers.
azd doesn't know (yet), how to deploy multiple containers for the same aca.

We have this POC #3032 from @wbreza to support sidecar pattern for aca, but the conversation for this is still WIP.

In case of azd+.NET Aspire, we have this PR #2992 adding support for azd to deploy dapr components (sidecars), which is not the same, but a related strategy

[wbreza]

@vicondoa - Thanks for all of your feedback. The initial implementation for ACA within azd targets the most common use cases we have seen from our customers.

Today we support a 1:1 mapping of service <-> container app where the code referenced in the service is built and deployed to a container app as the primary container image containers[0].

The initial implementation of ACA for azd followed the consistent azd model where infrastructure is defined in IaC (bicep) and code deployment during azd deploy. The ACA model introduced certain challenges here where container images could be referenced within the Bicep which has the ability to ship new revisions/code on provision in addition to deploy. The exists model was created here to help support this behavior by not forcing bicep to change the referenced images on any subsequent provision command that could force the app to reference an old or placeholder image.

We would consider the concept of using multiple containers, sidecar containers and init containers a more advanced usage scenario and rely on key customers like yourselves to help drive new features for azd and container apps.

We are considering new features to help support more advanced scenarios where multiple containers are used within an ACA deployment. Some of these paths are more of an AKS style manifest driven approach that will enable more scenarios like you described above.

Some ideas I'd like to get your feedback on for future development:

1. Move to a model where only the container apps environment is defined in bicep
   • Container apps themselves can be specified in Yaml style manifests
2. Ability to build/deploy containers to container registries as a 1st party service target
3. Expose a containers sections in azure.yaml that can be referenced by container based service targets like ACA & AKS

We need to create an easy to use default experience while ensuring the platform is extensible to support other use cases. An example of this is having auto-generated manifests for simple use cases where customers are using a single service/container but also provide flexibility to define your own manifest to drive any support ACA configuration you choose.

[vicondoa]

@wbreza. The ideas seem good:

1. I think moving to a model where only the container app environment is defined in bicep sounds like the best option if you want to keep the provision and deploy paradigm. The image can be made available to deploy using an SERVICE_<service>_IMAGE_NAME env variable and the dev can choose how to include that in the yaml manifest. I guess you'll need to provide a template language to help with the substitution and advanced conditional behavior.
2. this would have been useful when I was trying to work around not having something like the above item, but with the above items I wouldn't have to work around the deployment limitation.
3. I definitely could imagine a container app needing multiple built images, or different container apps sharing the same image. This could be useful.