Workflow for multi-environment management + promoting releases between environments

[savannahostrowski]

Developers may want different IaC for each environment (dev, test, prod).

• What is our guidance for managing that?
• How do we enable a developer to advance their app from one environment to another (e.g. dev -> prod)?

Note that using branches for management is sort of an anti-pattern/dated approach and there are developers who do not view this as an option (e.g. https://codefresh.io/blog/stop-using-branches-deploying-different-gitops-environments/)

[weikanglim]

I would lean heavily into our CI providers: GitHub and Azure Pipelines, and provide a mechanism that allows azd environments to easily be reused in those scenarios. I would also think about how we can build azd gestures that solve common problems around environment promotion, while ensuring that local workflow matches CI workflow as much as possible.

[wbreza]

@savannahostrowski @ellismg This likely should be required as part of our ADE integration

[ellismg]

What is our guidance for managing that?
How do we enable a developer to advance their app from one environment to another (e.g. dev -> prod)?

Our current strategy allows you to set per environment configuration for your IAC, so at a minimum you could imagine adding a new parameter to your infrastructure on what sort of environment type (dev, prod, etc) you were deploying to and your infrastructure template could conditionalize resources on that.

Or, you could just examine AZURE_ENV_NAME in your bicep and based on it's structure do things (perhaps your organization has a way to name environments that encodes some semantics).

I think that can get us very far today. To @weikanglim's point, you can define these values in your CI system (perhaps tied to branches, perhaps now).

It would be good to know where this falls over or where the rough edges are and we can start to sand them down.

@savannahostrowski @ellismg This likely should be required as part of our ADE integration

This sort of surprised me - given that some of the initial design work in the ADE stream led me to believe that in ADE mode we'd only allow a single environment per project. What do you think is going to push us to figure this out for ADE, @wbreza?

[wbreza]

This sort of surprised me - given that some of the initial design work in the ADE stream led me to believe that in ADE mode we'd only allow a single environment per project. What do you think is going to push us to figure this out for ADE, @wbreza?

ADE is all about separating out the governance and creation of infrastructure typically required in a more structured organization or enterprise. Devs won't have access to provision resources, but if the enterprise wants to leverage ADE + AZD for integration scenarios we should have a good story here.

ADE devs will likely only be working directly against lower level environment types like "dev" but outside of the dev focused scenario test/staging & prod environments still exist and should be supported in CI/CD scenarios.

Most of this boils down to correct environment configuration in Azdo or Github but once that has been setup azd should be able to easily leverage this configuration where calls to azd provision link to the correct ADE environments across the full phase of deployment rollouts.

[wbreza]

Multiple Environment Support in Azd (Work in Progress)

Proposal and use cases for multiple environment support in azd.

Use Case: Support multiple Azd environments in CICD

Today azd is limited to 1 environment definition via azd pipeline config command.

Both Github & Azdo support the concept of "environments". The concept maps directly towards environments in azd. It is a separation of configuration + azure resources that support deploying directly into each of these environments.

Note
Github environment are only available for public repos. Private repos require having a paid version of GitHub pro / enterprise

Today, azd creates repo level variables and secrets in Github and pipeline variables in Azdo.

Global Changes

1. Support optional --ci-environment flag to target which CI provider environment to use. Default convention would be to map to the same environment name as azd.
2. Default pipelines will need to be updated to reference environments, variable groups, etc.
   • You need to reference environments / variable groups by name and we would need to figure out if/how we do this within our default pipeline files unless we want to start getting into templating of pipelines.

Changes for GitHub

1. azd creates this new GitHub environment if it does not already exist
2. azd creates all secrets / variables within this environment

Warning
If environments aren't available for current repo default to single environment support?

Changes for Azdo

1. azd creates new AzDo environment if it does not exist
2. azd creates a new variable group for the environment (use environment name by default)
3. azd creates new secrets & variables within this variable group

Use Case: Use different infra for different environments

Q: I want to change the infrastructure that is provisioned per environment
A: For simple changes like supporting availability zones, different service SKUs, scaling settings users should leverage IaC template parameters to drive different deployment paths based on 1 or many input parameters. If param mode=dev use lower end skus. If param mode=prod use higher end skus.

Q: I want to use different service targets per environment. Container apps in dev, AKS in prod
A: This one can't be handled in pure IaC templates because the packaging and deployment needs to change per environment.

In this case users can author multiple versions of the azure.yaml. Base configuration can still exist in azure.yaml. azd will overlay additional changes for azure.{env}.yaml during project loading. Only deltas needs to be supplied in the environment specific version.

This provides the ability to swap out any configuration element per environment. IaC template paths, IaC providers and even services if needed.

Changes

• Detect environment before loading project
• Load root azure.yaml
• Load environment specific azure.{env}.yaml
• Overlay deltas from environment specific version over root version.

Use Case: Promoting app code releases between environments

For app code promotion we typically want to take the exact version deployed from a lower environment and promote/release it to an upper environment. This is important because this is the exact version that was tested and known to be in a good working state vs just re-building from your mainline branch.

Simple version

The simpler case relies less on Azd and more on writing the correct multi-stage pipelines in your CI provider of choice. This approach will do a direct code deployment to your configured service target.

1. Run azd run package <service> to generate the deployable artifacts for your services
   • We need better JSON output version to be able to programmatically get the artifact path from the package command.
2. Store artifacts in your CI provider
3. Upon promotion approval:
   • Download artifact from your artifact store
   • Run azd deploy <service> --from-package <package-path> referencing the appropriate artifact

Advanced version

• Add first class support to all our supported service targets for their supported deployment offerings.

Service Targets

• AppService
  • Deployment slots with swap capability
• ContainerApp
  • Revisions with ingress traffic routing

Add a --rollout or similar param to azd deploy to perform service target specific rollout of application over a period of time
...

[ellismg]

Overall, I love the proposals here and think they are a step forward in the right direction to having azd be a great way to manage Azure in CI/CD workflow which has multiple deployments of the application (production, staging, testing, etc)

A few comments based on what popped into my mind while reading.

If environments aren't available for current repo default to single environment support?

Yeah, this seems like about the best we can do. It is interesting, I think, that when we create a repository during pipeline config we create a private repository by default (IIRC), which wouldn't support this feature. Not sure how to think about that or if we should care.

Q: I want to use different service targets per environment. Container apps in dev, AKS in prod
A: This one can't be handled in pure IaC templates because the packaging and deployment needs to change per environment.

No opposed to a azure.<env-name>.yaml that overrides stuff from azure.yaml. Did want to note that I wonder if we could relax/remove the need to put the service type in azure.yaml at all or if long term we would want to try to detect this based on the target resource we discover for the service (based on tags, or some other strategy, like the infrastructure having exports of the form SERVICE_<name>_RESOURCE_ID or something).

I guess I also wonder in practice if anyone would actually do this. It feels sort of like an anti-pattern to have such a large change in the way your application is deployed between environments, not really sure how common such a thing would end up being in practice?

Run azd run package to generate the deployable artifacts for your services
We need better JSON output version to be able to programmatically get the artifact path from the package command.

I wonder if instead we should support something like azd package --package-path <path-to-file> and in this case, azd package just uses <path-to-file> instead of generating a filename itself. The pipeline author could then place the artifact wherever they please. This may be easier than running azd package --output=json | jq .packagePath or something.

Add a --rollout or similar param to azd deploy to perform service target specific rollout of application over a period of time

I do wonder again how much of this we could infer from the target resource. Would it possible to examine an AppService resource and see if it is using deployment slots and if so deploy that way? Similar for Container Apps, could we detect when the app is configured for single vs multi-revision mode and change our strategy based on that? There's also a question longer term if we'll want a way for someone to override what "deploy" means for a service to instead use some other deployment strategy (maybe something like spinnaker or some other deployment strategy that is not strictly tied to the service).

[pamelafox]

Is this the relevant issue to follow for deployment slots support for App Service?

A developer has asked how they can deploy to a slot using azd up: Azure-Samples/azure-search-openai-demo#1278

[jasontaylordev]

I did some work around this and created this template and some documentation. These demonstrate how I'm using AZD, GHA, and Bicep to create and configure support for multiple environments. I presented this topic at the Azure User Group: https://www.youtube.com/watch?v=nSebeKFG2s8.

As far as GHA is concerned, support for environment variables and secrets is needed. For AzDO, I have yet to dive as deep, but support for environments and variable groups is required.

I hope this is helpful; I am happy to discuss this further.

[mortcodes]

Are there any updates on this issue? Using azd to fastrack a developer from idea to Cloud dev environment is good, but I think there needs to be better support to take this through environments into production using pipelines or github actions with environment scoped secrets and variables. Even if it was one good example included in Awesome AZD templates similar to what jasontaylordev has done above, or with the current best approach from the azd team.

[pamelafox]

You can have environment-scoped secrets and variables in your pipelines now, by the way. We use them in https://github.com/Azure-Samples/azure-search-openai-demo/
There's a pending PR that adds a section about them to the azd pipeline documentation, as you need to modify both azure.yaml and your pipeline files to know about the variables.

[jasontaylordev]

You can have environment-scoped secrets and variables in your pipelines now, by the way. We use them in https://github.com/Azure-Samples/azure-search-openai-demo/ There's a pending PR that adds a section about them to the azd pipeline documentation, as you need to modify both azure.yaml and your pipeline files to know about the variables.

I had a review of the documentation here; https://learn.microsoft.com/en-us/azure/developer/azure-developer-cli/configure-devops-pipeline?tabs=GitHub#additional-secrets-or-variables.

This new feature seems to allow us to specify additional variables or secrets to include when running azd pipeline config. Regarding environment-scoped, the values will come from the current azd environment. If using GitHub as the pipeline provider, the necessary secrets and variables will be created, but they will be scoped to the repository rather than a specific GH environment.

This seems like a good feature, but it doesn't bring me closer to configuring multi-environment support for a GH pipeline.

I would like the azd environment to be mapped directly to the pipeline environment. This way I can create variables and secrets at the GH environment scope. Ideally, I would also like some variables or secrets to be repository-scoped. Then, it's just a case of creating a multi-stage pipeline to promote releases between environments.