Linkerd Proxy is now the default container on all pods since 2.11.0

[jasoares]

Bug Report

What is the issue?

Linkerd Proxy Injector changed the order by which it injects Linkerd Proxy container into the pod from version Stable-2.10.2 where it was last to version Stable-2.11.0 where it is now first. I'm assuming it's a bug since it does not seem to be documented anywhere.
Please feel free to correct me if this was an intended change (I'm curious about the motivation) or even if there's an option to change the order somewhere I couldn't find, and I did look for it.

How can it be reproduced?

Instal version Stable-2.10.2 and deploy any pod, main container will be first and linkerd-proxy will be inject after the main container. Upgrade to version Stable-2.11.0 and deploy the very same pod, now linkerd-proxy is the first container and main container is last.

Environment

• Kubernetes Version: 1.20.8
• Cluster Environment: Kops
• Host OS: ubuntu-focal-20.04-amd64-server-20210503
• Linkerd version: 2.11.0

Possible solution

Revert the behaviour change to what it was before 2.11.0 (linkerd-proxy is injected after main container) or add potential feature to define where injection should happen first/last should be more than enough for most use cases.
As an example Hashicorp Vault Agent injector allows the above setting for init containers. See here: https://github.com/hashicorp/vault-k8s/blob/f66bfc791717287288cbc0409131c34638e64fa4/agent-inject/agent/annotations.go#L88

Additional context

Main goal with this is that in my opinion the application/main container should always be first as it becomes the default container to be picked up for logs/ssh whenever a specific container is not explicitly specified, being this
particularly useful when using kubectl or Lens.

[jasoares]

cc @rmgpinto

[kleimkuhler]

This is a purposeful change that was missed in the 2.11 release notes; sorry about that! The change is explained in #5967.

In order to fix container startup ordering issues, we use the fact that containers are created in the order that they appear in the manifest. When linkerd-proxy is first, we can block the application container from becoming ready before the proxy. This allows the application container to startup and immediately start sending/receiving requests.

If you would like to disable this feature, you can use the config.linkerd.io/proxy-await: "disabled" annotation.

[jasoares]

Thank you so much for the quick response, we'll consider it internally. Closing this.