feat: Container build process improvements to reduce build times and improve Dockerfile organization

[nv-tusharma]

This DEP introduces a container build process optimization strategy for Dynamo to enhance developer experience and build efficiency by reorganizing Dockerfiles. We are introducing a new Dynamo base container which contains all the build logic to build Dynamo + NIXL wheels along with an executable base container (with Dynamo + NIXL pre-installed) that can be used for CI. We also define a structure for the backend-specific Dockerfiles and introduce other build improvements such as external caching.

[nvda-mesharma]

whenever we introduce a new container, customer has to do security scans, QA, etc. Do we want to reconsider the strategy to have backend specific containers?
LLM NIM has been doing this for a year now, one container with trt-llm, vllm, and sglang. Shall we also create a container with all backends included?

[nv-tusharma]

whenever we introduce a new container, customer has to do security scans, QA, etc. Do we want to reconsider the strategy to have backend specific containers?
LLM NIM has been doing this for a year now, one container with trt-llm, vllm, and sglang. Shall we also create a container with all backends included?

@nvda-mesharma It makes sense to have a single container to reduce complexity but there are a few things to consider with this approach:

• Is it feasible for a user to want deploy multiple inference engines at once with dynamo?
• Container size, if a user is only interested in one particular backend, we should remove the dependencies associated with other backends. We would need to provide tools for users to create backend-specific containers for their deployment.
• It is expected that Backends can differ on performance, which could be a result of dependencies.
• Build times, if we have to build each backend from source in one container, our build time would increase (since layers can't be parallelized without docker buildkit)

I think providing a Dynamo base image with minimal dependencies is sufficient for now but I'll add this suggestion in the Alternate Solutions section.

[whoisj]

@nv-tusharma please add an appropriate PR description.

[nv-tusharma]

Need to add some information about potentially publishing a base container which has NIXL + UCX pinned that we can build dynamo on top off. Since nixl is pretty much locked and does not change often and similar for other packages -- we could have a separate workflow that we can use to publish a public image that we can build on top of.
Not sure if we can do that on github or ngc, but something like -
Base image -- cuda+nixl+uv+cargo+..... We use this to build dynamo and other things
Dynamo image -- base image + dynamo rust/python builds
Framework image -- dynamo image + framework builds

[whoisj]

Overall this looks great. Added a comment about the readability of the Mermaid graph when using specified colors.

[whoisj]

LGTM, I'm happy with what we have here.