Design multi-manifest publishing

[baronfel]

It's possible for containers to be specified in a 'manifest list' - a set of container image manifests that represent the same application on different underlying OS/hardware configurations.

Fundamentally this would be something like a multitargeted build. For some selection of OS/OSVersions and Architectures we'd need to orchestrate

• building the app for that os/version/architecture
• publishing a container image for that os/version/architecture

then, once all of those were done, we'd need to

• create a new 'manifest list',
• add each of the created images to the list,
• ensure each one was 'annotated' with the correct os/platform/etc annotations, and
• push the manifest list to the registry

There are a couple hurdles we'd need to cover:

• if we would like to perform this by default for projects that specify multiple RIDs, then we may need a concept of a cross-RID/cross-TFM publish. This doesn't exist right now, and in fact is explicitly stopped by the cross-targeting targets in the .NET SDK.
• if we modeled this as a different, standlone target, then we wouldn't need to tie it to 'publish' necessarily, but we might lose the benefit of associating with 'publish' as a concept
• I'm unsure how much users would use this - is it worth pioneering a potentially-new publishing concept?
  • This would bring us parity with Jib and Ko - maybe parity is enough of a motivator

[baronfel]

I discussed this at MVP Summit this year, and feedback from folks was strong - we should do this so that we have parity with other ecosystems.

[baronfel]

There are two separate requirements here:

• create images for each architecture/platform that a user specifies that has a match with the base image chosen
• bundle those platform- and architecture-specific images into a single manifest list and publish it

The former is relatively straightforward today. We essentially want to 'multi-target' like you would with a TFM, but with RIDs:

A Directory.Build.targets file with a Containerize target that enables multi-rid container generation

<Project>
  <PropertyGroup>
    <!-- We have to build Publish AND PublishContainer because PublishContainer (and other
        PublishProfile-delivered targets) don't have an explicit Publish dependency. -->
    <_RequiredContainerPublishTargets>Publish;PublishContainer</_RequiredContainerPublishTargets>
  </PropertyGroup>

  <!-- Entrypoint, either from solution-level `/t:Containerize` or project-level `/t:Containerize` -->
  <Target Name="Containerize" Condition="'$(EnableSdkContainerSupport)' == 'true'">
    <!-- Strategy here is that we will figure out what proejct(s) to build the containerization targets(s) for
         based on project state. We use `AdditionalProperties` to customize the outputs of each of the builds. -->

    <!-- Properties set here:
      * TargetFramework - multitargeting - changes inference for base image based on TFM
      * VersionSuffix - without either explicitly setting `ContainerImageTag` or influencing the tag in some way
                        (I chose `VersionSuffix` because it lets folks still customize the 'base' of the version
                        and follows a Docker-ish convention of arch-specific info adding to the end of the tag)
                        we'll get the same 'tag' for each image, which would cause the images to override when
                        pushed to a registry or local daemon. NOTE: this is currently the RID, but it _should_
                        be a golang-style platform string (e.g linux-amd64 instead of linux-x64).
      * ContainerRuntimeIdentifier - if we're building for a specific RID, we need to set this so that the
                                     containerization targets know what RID to build for
      * RuntimeIdentifier - if we're building for a specific RID, we need to set this so that we get optimized 
                            RID-specific assets in the publish output

      NOTE: we could get away with setting `RuntimeIdentfier` here to control `ContainerRuntimeIdentifier` inference
            but this is also nice and explicit.
     -->

    <!-- TFMs but no TF -> multitarget, making image for each TFM -->
    <ItemGroup Condition="'$(TargetFrameworks)' != ''
                          and '$(TargetFramework)' == ''" >
      <_TFMItems Include="$(TargetFrameworks)" />
      <_SingleContainerPublish Include="$(MSBuildProjectFullPath)"
        AdditionalProperties="TargetFramework=%(_TFMItems.Identity);
                              VersionSuffix=$([MSBuild]::GetTargetFrameworkVersion('%(_TFMItems.Identity)', 2))" />
    </ItemGroup>
    
    <!-- TF but no TFMs -> single image (aka the default pathway) up until now -->
    <ItemGroup Condition="'$(TargetFramework)' != ''
                          and '$(RuntimeIdentifiers)' == ''">
        <_SingleContainerPublish Include="$(MSBuildProjectFullPath)" />
    </ItemGroup>

    <!-- TF with RIDs -> multi-arch, single image per arch -->
    <ItemGroup Condition="'$(TargetFramework)' != ''
                          and '$(RuntimeIdentifiers)' != ''">
      <_RIDItems Include="$(RuntimeIdentifiers)" />
      <_SingleContainerPublish Include="$(MSBuildProjectFullPath)"
        AdditionalProperties="ContainerRuntimeIdentifier=%(_RIDItems.Identity);
                              RuntimeIdentifier=%(_RIDItems.Identity);
                              VersionSuffix=%(_RIDItems.Identity);" />
    </ItemGroup>
    
    <MSBuild Projects="@(_SingleContainerPublish)" Targets="$(_RequiredContainerPublishTargets)" BuildInParallel="true" />
  </Target>
</Project>

Adding this target to a project lets you run dotnet build /t:Containerize and generate architecture- and platform-specific images. We should look at including something like this in the official build targets for the 7.0.400 time frame if at all possible. Adding such a target also enables a related use case: containerizing every project in a solution that can be containerized. This enables workflows like dotnet build /t:Containerize myapp.sln && docker-compose up, where there is a compose.yaml that specifies the relationships between the services in the usual way, just using image: instead of build: stanzas for the project.

A worked example of this can be seen with this diff of the eshoponcontainers project. The docker compose YAML specifically is a useful example.

[mu88]

I would love seeing this feature, as it's literally the last missing piece from throwing away my Dockerfiles and replacing them with the SDK Container Building Tools.

[baronfel]

Making multi-architecture images is pretty straightforward, as shown above. The next step is creating image manifests using those images. There's an example of this in my sdk-container-demo repository here that builds upon the snippet above by:

• building architecture-specific images with a certain labeling scheme
• pushing those images to the destination registry
• creating a container manifest using the docker CLI with each of those images
• pushing the generated manifest to the destination registry

Our tooling doesn't yet speak these manifests, but it could learn to.

[mu88]

That looks very promising @baronfel !

This of course sparks hope 😉 what's missing from adding it to the Container Building Tools?

[mu88]

So I gave @baronfel's prototype a try yesterday and it works nicely. The icing on the cake (despite being integrated into the SDK Container Building Tools) would be if pushing the arch-specific images to the container registry wouldn't be necessary - I prefer my build process not to rely on external things like a foreign container registry. Instead, it would be cool to build the multi-arch image completely within one's local environment.