Design and implement a way for staging to know where an asset came from in the VMR build

[mmitche]

There is a non-trivial amount of automation in staging pipeline today the knows about where a particular component was build. For instance, sign checking will need to know where the asset it's looking at was built so as to find the sign check exclusion files. Similar goes for SDL validation, aka.ms link creation (to some degree), etc.

We should determine whether we can avoid this requirement altogether, or augment the VMR automation to record what component built what.

T-Shirt Size: Medium

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[mmitche]

@MilenaHristova

[tkapin]

This is a prerequisite to other issues in the Release Infra area and should be figured out sooner than later. @mmitche - it's part of the VB area. Who do you see working on this, ideally?

[mmitche]

Probably @NikolaMilosavljevic? i think this is in the general publishing bucket:

#4101

[andriipatsula]

Plus, this issue is a prerequisite to https://github.com/dotnet/release/issues/783
It'd be nice to know the interface (how and where the data will be stored; json file in the repo or something else; the structure)

[tkapin]

@MichaelSimons - since this is a prerequisite for https://github.com/dotnet/release/issues/776, we need to get clarity of when this work will start and how can @andriipatsula and @MilenaHristova contribute to it so it doesn't become a blocker for the release infra UB tasks.

[andriipatsula]

It'd be nice to know the interface (how and where the data will be stored; json file in the repo or something else; the structure)

@mmitche proposed to add an attribute to the MergedManifest.xml file named Owner or Creator:
<Package Id="Microsoft.DotNet.ApiCompat.Tool" Version="9.0.100-preview.2.24112.2" NonShipping="false" Creator="sdk" />

I am making changes in the darc gather-drop command to extend the manifest.json file generation and I am relying on this additional attribute. It'd be great to know the final name (owner, creator or some other) of the attribute or if we change the direction - the interface.

[mmitche]

Creator is fine by me. I would default for now to the repo name in case of a missing attribute.

[tkapin]

For the naming of the new attribute, wouldn't Origin express the meaning more precisely? The repo itself doesn't create anything, it just contains the originating source code of the asset.

[mmitche]

For the naming of the new attribute, wouldn't Origin express the meaning more precisely? The repo itself doesn't create anything, it just contains the originating source code of the asset.

Yeah, Origin is better

[tkapin]

Next step: figure out who is going to implement the part that would add the Origin attribute to the MergedManifest.xml.

[NikolaMilosavljevic]

Would MergedManifest.xml be something that is produced in VMR after all repos have produced their individual asset manifests?

Since we don't do publishing yet and we don't have the manifests, this work would need to wait on completion of #4101, or even better, be included in that work. I am making good progress on that issue.

[NikolaMilosavljevic]

It seems that new attribute would be added to Package element, and would be something that is done during publishing in each repo. This shouldn't be blocked and can be done independently of my work, but I can certainly do it. Let me know the plan.

[MichaelSimons]

@andriipatsula has a workaround so this is not blocking at the moment. Whoever is free first or is blocked by this can pick up the work.

[andriipatsula]

Would MergedManifest.xml be something that is produced in VMR after all repos have produced their individual asset manifests?

Yes, we'll have a single MergedManifest.xml file produced for the whole VMR build. @dkurepa is working on this.

I think, we can inject new attribute in asset manifest files (Package & Blob) and forward to the MargedManifest.xml.

• arcade/PushToAzureDevOpsArtifacts - creates asset manifest.
• arcade-services/PushMetadataToBuildAssetRegistry - merges asset manifests into singe MergedManifest.xml

But the problem is determining the origin (repo name) during asset manifest creation. As Nikola mentioned, "we don't do publishing yet and we don't have the manifests," so I don't know what the Build.Name would be: original repo name, dotnet-dotnet, or nothing at all.

[andriipatsula]

Yesterday we had a conversation with Nikola and Djuradj. We are waiting for the results of the https://dev.azure.com/dnceng/internal/_build/results?buildId=2393215&view=results build to check how the published artifact manifest looks for the VMR build.

[andriipatsula]

Asset manifest for the VMR build has Build.Name == dotnet-dotnet (VMR repo), and this seems reasonable to me. I took @NikolaMilosavljevic 's code changes as a baseline, on top of which I implemented a PoC commit and tested it. I am going to introduce a new parameter, ManifestOriginRepoName, in the PushToAzureDevOpsArtifacts MSBuild task (renamed to PushToBuildStorage) and pass the repository name as a Build command parameter in /repo-projects/Directory.Build.props.

I'll wait until Nikola merges his changes, and I think I need about 1 day to make the PoC production-ready and test it properly. Feel free to check the commit.

The resulted asset manifest for the emsdk repository looks like:

<Build PublishingVersion="3" Name="dotnet-dotnet" ...>
<Package Id="Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100-preview.2" Version="9.0.0-preview.2.24121.1" Origin="emsdk" />
<Package Id="Microsoft.NET.Workload.Emscripten.Current.Manifest-9.0.100.Transport" Version="9.0.0-preview.2.24121.1" NonShipping="true" Origin="emsdk" />
<Package Id="Microsoft.NET.Workload.Emscripten.net6.Manifest-9.0.100-preview.2" Version="9.0.0-preview.2.24121.1" Origin="emsdk" />
<Package Id="Microsoft.NET.Workload.Emscripten.net7.Manifest-9.0.100-preview.2" Version="9.0.0-preview.2.24121.1" Origin="emsdk" />
<Package Id="Microsoft.NET.Workload.Emscripten.net8.Manifest-9.0.100-preview.2" Version="9.0.0-preview.2.24121.1" Origin="emsdk" />
...
</Build>

[andriipatsula]

This work has been put on hold for a week or two. The plan is the same as described in the previous comment.

[andriipatsula]

Arcade part: dotnet/arcade#14718
runtime: dotnet/runtime#101241

Test build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2433413&view=results

Currently we publish only Verticalmanifest.xml files. We have a tracking issue to allow publishing MergedManifest.xml #4199. I think we should also publish AssetManifests.xml files.

The PRs I created add the OriginRepo attribute to the AssetManifest.xml file, and it works as expected for all repos except the arcade and runtime repositories. For some reason, they are special.

I am passing RepositoryName as a build argument in SourceBuild: https://github.com/andriipatsula/installer/blob/dev/apatsula/source_build_3898/src/SourceBuild/content/repo-projects/Directory.Build.props#L86 . We may redefine build arguments somewhere else.

[andriipatsula]

PRs are merged. Example of a VerticalBuild.xml file here.