Create build manifest in PushToBlobFeed for orchestrated build final publish #1836

dagood · 2017-12-20T00:39:24Z

This adds functionality to PushToBlobFeed to create build output manifests. See core-eng/publish.md for details on usage.

https://github.com/dotnet/core-eng/issues/2287 "Add output manifest generation to PushToBlobFeed task"

This is intended for single/low-use feeds: the manifest(s) would build up over time on a central blob feed. Central feeds can ensure manifests are never pushed to them by putting a blob called disable-manifest-push at the root of the feed. (The blob's contents aren't examined and don't matter.)

I initially wanted to use Sleet feed settings for this. However, PushToBlobFeed by default runs Init on every call, which clears the settings. I think it's infeasible and error-prone to distribute an update that disables the Init calls, so I decided to go with disabling via a blob.

I'll add this blob to the dotnet-core central feed before merging this change.
(Done: https://dotnetfeed.blob.core.windows.net/dotnet-core/disable-manifest-push)

FYI @jcagme @weshaggard

I also added a small VersionTools test project. It doesn't match the existing BuildTools infra because it uses the new Core SDK project format (it won't run in CI). I wasn't able to quickly figure out how to get the existing BuildTools test project patterns to work any better than sometimes working with F5 in Visual Studio.

I also created a LaunchDebugger task that IMO is overdue. It makes it really easy to debug BuildTools tasks without temporarily hard-coding a Debugger.Launch().

mmitche · 2017-12-20T16:58:32Z

@dagood Do you have a generated sample manifest around?

dagood · 2017-12-20T17:10:19Z

Here's one that I created during test runthroughs:

<Build Name="anonymous">
  <Package Id="Feed.Initialization.Placeholder" Version="1.0.0-do-not-use" />
  <Dir Name="proj">
    <Dir Name="relative">
      <Blob foo="bar" more="testing" Id="foo.nupkg" />
    </Dir>
  </Dir>
</Build>

More "real" ones are in the tests: https://github.com/dotnet/buildtools/pull/1836/files#diff-4590351532b923e82d4a615c7d317a82R115 (besides formatting)

(Edit) New one with review changes based on the same artifact push:

<Build Name="anonymous" BuildId="no build id provided">
  <Package Id="Feed.Initialization.Placeholder" Version="1.0.0-do-not-use" />
  <Blob Id="proj/relative/foo.nupkg" foo="bar" more="testing" />
</Build>

mmitche

Generally looks good.

src/Microsoft.DotNet.Build.Tasks.Feed/PushToBlobFeed.cs

+                return;
+            }
+
+            string blobPath = $"assets/orchestration-metadata/manifests/{ManifestName}.xml";


src/Microsoft.DotNet.Build.Tasks.Feed/PushToBlobFeed.cs

 using MSBuild = Microsoft.Build.Utilities;

 namespace Microsoft.DotNet.Build.Tasks.Feed
 {
    public class PushToBlobFeed : MSBuild.Task
    {
+        private static readonly char[] ManifestDataPairSeparators = { ';' };
+        private const string DisableManifestPushConfigurationBlob = "disable-manifest-push";
+        private const string AssetsVirtualDir = "assets/";


src/Microsoft.DotNet.Build.Tasks.Feed/PushToBlobFeed.cs

+                return null;
+            }
+
+            string[] virtualPathSegments = virtualPath.Split('/');


src/Microsoft.DotNet.VersionTools/BuildManifest/Model/BlobArtifactModel.cs

+        public string VirtualDirectory
+        {
+            get { return string.Join("/", VirtualDirectorySegments); }
+            set { VirtualDirectorySegments = value.Split('/'); }


src/Microsoft.DotNet.VersionTools/BuildManifest/Model/BlobArtifactModel.cs

+            // an element like <Dir> or <Dir Name="">. This can happen if someone passes an extra
+            // slash like "foo/bar//baz". The "//" is a valid virtual dir, even though it breaks the
+            // virtual dir metaphor.
+            return dirs.Select(dir => dir.Attribute("Name")?.Value).ToArray();


src/Microsoft.DotNet.VersionTools/BuildManifest/Model/BuildIdentity.cs

+            string branch = null,
+            string commit = null)
+        {
+            Name = name;


src/Microsoft.DotNet.VersionTools/BuildManifest/Model/OrchestratedBuildModel.cs

+            }
+            if (string.IsNullOrEmpty(identity.BuildId))
+            {
+                throw new ArgumentException(


src/Microsoft.DotNet.VersionTools/BuildManifest/Model/BuildIdentity.cs

+            string branch = null,
+            string commit = null)
+        {
+            if (name == null)


src/Microsoft.DotNet.Build.Tasks.Feed/PushToBlobFeed.cs

@@ -34,6 +41,22 @@ public class PushToBlobFeed : MSBuild.Task

        public int UploadTimeoutInMinutes { get; set; } = 5;

+        public bool SkipCreateManifest { get; set; }
+
+        public string ManifestName { get; set; } = "anonymous";


dagood self-assigned this Dec 20, 2017

dagood requested a review from mmitche December 20, 2017 16:49

mmitche reviewed Dec 20, 2017

View reviewed changes

mmitche reviewed Dec 21, 2017

View reviewed changes

src/Microsoft.DotNet.VersionTools/BuildManifest/Model/BuildIdentity.cs Outdated

string branch = null,

string commit = null)

{

if (name == null)

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

mmitche approved these changes Dec 21, 2017

View reviewed changes

dagood added 2 commits December 21, 2017 14:16

Create build manifest in PushToBlobFeed

79e2b44

Add a LaunchDebugger task

fef5345

dagood force-pushed the generate-manifest branch from bd84b29 to fef5345 Compare December 21, 2017 20:17

dagood merged commit 4ef4b9a into dotnet:master Dec 21, 2017

dagood deleted the generate-manifest branch December 21, 2017 20:24

dagood mentioned this pull request Dec 22, 2017

Add build output manifest dotnet/versions client #1838

Merged

weshaggard reviewed Jan 2, 2018

View reviewed changes

maririos mentioned this pull request Feb 22, 2018

[Feed package] Decouple VersionTools manifest generation from Feed package dotnet/arcade#19

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create build manifest in PushToBlobFeed for orchestrated build final publish #1836

Create build manifest in PushToBlobFeed for orchestrated build final publish #1836

dagood commented Dec 20, 2017 •

edited

mmitche commented Dec 20, 2017

dagood commented Dec 20, 2017 •

edited

mmitche left a comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

Create build manifest in PushToBlobFeed for orchestrated build final publish #1836

Create build manifest in PushToBlobFeed for orchestrated build final publish #1836

Conversation

dagood commented Dec 20, 2017 • edited

mmitche commented Dec 20, 2017

dagood commented Dec 20, 2017 • edited

mmitche left a comment

Choose a reason for hiding this comment

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dagood commented Dec 20, 2017 •

edited

dagood commented Dec 20, 2017 •

edited