Publish symbols using release pipelines #2407

JohnTortugo · 2019-04-02T21:46:20Z

Closes: #1786

Bulk of the changes here is copying the PublishSymbols target from Publish.proj into a separate file that will be used by a release pipeline to publish symbols to MSDL & SymWeb.

I did other changes around formatting / commenting.

tmat · 2019-04-02T21:49:30Z

eng/common/PublishToPackageFeed.proj

-  manifest and publish the assets described in the manifest to
-  informed target feeds.
-->
+<!-- Licensed to the .NET Foundation under one or more agreements. The .NET Foundation licenses this file to you under the MIT license. See the LICENSE file in the project root for more information. -->
 <Project Sdk="Microsoft.NET.Sdk">


Sdk="Microsoft.NET.Sdk" [](start = 9, length = 23)

Why are these SDK projects?

Talked with @tmat. He said to leave it for now.

I haven't read most of the comments in this PR, and this has probably been discussed elsewhere; but this still feels like abusing the system to be distributing these projects via eng/common files.

@tmat has made that observation before. A more immediate solution would be to move these files to somewhere under the SDK. I've an issue to track that: #2371 I discussed a long term solution yesterday with @tmat and he shared some nice ideas. I need to put them on an issue, though.

That being said I'd love to hear more about the direction this should take. I'll ping you later or if you prefer plz post them here.

eng/common/PublishToSymbolServers.proj

src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj

mmitche

I think most repositories are publishing symbol.nupkg files are then those are getting unpacked and pushed to symweb/msdl. Are those covered here?

@dagood Is this your understanding?

jcagme · 2019-04-03T16:03:14Z

I'd look at how corefx is doing it. I know they end up publishing to those servers so we'd just need to see what they do/how

dagood · 2019-04-03T16:06:36Z

Bulk of the changes here is copying the PublishSymbols target from Publish.proj into a separate file

Why? Why was it already copied into Publish.proj as well? How is this maintainable?

I think most repositories are publishing symbol.nupkg files are then those are getting unpacked and pushed to symweb/msdl. Are those covered here?

@dagood Is this your understanding?

The PublishSymbols task does the digging around in the symbol package (zip) file.

The symbol packages do also need to be published to a build storage location so they can be reuploaded later at release time to extend the expiration time. (Not sure if that's what you were saying.)

eng/common/PublishToSymbolServers.proj

JohnTortugo · 2019-04-03T16:39:16Z

Why? Why was it already copied into Publish.proj as well? How is this maintainable?

@dagood the duplication should be only temporary. We are in the process of creating a drop-in replacement for the way Arcade publish packages. At some point of the work we'll remove the PublishSymbols target from Publish.proj. See plan here: https://github.com/dotnet/arcade/blob/master/Documentation/CorePackages/AsyncPublishing.md

Isn't this just on Publish.proj current? Is that (PublishSymbols in Publish.proj) already a copy from somewhere else?

dagood · 2019-04-03T16:53:18Z

At some point of the work we'll remove the PublishSymbols target from Publish.proj.

Will it be removed from PublishToSymbolServers.proj too to get rid of duplication from the symbol upload package's PublishSymbols.targets?

Edit: Looks like I missed an edit there (I try to F5 before each comment but I guess I missed that), question about Publish.proj wasn't there yet, but this is an answer. 😄

JohnTortugo · 2019-04-03T17:11:01Z

Will it be removed from PublishToSymbolServers.proj too to get rid of duplication from the symbol upload package's PublishSymbols.targets?

The PublishSymbols target in current PR and the linked PublishSymbols target don't do the same thing. I can replace the current call to PublishSymbols task to calling PublishSymbols target on the linked file if that is an issue.

dagood · 2019-04-03T17:20:14Z

Yeah, that's the duplication I meant, up to you if you think this isn't a maintenance concern. I don't have particular reasons to think the task's API will change, I may have overreacted when I saw duplication on top of duplication.

JohnTortugo · 2019-04-03T17:51:41Z

The symbol packages do also need to be published to a build storage location so they can be reuploaded later at release time to extend the expiration time. (Not sure if that's what you were saying.)

Silly question: you mean the .symbols.nupkg? They get published together with the .nupkgs.

dagood · 2019-04-03T17:55:45Z

Yep, thanks for confirming. (With dotnet/arcade-services#2466 from the CR thread, though, we might not always have to do that.)

JohnTortugo · 2019-04-03T20:16:16Z

I think I've addressed all your comments. Is there any other thing that I should address here?

I built this PR in these builds:

I tested it's SDK in these builds:

JohnTortugo · 2019-04-03T20:17:21Z

Tested the symbol publishing in this pipeline / releases:

JohnTortugo · 2019-04-04T17:27:59Z

Anything else?

dagood

For me, seems reasonable, I don't have a lot of context in how this is run though.

Updates (dotnet#2361) Update dependencies from https://github.com/dotnet/arcade build 20190327.11 (dotnet#2363) - Microsoft.DotNet.Arcade.Sdk - 1.0.0-beta.19177.11 - Microsoft.DotNet.Build.Tasks.Feed - 2.2.0-beta.19177.11 - Microsoft.DotNet.Helix.Sdk - 2.0.0-beta.19177.11 - Microsoft.DotNet.SignTool - 1.0.0-beta.19177.11 - Microsoft.DotNet.SwaggerGenerator.MSBuild - 1.0.0-beta.19177.11 Add missing corefx test files for mono Update JenkinsShutdownStatus.md Update JenkinsShutdownStatus.md Update JenkinsShutdownStatus.md Move images folder to correct subfolder (dotnet#2372) Added partial facade generator (dotnet#2221) * added partial facade genertor * c# coding done * making corefx projects work with new genfacades * removing nesting types from reference assembly, adding generic support nad increasing restriction of public types * Minor naming changes and other minor feedback * making nested, full facades and some dwarning disables * using ispartialfacade to use new tool * making compile errors work * Added missingtypes List * eric feedback * name changes, catch exception removed and spport for generic type parameters greater than 9 * GenFacadesOmitType added and IgnoreMissitypeList prefixed with genfacades * avoiding doube casting, omittype name change WIP Create a GH issue on publishing error(s) Try to tag non-bot accounts only Make a NuGet package instead Remove Octo using Correct passed values to .proj Correct passed values to .proj Revert changes in .proj Just create the package PR feedback Add envs for artifacts, toolset, tools, and log directories fixes dotnet#2276 use VSO vars instead remove spaces between ] and $ Update NGEN optimization doc (dotnet#2355) Add support for signing validation in the publishing release pipelines. (dotnet#2362) * Add support for signing validation in the publishing release pipelines. * Some adjustments * More comments. * PR feedback: add --traverse-subfolders Fix typos in ApiCompat project (dotnet#2293) * Fix typos in output text * Fix typos in comments Update dependencies from https://github.com/dotnet/arcade build 20190329.1 (dotnet#2380) - Microsoft.DotNet.Arcade.Sdk - 1.0.0-beta.19179.1 - Microsoft.DotNet.Build.Tasks.Feed - 2.2.0-beta.19179.1 - Microsoft.DotNet.Helix.Sdk - 2.0.0-beta.19179.1 - Microsoft.DotNet.SignTool - 1.0.0-beta.19179.1 - Microsoft.DotNet.SwaggerGenerator.MSBuild - 1.0.0-beta.19179.1 Update JenkinsShutdownStatus.md Fix race with AssemblyLoadContext.ResolveUsingResolvingEvent (dotnet#2360) Remove xunit.console.deps.json from output dir (dotnet#2381) 2286: Return passing result test for exit code 0 when no result files are found. Fix NGEN method logging The NGEN method logging code didn't account for the fact that we run IBCMerge for the same assembly multiple times when the project uses multi-targeting. This created race conditions on both writing the temp XML file and the NGEN method list. This PR fixes this by properly writing out separate files for each target framework of an assembly. closes dotnet#2309 Remove MVID After verifying the race locally removing the MVID. It's clear the race is about TF which is now properly covered. Keeping the MVID out of the name means that we can more easily diff NGEN between builds. Respond to PR feedback Use verbose for xunit execution By default the xunit output only includes tests which were skipped or failed. It provides no information about how tests are executed. This is fine for day to day execution but insufficient for CI jobs. When tests crash rudely then there is no context for why a test failed. For instance if there is a `StackOverflowException` or sigsegv then the log doesn't have any data indicating what test failed. The `-verbose` flag causes xunit to output detailed information about test execution including Starting and Finished notations. With this output its clear what test caused the crash as there will be a Starting notation as the last entry in the log. This can be invaluable when tracking down crashes / hangs. Particularly on Unix where we don't have dump support yet. Example output for a successfully executing test. ``` Microsoft.CodeAnalysis.CSharp.UnitTests.PreprocessorTests.TestIfFalseElifFalseElseEndif [STARTING] Microsoft.CodeAnalysis.CSharp.UnitTests.PreprocessorTests.TestIfFalseElifFalseElseEndif [FINISHED] Time: 0.0003478s ``` Update dependencies from https://github.com/dotnet/arcade build 20190329.2 (dotnet#2387) - Microsoft.DotNet.Arcade.Sdk - 1.0.0-beta.19179.2 - Microsoft.DotNet.Build.Tasks.Feed - 2.2.0-beta.19179.2 - Microsoft.DotNet.Helix.Sdk - 2.0.0-beta.19179.2 - Microsoft.DotNet.SignTool - 1.0.0-beta.19179.2 - Microsoft.DotNet.SwaggerGenerator.MSBuild - 1.0.0-beta.19179.2 Update JenkinsShutdownStatus.md Update JenkinsShutdownStatus.md Add angular language to swagger generator (dotnet#2378) Fix docs URL reference Fix docs URL reference add append line in sourcegenerator to comment that the following is autogenerated Add xunit.console.deps.json only for netcoreapp (dotnet#2388) Make project packable Add Microsoft.DotNet.RemoteExecutor (dotnet#2176) * Add Microsoft.DotNet.RemoteExecutor Update dependencies from https://github.com/dotnet/arcade build 20190401.12 (dotnet#2400) - Microsoft.DotNet.Arcade.Sdk - 1.0.0-beta.19201.12 - Microsoft.DotNet.Build.Tasks.Feed - 2.2.0-beta.19201.12 - Microsoft.DotNet.Helix.Sdk - 2.0.0-beta.19201.12 - Microsoft.DotNet.SignTool - 1.0.0-beta.19201.12 - Microsoft.DotNet.SwaggerGenerator.MSBuild - 1.0.0-beta.19201.12 Remove deps from test archive (dotnet#2399) Update JenkinsShutdownStatus.md Fix a typo in StartHere.md (dotnet#2325) Fix exit codes (dotnet#2394) It used to be that a return code of 2 was success (indicating the client was disconnected). Now, 0 appears to be the correct return code. Addressing issues with the new GenfacadesSourceTask (dotnet#2404) * moving using task to main file for shims, adding license header, exposing assembly path * handing platformNotSupportedGeneratedCode * Running GenerateNotSupportedSource as first target of coreCompileDependsOn, hence removing the dependency of genfacades Use reference Id for enum names if it exists. (dotnet#2405) Start consuming the latest portable IBCMerge (dotnet#2365) Update dependencies from https://github.com/dotnet/arcade build 20190402.13 (dotnet#2410) - Microsoft.DotNet.Arcade.Sdk - 1.0.0-beta.19202.13 - Microsoft.DotNet.Build.Tasks.Feed - 2.2.0-beta.19202.13 - Microsoft.DotNet.Helix.Sdk - 2.0.0-beta.19202.13 - Microsoft.DotNet.SignTool - 1.0.0-beta.19202.13 - Microsoft.DotNet.SwaggerGenerator.MSBuild - 1.0.0-beta.19202.13 missing property Tag (dotnet#2414) [Pool Provider] Prod, more logging, and correlation script (dotnet#2409) * Fix exit codes It used to be that a return code of 2 was success (indicating the client was disconnected). Now, 0 appears to be the correct return code. * [Pool Provider] Prod, more logging, and correlation script - Add a prod deployment (which is really ahead of int at this point). - Add a additional logging in based on the Maestro logging setup - Add a rough script that can map from a build onto the helix work items (and other info) that was submitted for that build Move TheoryData helper to XUnitExtensions (dotnet#2403) * Move TheoryData helper to XUnitExtensions * Move tests over to XUnitExtensions Add support for junit results (dotnet#2421) * Add support for junit results This changes adds support to the helix sdk azure pipelines reporter to process junit xml results from junit-results.xml files. * skip_reason can't be none * Use if instead of or because of FutureWarning * Add validation of expected failures * Use correct property name * Make expected failed tests correct. Update dependencies from https://github.com/dotnet/arcade build 20190403.10 (dotnet#2425) - Microsoft.DotNet.Arcade.Sdk - 1.0.0-beta.19203.10 - Microsoft.DotNet.Build.Tasks.Feed - 2.2.0-beta.19203.10 - Microsoft.DotNet.Helix.Sdk - 2.0.0-beta.19203.10 - Microsoft.DotNet.SignTool - 1.0.0-beta.19203.10 - Microsoft.DotNet.SwaggerGenerator.MSBuild - 1.0.0-beta.19203.10 Update AsyncPublishing_HowToUse.md (dotnet#2424) Adding other repo feeds Publish symbols using release pipelines (dotnet#2407) * Adding PublishToSymbolServers Fix itemgroup adding Fix filename Fix bad merge Fix bad merge remove verbose

JohnTortugo added 5 commits April 1, 2019 22:05

Adding PublishToSymbolServers

e4787c8

fix 1

53e842a

fix 2

13a49bf

fix 3

3f14fc8

fix 4

40ce2fe

JohnTortugo self-assigned this Apr 2, 2019

JohnTortugo requested review from mmitche, chcosta and jcagme April 2, 2019 21:46

tmat reviewed Apr 2, 2019

View reviewed changes

jcagme reviewed Apr 2, 2019

View reviewed changes

eng/common/PublishToSymbolServers.proj Outdated Show resolved Hide resolved

PR feedback

c2515fd

mmitche reviewed Apr 3, 2019

View reviewed changes

src/Microsoft.DotNet.Arcade.Sdk/tools/Publish.proj Show resolved Hide resolved

mmitche requested changes Apr 3, 2019

View reviewed changes

dagood reviewed Apr 3, 2019

View reviewed changes

eng/common/PublishToSymbolServers.proj Show resolved Hide resolved

dagood approved these changes Apr 4, 2019

View reviewed changes

mmitche approved these changes Apr 4, 2019

View reviewed changes

JohnTortugo merged commit dd7a71f into master Apr 4, 2019

JohnTortugo deleted the PublishSymbols branch April 4, 2019 18:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish symbols using release pipelines #2407

Publish symbols using release pipelines #2407

JohnTortugo commented Apr 2, 2019

tmat Apr 2, 2019

JohnTortugo Apr 3, 2019

chcosta Apr 3, 2019

JohnTortugo Apr 3, 2019

mmitche left a comment

jcagme commented Apr 3, 2019

dagood commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019 •

edited

Loading

dagood commented Apr 3, 2019 •

edited

Loading

JohnTortugo commented Apr 3, 2019

dagood commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019

dagood commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019

JohnTortugo commented Apr 4, 2019

dagood left a comment

Publish symbols using release pipelines #2407

Publish symbols using release pipelines #2407

Conversation

JohnTortugo commented Apr 2, 2019

tmat Apr 2, 2019

Choose a reason for hiding this comment

JohnTortugo Apr 3, 2019

Choose a reason for hiding this comment

chcosta Apr 3, 2019

Choose a reason for hiding this comment

JohnTortugo Apr 3, 2019

Choose a reason for hiding this comment

mmitche left a comment

Choose a reason for hiding this comment

jcagme commented Apr 3, 2019

dagood commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019 • edited Loading

dagood commented Apr 3, 2019 • edited Loading

JohnTortugo commented Apr 3, 2019

dagood commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019

dagood commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019

JohnTortugo commented Apr 3, 2019

JohnTortugo commented Apr 4, 2019

dagood left a comment

Choose a reason for hiding this comment

JohnTortugo commented Apr 3, 2019 •

edited

Loading

dagood commented Apr 3, 2019 •

edited

Loading