Bump up the Microsoft.CodeAnalysis version used by packages in the repo #6701

mavasani · 2023-06-21T12:14:54Z

Bump up the Microsoft.CodeAnalysis version used by packages in the repo to 3.11.0 (VS2019 16.11), except PublicApiAnalyzers, and also bump up the version of the generated analyzer packages in the repo.

This brings in the nullable annotated API surfaces for ISymbol, IOperation and syntax APIs, which generates a large number of new nullable warnings. This PR also fixes/suppresses these warnings as appropriate.

Closes #6089

…po (except PublicApiAnalyzers) to 3.11.0 (VS2019 16.11) and also bump up the version of the generated analyzer packages in the repo. This brings in the nullable annotated API surfaces for ISymbol, IOperation and syntax APIs, which generates a large number of new nullable warnings. This PR also fixes/suppresses these warnings, as appropriate.

mavasani · 2023-06-21T12:15:27Z

@Youssef1313

eng/Versions.props

src/Microsoft.CodeAnalysis.Analyzers/Core/FixAnalyzers/FixerWithFixAllAnalyzer.cs

...crosoft.CodeQuality.Analyzers/ApiDesignGuidelines/CancellationTokenParametersMustComeLast.cs

src/NetAnalyzers/Core/Microsoft.NetCore.Analyzers/Security/DoNotDisableRequestValidation.cs

Youssef1313

Haven't looked into every line (it's a quite large PR), but going through quickly, LGTM!

mavasani · 2023-06-22T06:44:09Z

eng/Versions.props

    <!-- Microsoft.CodeAnalysis versions for different purposes. -->
    <!-- Surface area that various projects compile against. These should have source-build reference packages -->
-    <MicrosoftCodeAnalysisForRoslynDiagnosticsAnalyzersVersion>3.7.0</MicrosoftCodeAnalysisForRoslynDiagnosticsAnalyzersVersion>
+    <MicrosoftCodeAnalysisForRoslynDiagnosticsAnalyzersVersion>3.11.0</MicrosoftCodeAnalysisForRoslynDiagnosticsAnalyzersVersion>
    <MicrosoftCodeAnalysisVersionForPublicApiAnalyzers>1.2.1</MicrosoftCodeAnalysisVersionForPublicApiAnalyzers>
-    <MicrosoftCodeAnalysisVersionForBannedApiAnalyzers>2.9.0</MicrosoftCodeAnalysisVersionForBannedApiAnalyzers>
-    <MicrosoftCodeAnalysisVersionForBannedApiAnalyzersTests>3.3.1</MicrosoftCodeAnalysisVersionForBannedApiAnalyzersTests>
-    <MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzers>2.9.0</MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzers>
-    <MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzersTests>3.3.1</MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzersTests>
+    <MicrosoftCodeAnalysisVersionForBannedApiAnalyzers>3.11.0</MicrosoftCodeAnalysisVersionForBannedApiAnalyzers>
+    <MicrosoftCodeAnalysisVersionForBannedApiAnalyzersTests>3.11.0</MicrosoftCodeAnalysisVersionForBannedApiAnalyzersTests>
+    <MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzers>3.11.0</MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzers>
+    <MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzersTests>3.11.0</MicrosoftCodeAnalysisVersionForPerfSensitiveAnalyzersTests>
    <MicrosoftCodeAnalysisVersionForResxSourceGenerators>4.0.1</MicrosoftCodeAnalysisVersionForResxSourceGenerators>
-    <MicrosoftCodeAnalysisVersionForNetAnalyzers>3.3.1</MicrosoftCodeAnalysisVersionForNetAnalyzers>
-    <MicrosoftCodeAnalysisVersionForTextAnalyzers>3.3.1</MicrosoftCodeAnalysisVersionForTextAnalyzers>
-    <MicrosoftCodeAnalysisVersionForCodeAnalysisAnalyzers>3.3.1</MicrosoftCodeAnalysisVersionForCodeAnalysisAnalyzers>
-    <MicrosoftCodeAnalysisVersionForToolsAndUtilities>3.3.1</MicrosoftCodeAnalysisVersionForToolsAndUtilities>
+    <MicrosoftCodeAnalysisVersionForNetAnalyzers>3.11.0</MicrosoftCodeAnalysisVersionForNetAnalyzers>
+    <MicrosoftCodeAnalysisVersionForTextAnalyzers>3.11.0</MicrosoftCodeAnalysisVersionForTextAnalyzers>
+    <MicrosoftCodeAnalysisVersionForCodeAnalysisAnalyzers>3.11.0</MicrosoftCodeAnalysisVersionForCodeAnalysisAnalyzers>
+    <MicrosoftCodeAnalysisVersionForToolsAndUtilities>3.11.0</MicrosoftCodeAnalysisVersionForToolsAndUtilities>


@mmitche @MichaelSimons This PR moves the referenced Microsoft.CodeAnalysis versions for the packages to 3.11.0 (shipped with VS2019 16.11) and fixes a bunch of nullable compiler warnings that were coming from the new packages.

However, this fails the pre-build detection step for source build that you added with #6610. I see that dotnet/source-build-reference-packages#692 added the SB reference packages for the prior versions of the dependency packages, and are being mapped from here:

roslyn-analyzers/eng/Version.Details.xml

Lines 8 to 12 in b4dc141

<Dependency Name="Microsoft.SourceBuild.Intermediate.source-build-reference-packages" Version="8.0.0-alpha.1.23305.4">

<Uri>https://github.com/dotnet/source-build-reference-packages</Uri>

<Sha>540ccf1a6831993d8ac8bf10bdfd3fde93ecee2e</Sha>

<SourceBuild RepoName="source-build-reference-packages" ManagedOnly="true" />

</Dependency>

I presume this PR would require an update to the SB packages to add the 3.11.0 version of the packages before we can pass the pre-build detection leg. Is that correct? If so, who can help me out to perform this update in https://github.com/dotnet/source-build-reference-packages? If there are documentation pointers on how to go about this, please share them so I can try to do this myself. Thanks!

Ah, I see the steps at https://github.com/dotnet/source-build-reference-packages/tree/main#adding-new-packages. I'll give it a shot.

I tried executing those steps locally, but I get a build failure at the first step itself:

generate.cmd -package microsoft.codeanalysis,3.11.0

Error:

PackageSourceGenerator -> Generating package source for Humanizer.Core (v2.2.0)... Determining projects to restore... Restored C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj (in 5 ms). Restored C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGeneratorTask\PackageSourceGenera torTask.csproj (in 136 ms). C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj(106,5): error MSB4018: The "G etPackageItems" task failed unexpectedly. C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj(106,5): error MSB4018: System .ArgumentException: Found strong name key that doesn't map: Key=979442b78dfc278e, Assembly=Humanizer C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj(106,5): error MSB4018: at Microsoft.DotNet.SourceBuild.Tasks.GetPackageItems.TryGetStrongNameData(AssemblyName assemblyName, StrongNameData& stro ngNameData) in C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGeneratorTask\GetPackageItems .cs:line 241 C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj(106,5): error MSB4018: at Microsoft.DotNet.SourceBuild.Tasks.GetPackageItems.SetCompileItems(PackageArchiveReader packageArchiveReader, TargetFra meworkRegexFilter targetFrameworkRegexFilter) in C:\source-build-reference-packages\src\packageSourceGenerator\PackageS ourceGeneratorTask\GetPackageItems.cs:line 153 C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj(106,5): error MSB4018: at Microsoft.DotNet.SourceBuild.Tasks.GetPackageItems.Execute() in C:\source-build-reference-packages\src\packageSourceGen erator\PackageSourceGeneratorTask\GetPackageItems.cs:line 84 C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj(106,5): error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() C:\source-build-reference-packages\src\packageSourceGenerator\PackageSourceGenerator.proj(106,5): error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext ta skLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)

Warnings:

C:\Users\mavasani\.nuget\packages\microsoft.dotnet.genapi.task\8.0.100-preview.6.23319.8\build\Microsoft.DotNet.GenAPI. Task.targets(45,5): warning CP1002: Could not resolve reference 'SQLitePCLRaw.core.dll' in any of the provided search d irectories. [C:\source-build-reference-packages\artifacts\bin\PackageSourceGenerator\microsoft.codeanalysis.csharp.work spaces\3.11.0\microsoft.codeanalysis.csharp.workspaces.3.11.0.csproj::TargetFramework=netstandard2.0] C:\Users\mavasani\.nuget\packages\microsoft.dotnet.genapi.task\8.0.100-preview.6.23319.8\build\Microsoft.DotNet.GenAPI. Task.targets(45,5): warning CP1002: Could not resolve reference 'SQLitePCLRaw.batteries_v2.dll' in any of the provided search directories. [C:\source-build-reference-packages\artifacts\bin\PackageSourceGenerator\microsoft.codeanalysis.csh arp.workspaces\3.11.0\microsoft.codeanalysis.csharp.workspaces.3.11.0.csproj::TargetFramework=netstandard2.0] C:\Users\mavasani\.nuget\packages\microsoft.dotnet.genapi.task\8.0.100-preview.6.23319.8\build\Microsoft.DotNet.GenAPI. Task.targets(45,5): warning CP1002: Could not resolve reference 'Humanizer.dll' in any of the provided search directori es. [C:\source-build-reference-packages\artifacts\bin\PackageSourceGenerator\microsoft.codeanalysis.csharp.workspaces\3 .11.0\microsoft.codeanalysis.csharp.workspaces.3.11.0.csproj::TargetFramework=netstandard2.0] C:\Users\mavasani\.nuget\packages\microsoft.dotnet.genapi.task\8.0.100-preview.6.23319.8\build\Microsoft.DotNet.GenAPI. Task.targets(45,5): warning CP1002: Could not resolve reference 'Humanizer.dll' in any of the provided search directori es. [C:\source-build-reference-packages\artifacts\bin\PackageSourceGenerator\microsoft.codeanalysis.csharp.workspaces\3 .11.0\microsoft.codeanalysis.csharp.workspaces.3.11.0.csproj::TargetFramework=netstandard2.0] C:\Users\mavasani\.nuget\packages\microsoft.dotnet.genapi.task\8.0.100-preview.6.23319.8\build\Microsoft.DotNet.GenAPI. Task.targets(45,5): warning CP1002: Could not resolve reference 'SQLitePCLRaw.core.dll' in any of the provided search d irectories. [C:\source-build-reference-packages\artifacts\bin\PackageSourceGenerator\microsoft.codeanalysis.workspaces. common\3.11.0\microsoft.codeanalysis.workspaces.common.3.11.0.csproj::TargetFramework=netstandard2.0] C:\Users\mavasani\.nuget\packages\microsoft.dotnet.genapi.task\8.0.100-preview.6.23319.8\build\Microsoft.DotNet.GenAPI. Task.targets(45,5): warning CP1002: Could not resolve reference 'SQLitePCLRaw.batteries_v2.dll' in any of the provided search directories. [C:\source-build-reference-packages\artifacts\bin\PackageSourceGenerator\microsoft.codeanalysis.wor kspaces.common\3.11.0\microsoft.codeanalysis.workspaces.common.3.11.0.csproj::TargetFramework=netstandard2.0] C:\Users\mavasani\.nuget\packages\microsoft.dotnet.genapi.task\8.0.100-preview.6.23319.8\build\Microsoft.DotNet.GenAPI. Task.targets(45,5): warning CP1002: Could not resolve reference 'Humanizer.dll' in any of the provided search directori es. [C:\source-build-reference-packages\artifacts\bin\PackageSourceGenerator\microsoft.codeanalysis.workspaces.common\3 .11.0\microsoft.codeanalysis.workspaces.common.3.11.0.csproj::TargetFramework=netstandard2.0]

@mavasani - you will need to add the strong name key for Humanizer here - https://github.com/dotnet/source-build-reference-packages/blob/4163a212b06740e791c3dfe7ea1258565229af07/src/packageSourceGenerator/PackageSourceGeneratorTask/GetPackageItems.cs#L23

{ "979442b78dfc278e", ("Humanizer", "Humanizer") }

….0.1, it doesn't work against 3.11.0 version of Microsoft.CodeAnalysis.Features package. Also make sure that we don't package unnecessary third-party assemblies within the Metrics package, these fail the signing step.

codecov · 2023-06-22T11:38:22Z

Codecov Report

Merging #6701 (1b07c44) into main (21a3b16) will decrease coverage by 0.04%.
The diff coverage is 78.72%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6701      +/-   ##
==========================================
- Coverage   96.36%   96.33%   -0.04%     
==========================================
  Files        1395     1393       -2     
  Lines      325074   325082       +8     
  Branches    10676    10711      +35     
==========================================
- Hits       313265   313165     -100     
+ Misses       9242     9212      -30     
- Partials     2567     2705     +138

sharwell · 2023-06-22T15:19:54Z

...Analysis/FlowAnalysis/Analysis/TaintedDataAnalysis/HardcodedSymmetricAlgorithmKeysSources.cs

                            arguments.Length == 5 &&
-                            arguments[0].Parameter.Type.SpecialType == SpecialType.System_String,
+                            arguments[0].Parameter?.Type.SpecialType == SpecialType.System_String,


The length and value check can be combined:

arguments is [{ Parameter.Type.SpecialType: SpecialType.System_String }, _, _, _, _]

Required for dotnet/roslyn-analyzers#6701, which moves the analyzer packages in that repo to Microsoft.CodeAnalysis 3.11.0.

mavasani · 2023-06-23T05:10:27Z

The PR is now green, except for the Source build failure due to missing pre-built packages detection. Opened dotnet/source-build-reference-packages#727 to help with that.

* Add Microsoft.CodeAnalysis 3.11.0 SB package dependencies Required for dotnet/roslyn-analyzers#6701, which moves the analyzer packages in that repo to Microsoft.CodeAnalysis 3.11.0. * Revert previous updates to existing projects This reverts commit 64a9f53. * Update build.props * Regen Microsoft.CodeAnalysis 3.11.0 * Fix syntax errors in generated code. * Remove Microsoft.CodeAnalysis.Analyzers reference * Update meta package reference --------- Co-authored-by: Michael Simons <msimons@microsoft.com>

…lysis 3.11.0 SB package dependencies (#727) * Add Microsoft.CodeAnalysis 3.11.0 SB package dependencies Required for dotnet/roslyn-analyzers#6701, which moves the analyzer packages in that repo to Microsoft.CodeAnalysis 3.11.0. * Revert previous updates to existing projects This reverts commit 64a9f5352401bcbc718799cb91cccc9115acd69e. * Update build.props * Regen Microsoft.CodeAnalysis 3.11.0 * Fix syntax errors in generated code. * Remove Microsoft.CodeAnalysis.Analyzers reference * Update meta package reference --------- Co-authored-by: Michael Simons <msimons@microsoft.com> Original commit: dotnet/source-build-reference-packages@e910f7b [[ commit created by automation ]]

mavasani · 2023-06-24T15:31:45Z

eng/Version.Details.xml

      <Uri>https://github.com/dotnet/roslyn</Uri>
-      <Sha>66a912c9463eebe832cf742d2fe8bb2e1a4600ec</Sha>
+      <Sha>ae1fff344d46976624e68ae17164e0607ab68b10</Sha>
    </Dependency>
    <Dependency Name="Microsoft.SourceBuild.Intermediate.source-build-reference-packages" Version="8.0.0-alpha.1.23305.4">


@MichaelSimons I presume we would need to update the package version here and also update the SHA below. Can you please point me to the package feed where this package is published OR point out the new package version to use?

@MichaelSimons @mmitche Can you please help here?

A darc subscription should be defined to automatically update this dependency. You can use the darc cli to find the latest version info and feed.

> darc get-asset --name Microsoft.SourceBuild.Intermediate.source-build-reference-packages Looking up assets with name 'Microsoft.SourceBuild.Intermediate.source-build-reference-packages' in the last 30 days Microsoft.SourceBuild.Intermediate.source-build-reference-packages @ 8.0.0-alpha.1.23328.2 Repository: https://github.com/dotnet/source-build-reference-packages Branch: main Commit: 5740876021887c3a50d3dae34cc125dbab36316f Build Number: 20230628.2 Date Produced: 2023-06-28 11:31 AM Build Link: https://dev.azure.com/dnceng/internal/_build/results?buildId=2211175 BAR Build Id: 183409 Released: False Channels: - .NET Eng - Latest Locations: - https://dev.azure.com/dnceng/internal/_apis/build/builds/2211175/artifacts (Container) - https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json (NugetFeed) ...

mavasani · 2023-06-29T15:36:38Z

@MichaelSimons Seems there are more source build related failures. I'll work on fixing these.

.packages/microsoft.dotnet.arcade.sdk/8.0.0-beta.23269.1/tools/SourceBuild/AfterSourceBuild.proj(68,5): error : (NETCORE_ENGINEERING_TELEMETRY=AfterSourceBuild) 2 new pre-builts discovered! Detailed usage report can be found at /mnt/vss/_work/1/s/artifacts/source-build/self/prebuilt-report/baseline-comparison.xml.
See aka.ms/dotnet/prebuilts for guidance on what pre-builts are and how to eliminate them.
Package IDs are:
System.IO.Pipelines.6.0.3
System.Threading.Channels.6.0.0

dotnet/roslyn-analyzers#6701 (comment) requires couple more package dependencies: - System.IO.Pipelines.6.0.3 - System.Threading.Channels.6.0.0

eng/Version.Details.xml

github-actions bot added the Community label Jun 21, 2023

mavasani removed the Community label Jun 21, 2023

Youssef1313 reviewed Jun 21, 2023

View reviewed changes

eng/Versions.props Outdated Show resolved Hide resolved

Youssef1313 reviewed Jun 21, 2023

View reviewed changes

src/Microsoft.CodeAnalysis.Analyzers/Core/FixAnalyzers/FixerWithFixAllAnalyzer.cs Show resolved Hide resolved

Youssef1313 reviewed Jun 21, 2023

View reviewed changes

...crosoft.CodeQuality.Analyzers/ApiDesignGuidelines/CancellationTokenParametersMustComeLast.cs Show resolved Hide resolved

Youssef1313 reviewed Jun 21, 2023

View reviewed changes

src/NetAnalyzers/Core/Microsoft.NetCore.Analyzers/Security/DoNotDisableRequestValidation.cs Outdated Show resolved Hide resolved

Youssef1313 approved these changes Jun 21, 2023

View reviewed changes

mavasani added 5 commits June 21, 2023 21:29

Fix more warnings

60e5167

Fix tests by passing in non-null parse options

0ba57e6

Update versions

00425e7

Address feedback

e07758c

Merge remote-tracking branch 'upstream/main' into MoveToNewMSCAReference

4ef4d64

mavasani marked this pull request as ready for review June 22, 2023 06:16

mavasani requested a review from a team as a code owner June 22, 2023 06:16

Fix more tests

0853807

mavasani commented Jun 22, 2023

View reviewed changes

sharwell reviewed Jun 22, 2023

View reviewed changes

mavasani added 2 commits June 23, 2023 08:43

Merge remote-tracking branch 'upstream/main' into MoveToNewMSCAReference

472771e

Use GetTextOrEmpty

1b0cd49

mavasani mentioned this pull request Jun 23, 2023

Add Microsoft.CodeAnalysis 3.11.0 SB package dependencies dotnet/source-build-reference-packages#727

Merged

mavasani commented Jun 24, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/main' into MoveToNewMSCAReference

ab1e600

mavasani added 2 commits June 29, 2023 20:10

Move to newer SB reference package

d88a95d

Fix more warnings

bfa7eb0

mavasani mentioned this pull request Jun 29, 2023

Add few more SB package dependencies dotnet/source-build-reference-packages#732

Merged

mavasani commented Jul 1, 2023

View reviewed changes

eng/Version.Details.xml Outdated Show resolved Hide resolved

mavasani commented Jul 1, 2023

View reviewed changes

eng/Version.Details.xml Outdated Show resolved Hide resolved

Apply suggestions from code review

1b07c44

mavasani merged commit 8baa571 into dotnet:main Jul 1, 2023
11 checks passed

mavasani deleted the MoveToNewMSCAReference branch July 1, 2023 18:58

mthalman mentioned this pull request Jul 17, 2023

Poison leak of Microsoft.CodeAnalysis.AnalyzerUtilities.dll dotnet/source-build#3556

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump up the Microsoft.CodeAnalysis version used by packages in the repo #6701

Bump up the Microsoft.CodeAnalysis version used by packages in the repo #6701

mavasani commented Jun 21, 2023 •

edited

Loading

mavasani commented Jun 21, 2023

Youssef1313 left a comment

mavasani Jun 22, 2023 •

edited

Loading

mavasani Jun 22, 2023

mavasani Jun 22, 2023

MichaelSimons Jun 22, 2023

codecov bot commented Jun 22, 2023 •

edited

Loading

sharwell Jun 22, 2023

mavasani commented Jun 23, 2023

mavasani Jun 24, 2023 •

edited

Loading

mavasani Jun 27, 2023

MichaelSimons Jun 29, 2023

mavasani commented Jun 29, 2023

	<Dependency Name="Microsoft.SourceBuild.Intermediate.source-build-reference-packages" Version="8.0.0-alpha.1.23305.4">
	<Uri>https://github.com/dotnet/source-build-reference-packages</Uri>
	<Sha>540ccf1a6831993d8ac8bf10bdfd3fde93ecee2e</Sha>
	<SourceBuild RepoName="source-build-reference-packages" ManagedOnly="true" />
	</Dependency>

Bump up the Microsoft.CodeAnalysis version used by packages in the repo #6701

Bump up the Microsoft.CodeAnalysis version used by packages in the repo #6701

Conversation

mavasani commented Jun 21, 2023 • edited Loading

mavasani commented Jun 21, 2023

Youssef1313 left a comment

Choose a reason for hiding this comment

mavasani Jun 22, 2023 • edited Loading

Choose a reason for hiding this comment

mavasani Jun 22, 2023

Choose a reason for hiding this comment

mavasani Jun 22, 2023

Choose a reason for hiding this comment

MichaelSimons Jun 22, 2023

Choose a reason for hiding this comment

codecov bot commented Jun 22, 2023 • edited Loading

Codecov Report

sharwell Jun 22, 2023

Choose a reason for hiding this comment

mavasani commented Jun 23, 2023

mavasani Jun 24, 2023 • edited Loading

Choose a reason for hiding this comment

mavasani Jun 27, 2023

Choose a reason for hiding this comment

MichaelSimons Jun 29, 2023

Choose a reason for hiding this comment

mavasani commented Jun 29, 2023

mavasani commented Jun 21, 2023 •

edited

Loading

mavasani Jun 22, 2023 •

edited

Loading

codecov bot commented Jun 22, 2023 •

edited

Loading

mavasani Jun 24, 2023 •

edited

Loading