Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Src gen: use GetBestTypeByMetadataName instead of GetTypeByMetadataName #59092

Merged
merged 11 commits into from
Sep 29, 2021

Conversation

steveharter
Copy link
Member

@steveharter steveharter commented Sep 14, 2021

Change remaining source generator locations that use Compilation.GetTypeByMetadataName to an extension that addresses conflict cases. The guidance and extension implementation is from the Roslyn team.

See #57349

This is a simple 1:1 method replacement; has been previously tested in System.Text.Json (which has a unit test) and by Roslyn.

This may be ported to 6.0.

@ghost
Copy link

ghost commented Sep 14, 2021

Tagging subscribers to this area: @eiriktsarpalis, @layomia
See info in area-owners.md if you want to be subscribed.

Issue Details

Change remaining source generator locations that use Compilation.GetTypeByMetadataName to an extension that address conflict cases. The guidance and extension implementation is from the Roslyn team.

See #57349

This is a simple 1:1 method replacement; has been previously tested in System.Text.Json (which has a unit test) and by Roslyn.

This may be ported to 6.0.

Author: steveharter
Assignees: steveharter
Labels:

area-System.Text.Json

Milestone: 6.0.0

@steveharter steveharter force-pushed the SrcGenGetBestType branch 2 times, most recently from db8fb31 to 129b716 Compare September 17, 2021 20:50
@steveharter
Copy link
Member Author

steveharter commented Sep 20, 2021

WASM issue likely caused by #51961

binlog section (crash on publish/ProxyProjectForAOTOnHelix.proj):

/datadisks/disk1/work/AF0109AA/p/build/wasm/WasmApp.Native.targets(306,5): Failed to compile /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -> /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.o
 "/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpDfyD7v.tmp
clang version 13.0.0 (/b/s/w/ir/cache/git/chromium.googlesource.com-external-github.com-llvm-llvm--project 5852582532b3eb3ea8da51a1e272d8d017bd36c9)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin
 (in-process)
 "/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++" -cc1 -triple wasm32-unknown-emscripten -emit-obj --mrelax-relocations -disable-free -main-file-name Microsoft.CodeAnalysis.CSharp.dll.bc -mrelocation-model static -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -target-cpu generic -fvisibility hidden -debug-info-kind=limited -dwarf-version=4 -debugger-tuning=gdb -v -fcoverage-compilation-dir=/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish -resource-dir /datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/lib/clang/13.0.0 -Oz -Werror=implicit-function-declaration -fdebug-compilation-dir=/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish -ferror-limit 19 -fgnuc-version=4.2.1 -fno-inline-functions -vectorize-slp -iwithsysroot/include/SDL -iwithsysroot/include/compat -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -o /tmp/tmpDfyD7v.tmp -x ir /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc
clang -cc1 version 13.0.0 based upon LLVM 13.0.0git default target x86_64-unknown-linux-gnu
emcc: error: '/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AF0109AA/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AF0109AA/w/AB960941/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpDfyD7v.tmp' failed (received SIGKILL (-9)) [took 356.790s] [/datadisks/disk1/work/AF0109AA/w/AB960941/e/publish/ProxyProjectForAOTOnHelix.proj]

console log:

/datadisks/disk1/work/AAE90973/p/build/wasm/WasmApp.Native.targets(306,5): error : Failed to compile /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -> /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.o [/datadisks/disk1/work/AAE90973/w/B4AD09E0/e/publish/ProxyProjectForAOTOnHelix.proj]
/datadisks/disk1/work/AAE90973/p/build/wasm/WasmApp.Native.targets(306,5): error :  "/datadisks/disk1/work/AAE90973/p/build/emsdk/upstream/bin/clang++" -target wasm32-unknown-emscripten -DEMSCRIPTEN -fno-inline-functions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-cxx-exceptions -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -D__EMSCRIPTEN_major__=2 -D__EMSCRIPTEN_minor__=0 -D__EMSCRIPTEN_tiny__=23 -D_LIBCPP_ABI_VERSION=2 -Dunix -D__unix -D__unix__ -Werror=implicit-function-declaration -Xclang -iwithsysroot/include/SDL --sysroot=/datadisks/disk1/work/AAE90973/p/build/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -Oz -g -v -c /datadisks/disk1/work/AAE90973/w/B4AD09E0/e/wasm_build/obj/wasm/for-build/Microsoft.CodeAnalysis.CSharp.dll.bc -o /tmp/tmpyZM8kY.tmp [/datadisks/disk1/work/AAE90973/w/B4AD09E0/e/publish/ProxyProjectForAOTOnHelix.proj]

@lewing
Copy link
Member

lewing commented Sep 20, 2021

cc @vargaz

@lewing
Copy link
Member

lewing commented Sep 20, 2021

Something is killing the processes here, we need to figure out what and why before we can work on solving it.

@steveharter
Copy link
Member Author

@lewing unless there is an imminent fix for the crash, can you supply the build condition to avoid the crash? I tried this but the crash is still occurring. Thanks

@lewing
Copy link
Member

lewing commented Sep 21, 2021

@steveharter have you checked against main after #59352 landed?

@@ -19,6 +19,12 @@
<ProjectExclusions Include="$(MSBuildThisFileDirectory)Microsoft.Extensions.Logging\tests\Microsoft.Extensions.Logging.Generators.Tests\Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj" />
</ItemGroup>

<ItemGroup Condition="'$(TargetOS)' == 'Browser' and '$(BuildAOTTestsOnHelix)' == 'true' and '$(RunDisabledWasmTests)' != 'true' and '$(RunAOTCompilation)' == 'true'">
Copy link
Member

@lewing lewing Sep 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you shouldn't need a new group for this, the exclusions 4 lines above should be enough but it looks like the paths are incorrect? I see

./Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.csproj
./Microsoft.Extensions.Logging.Abstractions/tests/Microsoft.Extensions.Logging.Generators.Tests/Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.csproj

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous section (Condition="'$(TargetsMobile)' == 'true' and '$(TargetOS)' != 'Browser'") also excludes the same tests.

I obtained the conditions from #54833 which is per #51961 (but for System.Text.Json).

The paths appear correct to me and are consistent with other references to them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't the path start with Microsoft.Extensions.Logging.Abstractions\tests not Microsoft.Extensions.Logging\tests ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes thanks. Missed that.

@eerhardt it looks like the test exclusion added in 9035d94 has the wrong path. I left it as-is in this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since it's pointing at the wrong path and passing those old exclusions can probably be removed. I'll do it in a separate pr

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW Eric fixed that in #59608

@steveharter
Copy link
Member Author

@steveharter have you checked against main after #59352 landed?

I just did a rebase and the same crash is occuring.

src/libraries/tests.proj Outdated Show resolved Hide resolved
@ericstj
Copy link
Member

ericstj commented Sep 28, 2021

Looks like test failure is in regex source generator on checked coreclr. May not be related to this change, see #10680. cc @stephentoub

https://dev.azure.com/dnceng/public/_build/results?buildId=1388390&view=ms.vss-test-web.build-test-results-tab&runId=40412074&resultId=192547&paneView=debug

      Timed out at 9/27/2021 8:13:44 PM after 60000ms waiting for remote process.
      Wrote mini dump to: C:\h\w\B3E009D5\w\9C35086E\uploads\3232.igz15kp2.52b.dmp
      	Process ID: 3232
      	Handle: 1756
      	Name: dotnet
      	MainModule: C:\h\w\B3E009D5\p\dotnet.exe
      	StartTime: 9/27/2021 8:12:44 PM
      	TotalProcessorTime: 00:01:02.7812500
      	Threads:
      		Thread #1 (OS 0x16D0)   [MTA]
      			[PrestubMethodFrame] (Microsoft.CodeAnalysis.GeneratorRunResult.get_GeneratedSources)
      			Microsoft.CodeAnalysis.GeneratorDriverRunResult+<>c.<get_GeneratedTrees>b__9_0(Microsoft.CodeAnalysis.GeneratorRunResult)
      			System.Linq.Enumerable+SelectManySingleSelectorIterator`2[[Microsoft.CodeAnalysis.GeneratorRunResult, Microsoft.CodeAnalysis],[System.__Canon, System.Private.CoreLib]].ToArray()
      			System.Linq.Enumerable.ToArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
      			System.Collections.Immutable.ImmutableArray.CreateRange[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
      			System.Collections.Immutable.ImmutableArray.ToImmutableArray[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
      			Microsoft.CodeAnalysis.GeneratorDriverRunResult.get_GeneratedTrees()
      			System.Text.RegularExpressions.Tests.RegexGeneratorHelper+<SourceGenRegexAsync>d__6.MoveNext()
      			System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(System.String, System.Nullable`1<System.Text.RegularExpressions.RegexOptions>, System.Nullable`1<System.TimeSpan>, System.Threading.CancellationToken)
      			System.Text.RegularExpressions.Tests.RegexHelpers+<GetRegexAsync>d__8.MoveNext()
      			System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(System.Text.RegularExpressions.Tests.RegexEngine, System.String, System.Nullable`1<System.Text.RegularExpressions.RegexOptions>, System.Nullable`1<System.TimeSpan>)
      			System.Text.RegularExpressions.Tests.RegexMatchTests+<>c+<<Match_ExcessPrefix>b__29_0>d.MoveNext()
      			System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib]](System.__Canon ByRef)
      			System.Text.RegularExpressions.Tests.RegexMatchTests+<>c.<Match_ExcessPrefix>b__29_0(System.String)
      			[DebuggerU2MCatchHandlerFrame]
      			[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)
      			System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
      			System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
      			Microsoft.DotNet.RemoteExecutor.Program.Main(System.String[])
      		Thread #2 (OS 0x224) [Finalizer] [Background] [MTA]
      			[DebuggerU2MCatchHandlerFrame]
      		Thread #3 (OS 0x640)  [Background] [MTA]
      			[DebuggerU2MCatchHandlerFrame]
      		Thread #5 (OS 0x1434) [Thread pool worker] [Background] [MTA]
      			[HelperMethodFrame] (System.Threading.WaitHandle.WaitOneCore)
      			System.Threading.WaitHandle.WaitOneNoCheck(Int32)
      			System.Threading.WaitHandle.WaitOne()
      			System.Threading.PortableThreadPool+GateThread.GateThreadStart()
      			System.Threading.Thread.StartCallback()
      			[DebuggerU2MCatchHandlerFrame]

@ericstj
Copy link
Member

ericstj commented Sep 28, 2021

I see @stephentoub disabled that test here: 5d71e32

@stephentoub
Copy link
Member

I see @stephentoub disabled that test here: 5d71e32

Yeah, the actual issue is the test is attributed with a SkipOnCoreClr attribute in order to avoid running on checked runtimes but the test isn't being skipped, as the attribute is being processed in a way different from what was expected. I disabled it while the attribute gets fixed.

@ericstj
Copy link
Member

ericstj commented Sep 28, 2021

@stephentoub while you're here, this PR might be relevant for your RegEx source generator:

// TODO: Use https://github.com/dotnet/runtime/pull/59092
INamedTypeSymbol? regexSymbol = compilation.GetTypeByMetadataName(RegexName);
INamedTypeSymbol? regexGeneratorAttributeSymbol = compilation.GetTypeByMetadataName(RegexGeneratorAttributeName);

edit, looks like I should expand my search by a couple lines.

@ericstj
Copy link
Member

ericstj commented Sep 29, 2021

/azp run runtime

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@steveharter
Copy link
Member Author

Build failure in Build windows x64 Release SingleFile caused by timeout and not related to this PR.

@steveharter steveharter merged commit 04b6c41 into dotnet:main Sep 29, 2021
@steveharter steveharter deleted the SrcGenGetBestType branch September 29, 2021 18:42
@steveharter
Copy link
Member Author

/backport to release/6.0

@github-actions
Copy link
Contributor

Started backporting to release/6.0: https://github.com/dotnet/runtime/actions/runs/1288124568

@github-actions
Copy link
Contributor

@steveharter backporting to release/6.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: Use GetBestTypeByMetadataName instead of GetTypeByMetadataName
Using index info to reconstruct a base tree...
M	src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
Falling back to patching base and 3-way merge...
Auto-merging src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
CONFLICT (content): Merge conflict in src/libraries/Microsoft.Extensions.Logging.Abstractions/gen/LoggerMessageGenerator.Parser.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Use GetBestTypeByMetadataName instead of GetTypeByMetadataName
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

steveharter added a commit to steveharter/runtime that referenced this pull request Sep 29, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants