Reference documentation generation with CSharp Extension methods fails

[matthid]

When a function and a module is marked with the System.Runtime.CompilerServices.Extension attribute, for example

[<System.Runtime.CompilerServices.Extension>]
module Test =
    [<System.Runtime.CompilerServices.Extension>]
    let MyExtension (o : obj) = o

exceptions like the following appear while generating the API reference documentation:

Error while generating docs for C:\Projects\myproject\build\DLL_WITH_CSHARP_EXTENSIONS.dll: System.AggregateException: One or more errors occurred. ---> Microsoft.FSharp.Compiler.ErrorLogger+ReportedError: Exception of type 'Microsoft.FSharp.Compiler.Tast+InternalUndefinedItemRef' was thrown.
   at Microsoft.FSharp.Compiler.ErrorLogger.ErrorLoggerExtensions.ErrorLogger.Error[b](ErrorLogger x, Exception exn) in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\ErrorLogger.fs:line 321
   at Microsoft.FSharp.Compiler.Tast.EntityRef.get_Deref() in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\tast.fs:line 2546
   at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpEntity.get_CompiledName() in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\vs\Symbols.fs:line 147
   at FSharp.MetadataFormat.ValueReader.hasAttrib@193.Invoke(FSharpAttribute a) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 193
   at Microsoft.FSharp.Collections.SeqModule.Exists[T](FSharpFunc`2 predicate, IEnumerable`1 source)
   at FSharp.MetadataFormat.ValueReader.buildUsage@314(FSharpMemberFunctionOrValue v, FSharpOption`1 args) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 315
   at FSharp.MetadataFormat.ValueReader.buildShortUsage@375.Invoke(Int32 length) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 375
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at CompiledRazorTemplates.Dynamic.dadaedfcebc.Execute()
   at RazorEngine.Templating.TemplateBase.RazorEngine.Templating.ITemplate.Run(ExecuteContext context) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateBase.cs:line 126
   at RazorEngine.Templating.TemplateService.Run(ITemplate template, DynamicViewBag viewBag) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 608
   at RazorEngine.Templating.TemplateService.Parse(String razorTemplate, Object model, DynamicViewBag viewBag, String cacheName) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 437
   at FSharp.MetadataFormat.DocPageTemplateBase`1.RenderPart(String name, Object model) in c:\Tomas\Public\FSharp.Formatting\src\Common\Razor.fs:line 140
   at CompiledRazorTemplates.Dynamic.afbdccf.Execute()
   at RazorEngine.Templating.TemplateBase.RazorEngine.Templating.ITemplate.Run(ExecuteContext context) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateBase.cs:line 126
   at RazorEngine.Templating.TemplateService.Run(ITemplate template, DynamicViewBag viewBag) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 608
   at RazorEngine.Templating.TemplateService.Parse(String razorTemplate, Object model, DynamicViewBag viewBag, String cacheName) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 437
   at FSharp.MetadataFormat.RazorRender.ProcessFile[a](String source, FSharpOption`1 properties) in c:\Tomas\Public\FSharp.Formatting\src\Common\Razor.fs:line 89
   at <StartupCode$FSharp-MetadataFormat>.$Main.Generate@903-3.Invoke(Module modul, ParallelLoopState _arg1, RazorRender razor) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 906
   at System.Threading.Tasks.Parallel.<>c__DisplayClass32`2.<PartitionerForEachWorker>b__30()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner`1 source, ParallelOptions parallelOptions, Action`1 simpleBody, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource,TLocal](IEnumerable`1 source, Func`1 localInit, Func`4 body, Action`1 localFinally)
   at FSharp.MetadataFormat.Parallel.pfor[TSource,TLocal](IEnumerable`1 input, FSharpFunc`2 localInit, FSharpFunc`2 body) in c:\Tomas\Public\FSharp.Formatting\src\Common\Razor.fs:line 18
   at FSharp.MetadataFormat.MetadataFormat.Generate(FSharpList`1 dllFiles, String outDir, IEnumerable`1 layoutRoots, FSharpOption`1 parameters, FSharpOption`1 namespaceTemplate, FSharpOption`1 moduleTemplate, FSharpOption`1 typeTemplate, FSharpOption`1 xmlFile, FSharpOption`1 sourceRepo, FSharpOption`1 sourceFolder, FSharpOption`1 publicOnly, FSharpOption`1 libDirs, FSharpOption`1 otherFlags, FSharpOption`1 markDownComments, FSharpOption`1 urlRangeHighlight) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 903
   at FSharp.MetadataFormat.MetadataFormat.Generate(String dllFile, String outDir, IEnumerable`1 layoutRoots, FSharpOption`1 parameters, FSharpOption`1 namespaceTemplate, FSharpOption`1 moduleTemplate, FSharpOption`1 typeTemplate, FSharpOption`1 xmlFile, FSharpOption`1 sourceRepo, FSharpOption`1 sourceFolder, FSharpOption`1 publicOnly, FSharpOption`1 libDirs, FSharpOption`1 otherFlags, FSharpOption`1 markDownComments, FSharpOption`1 urlRangeHighlight) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 770
   at FSI_0001.buildReference@356.Invoke(String binary) in C:\Projects\myproject\build.fsx:line 360
---> (Inner Exception #0) Microsoft.FSharp.Compiler.ErrorLogger+ReportedError: Exception of type 'Microsoft.FSharp.Compiler.Tast+InternalUndefinedItemRef' was thrown.
   at Microsoft.FSharp.Compiler.ErrorLogger.ErrorLoggerExtensions.ErrorLogger.Error[b](ErrorLogger x, Exception exn) in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\ErrorLogger.fs:line 321
   at Microsoft.FSharp.Compiler.Tast.EntityRef.get_Deref() in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\tast.fs:line 2546
   at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpEntity.get_CompiledName() in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\vs\Symbols.fs:line 147
   at FSharp.MetadataFormat.ValueReader.hasAttrib@193.Invoke(FSharpAttribute a) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 193
   at Microsoft.FSharp.Collections.SeqModule.Exists[T](FSharpFunc`2 predicate, IEnumerable`1 source)
   at FSharp.MetadataFormat.ValueReader.buildUsage@314(FSharpMemberFunctionOrValue v, FSharpOption`1 args) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 315
   at FSharp.MetadataFormat.ValueReader.buildShortUsage@375.Invoke(Int32 length) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 375
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at CompiledRazorTemplates.Dynamic.dadaedfcebc.Execute()
   at RazorEngine.Templating.TemplateBase.RazorEngine.Templating.ITemplate.Run(ExecuteContext context) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateBase.cs:line 126
   at RazorEngine.Templating.TemplateService.Run(ITemplate template, DynamicViewBag viewBag) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 608
   at RazorEngine.Templating.TemplateService.Parse(String razorTemplate, Object model, DynamicViewBag viewBag, String cacheName) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 437
   at FSharp.MetadataFormat.DocPageTemplateBase`1.RenderPart(String name, Object model) in c:\Tomas\Public\FSharp.Formatting\src\Common\Razor.fs:line 140
   at CompiledRazorTemplates.Dynamic.afbdccf.Execute()
   at RazorEngine.Templating.TemplateBase.RazorEngine.Templating.ITemplate.Run(ExecuteContext context) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateBase.cs:line 126
   at RazorEngine.Templating.TemplateService.Run(ITemplate template, DynamicViewBag viewBag) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 608
   at RazorEngine.Templating.TemplateService.Parse(String razorTemplate, Object model, DynamicViewBag viewBag, String cacheName) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 437
   at FSharp.MetadataFormat.RazorRender.ProcessFile[a](String source, FSharpOption`1 properties) in c:\Tomas\Public\FSharp.Formatting\src\Common\Razor.fs:line 89
   at <StartupCode$FSharp-MetadataFormat>.$Main.Generate@903-3.Invoke(Module modul, ParallelLoopState _arg1, RazorRender razor) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 906
   at System.Threading.Tasks.Parallel.<>c__DisplayClass32`2.<PartitionerForEachWorker>b__30()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)<---

---> (Inner Exception #1) Microsoft.FSharp.Compiler.ErrorLogger+ReportedError: Exception of type 'Microsoft.FSharp.Compiler.Tast+InternalUndefinedItemRef' was thrown.
   at Microsoft.FSharp.Compiler.ErrorLogger.ErrorLoggerExtensions.ErrorLogger.Error[b](ErrorLogger x, Exception exn) in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\ErrorLogger.fs:line 321
   at Microsoft.FSharp.Compiler.Tast.EntityRef.get_Deref() in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\tast.fs:line 2546
   at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpEntity.get_CompiledName() in d:\Local\Public\FSharp.Compiler.Service\src\fsharp\vs\Symbols.fs:line 147
   at FSharp.MetadataFormat.ValueReader.hasAttrib@193.Invoke(FSharpAttribute a) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 193
   at Microsoft.FSharp.Collections.SeqModule.Exists[T](FSharpFunc`2 predicate, IEnumerable`1 source)
   at FSharp.MetadataFormat.ValueReader.buildUsage@314(FSharpMemberFunctionOrValue v, FSharpOption`1 args) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 315
   at FSharp.MetadataFormat.ValueReader.buildShortUsage@375.Invoke(Int32 length) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 375
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at CompiledRazorTemplates.Dynamic.affbbebdb.Execute()
   at RazorEngine.Templating.TemplateBase.RazorEngine.Templating.ITemplate.Run(ExecuteContext context) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateBase.cs:line 126
   at RazorEngine.Templating.TemplateService.Run(ITemplate template, DynamicViewBag viewBag) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 608
   at RazorEngine.Templating.TemplateService.Parse(String razorTemplate, Object model, DynamicViewBag viewBag, String cacheName) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 437
   at FSharp.MetadataFormat.DocPageTemplateBase`1.RenderPart(String name, Object model) in c:\Tomas\Public\FSharp.Formatting\src\Common\Razor.fs:line 140
   at CompiledRazorTemplates.Dynamic.bfabffcf.Execute()
   at RazorEngine.Templating.TemplateBase.RazorEngine.Templating.ITemplate.Run(ExecuteContext context) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateBase.cs:line 126
   at RazorEngine.Templating.TemplateService.Run(ITemplate template, DynamicViewBag viewBag) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 608
   at RazorEngine.Templating.TemplateService.Parse(String razorTemplate, Object model, DynamicViewBag viewBag, String cacheName) in c:\_git\RazorEngine\src\Core\RazorEngine.Core\Templating\TemplateService.cs:line 437
   at FSharp.MetadataFormat.RazorRender.ProcessFile[a](String source, FSharpOption`1 properties) in c:\Tomas\Public\FSharp.Formatting\src\Common\Razor.fs:line 89
   at <StartupCode$FSharp-MetadataFormat>.$Main.Generate@903-3.Invoke(Module modul, ParallelLoopState _arg1, RazorRender razor) in c:\Tomas\Public\FSharp.Formatting\src\FSharp.MetadataFormat\Main.fs:line 906
   at System.Threading.Tasks.Parallel.<>c__DisplayClass32`2.<PartitionerForEachWorker>b__30()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)<---

[ovatsus]

Interesting, I don't even get an exception, but extension methods don't appear on the docs: #156

[matthid]

I'm still seeing this btw on various occasions maybe I take a closer look in the future...

[matthid]

I noticed that this only happens when the project with the extension methods (the code from the initial post) is build against net45 (and not when build against net40).

Another note: You can workaround this with [omit], just make sure to use markDownComments is set to true. I will send a PR to make [omit] work with markDownComments = false as well.

[matthid]

Ok after countless hours of testing and debugging I think I figured this one out. Strictly speaking this is no bug in FCS, as it happens if we use the net40 reference assemblies in combination with an assembly compiled against net45 in the checker.GetProjectOptionsFromCommandLineArgs API (the exception, see the stacktrace above, happens if try to get the CompiledName property of the ExtensionAttribute). However I think there is a lot of room to improve the exception message (I noticed some other messages have been improved over time as well). /cc @dsyme I can try to provide a repro for FCS if needed.

Now to the FSF issue: Ideally we would look at the assembly we currently want to analyze and use the matching reference assemblies, however I didn't found an ideal way to do this:

• We either need to load the assembly (LoadFrom / ReflectionOnlyLoadFrom), which means it cannot be unloaded later on. Needless to say: It doesn't work in all cases (assemblies build for x86 for example).
• We use Mono.Cecil -> Another dependency
• We could probably use FSharp.Compiler.Service to load the IL and analyze it (in theory this should be possible, however I didn't find any promising entry point. Can anybody help here? Is it internal only?; this is what I'd like to do)

Currently I just updated the reference assemblies to use the 4.5 ones, this seems to fix the problem temporarily (but this or other problems may come back sooner or later considering 4.0, 4.6, 4.5.1, 4.5.2 and so on). (However, I could not reproduce this exact problem with later versions and 4.0 assemblies work as well).

/cc @tpetricek Do you think just updating to 4.5 is fine for now? Should we go into any of the solutions mentioned above? Is there someone who could help me with the FCS API (I don't think this is a use case the API was designed for, but we wouldn't need another dependency)?