Internal compiler error when creating Portable PDB files for source files with very long lines

[praeclarum]

When compiling code with long source lines and generating debug symbols, the F# compiler complains:

error FS0193 : internal error : Specified argument was out of the range of valid values.Parameter name: value

From the error log it looks like the PDB writer is failing to write some Int32 or UInt32 values.

This was originally reported here: https://bugzilla.xamarin.com/show_bug.cgi?id=60467

Repro steps

Provide the steps required to reproduce the problem

1. Create a complex app in F#

2. Compile it with Portable PDB symbols

Expected behavior

Compilation succeeds.

Actual behavior

Compiler fails with:

    error FS0193 : internal error : Specified argument was out of the range of valid values.Parameter name: value
    
    Unhandled Exception:
    System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
    Parameter name: value
      at System.Reflection.Throw.ValueArgumentOutOfRange () [0x00000] in <6e996dbb3f9b4d42bb708be9f37ced22>:0 
      at System.Reflection.Metadata.BlobWriterImpl.WriteCompressedInteger (System.Reflection.Metadata.BlobBuilder writer, System.UInt32 value) [0x0003b] in <6e996dbb3f9b4d42bb708be9f37ced22>:0 
      at System.Reflection.Metadata.BlobBuilder.WriteCompressedInteger (System.Int32 value) [0x00000] in <6e996dbb3f9b4d42bb708be9f37ced22>:0 
      at Microsoft.FSharp.Compiler.AbstractIL.ILPdbWriter.generatePortablePdb$cont@341 (Microsoft.FSharp.Compiler.AbstractIL.IL+ILSourceDocument[] docs, System.Reflection.Metadata.Ecma335.MetadataBuilder metadata, System.Collections.Generic.Dictionary`2[TKey,TValue] documentIndex, Microsoft.FSharp.Compiler.AbstractIL.ILPdbWriter+PdbSequencePoint[] sps, System.Reflection.Metadata.BlobBuilder builder, Microsoft.FSharp.Core.Unit unitVar) [0x0018f] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.AbstractIL.ILPdbWriter.f@1-13 (Microsoft.FSharp.Compiler.AbstractIL.IL+ILSourceDocument[] docs, System.Reflection.Metadata.Ecma335.MetadataBuilder metadata, System.Collections.Generic.Dictionary`2[TKey,TValue] documentIndex, Microsoft.FSharp.Core.FSharpRef`1[T] lastLocalVariableHandle, Microsoft.FSharp.Compiler.AbstractIL.ILPdbWriter+PdbMethodData minfo) [0x00075] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.AbstractIL.ILPdbWriter.generatePortablePdb (System.Boolean embedAllSource, Microsoft.FSharp.Collections.FSharpList`1[T] embedSourceList, System.String sourceLink, System.Boolean showTimes, Microsoft.FSharp.Compiler.AbstractIL.ILPdbWriter+PdbData info, System.Boolean isDeterministic) [0x002a5] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.AbstractIL.ILBinaryWriter.writeBinaryAndReportMappings (System.String outfile, Microsoft.FSharp.Compiler.AbstractIL.IL+ILGlobals ilg, Microsoft.FSharp.Core.FSharpOption`1[T] pdbfile, Microsoft.FSharp.Core.FSharpOption`1[T] signer, System.Boolean portablePDB, System.Boolean embeddedPDB, System.Boolean embedAllSource, Microsoft.FSharp.Collections.FSharpList`1[T] embedSourceList, System.String sourceLink, System.Boolean emitTailcalls, System.Boolean deterministic, System.Boolean showTimes, System.Boolean dumpDebugInfo, Microsoft.FSharp.Compiler.AbstractIL.IL+ILModuleDef modul) [0x019b3] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.AbstractIL.ILBinaryWriter.WriteILBinary (System.String filename, Microsoft.FSharp.Compiler.AbstractIL.ILBinaryWriter+options options, Microsoft.FSharp.Compiler.AbstractIL.IL+ILModuleDef input) [0x00001] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.Driver.main4[a] (Microsoft.FSharp.Core.FSharpOption`1[T] dynamicAssemblyCreator, Microsoft.FSharp.Compiler.Driver+Args`1[T] _arg1) [0x00189] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.ErrorLogger+ErrorLoggerExtensions.ReraiseIfWatsonable (System.Exception exn) [0x00040] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.ErrorLogger+ErrorLoggerExtensions.ErrorLogger.ErrorRecovery (Microsoft.FSharp.Compiler.ErrorLogger+ErrorLogger x, System.Exception exn, Microsoft.FSharp.Compiler.Range+range m) [0x000d2] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.Driver.main4[a] (Microsoft.FSharp.Core.FSharpOption`1[T] dynamicAssemblyCreator, Microsoft.FSharp.Compiler.Driver+Args`1[T] _arg1) [0x001ae] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.Driver.typecheckAndCompile (Microsoft.FSharp.Compiler.AbstractIL.Internal.Library+CompilationThreadToken ctok, System.String[] argv, Microsoft.FSharp.Compiler.ReferenceResolver+Resolver legacyReferenceResolver, System.Boolean bannerAlreadyPrinted, System.Boolean openBinariesInMemory, System.Boolean defaultCopyFSharpCore, Microsoft.FSharp.Compiler.ErrorLogger+Exiter exiter, Microsoft.FSharp.Compiler.Driver+ErrorLoggerProvider loggerProvider, Microsoft.FSharp.Core.FSharpOption`1[T] tcImportsCapture, Microsoft.FSharp.Core.FSharpOption`1[T] dynamicAssemblyCreator) [0x00039] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.Driver.mainCompile (Microsoft.FSharp.Compiler.AbstractIL.Internal.Library+CompilationThreadToken ctok, System.String[] argv, Microsoft.FSharp.Compiler.ReferenceResolver+Resolver legacyReferenceResolver, System.Boolean bannerAlreadyPrinted, System.Boolean openBinariesInMemory, System.Boolean defaultCopyFSharpCore, Microsoft.FSharp.Compiler.ErrorLogger+Exiter exiter, Microsoft.FSharp.Compiler.Driver+ErrorLoggerProvider loggerProvider, Microsoft.FSharp.Core.FSharpOption`1[T] tcImportsCapture, Microsoft.FSharp.Core.FSharpOption`1[T] dynamicAssemblyCreator) [0x00001] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.CommandLineMain+Driver.main (System.String[] argv) [0x0003e] in <59c44fc0c960099ba7450383c04fc459>:0 
      at Microsoft.FSharp.Compiler.CommandLineMain.main (System.String[] argv) [0x00042] in <59c44fc0c960099ba7450383c04fc459>:0 
      at Microsoft.FSharp.Compiler.ErrorLogger+ErrorLoggerExtensions.ReraiseIfWatsonable (System.Exception exn) [0x00040] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.ErrorLogger+ErrorLoggerExtensions.ErrorLogger.ErrorRecovery (Microsoft.FSharp.Compiler.ErrorLogger+ErrorLogger x, System.Exception exn, Microsoft.FSharp.Compiler.Range+range m) [0x000d2] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.ErrorLogger.errorRecovery (System.Exception exn, Microsoft.FSharp.Compiler.Range+range m) [0x00006] in <59c44fb6f1806346a7450383b64fc459>:0 
      at Microsoft.FSharp.Compiler.CommandLineMain.main (System.String[] argv) [0x00056] in <59c44fc0c960099ba7450383c04fc459>:0 

Known workarounds

If I disable debugging information, compilation succeeds.

[cartermp]

@praeclarum Is it possible to share the source code where you're seeing this? Even if you can't share publicly, we could sign an NDA and take a look through private channels.

There are still some issues in Portable PDBs we'll have to iron out after completing the .NET Core/.NET Standard support work we're finishing up now.

[praeclarum]

Yeah I should be able to share the code with you. Checking now.

[nosami]

@cartermp: check out Xamarin F# slack for repro details

[mattwarren]

As part of writing A DoS Attack against the C# Compiler I stumbled across another way to repo this, using this code and dotnet build -c Release -v:detailed

class Class<A, B, C, D, E, F>
{
    class Inner : Class<Inner, Inner, Inner, Inner, Inner, Inner>
    {
        Inner.Inner.Inner.Inner.Inner.Inner.Inner.Inner.Inner.Inner inner;
    }
}

Warning, it will take almost 3 minutes to compile it and will use over 2GB of memory, so I'm not sure how useful the repo is!!!

But it will eventually fail with this stack trace (not identical to the one above, but ends up in the same place, with the same error message):

Unhandled Exception: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: value
   at System.Reflection.Throw.ValueArgumentOutOfRange()
   at System.Reflection.Metadata.BlobWriterImpl.WriteCompressedInteger(BlobWriter& writer, UInt32 value)
   at System.Reflection.Metadata.Ecma335.MetadataBuilder.WriteAlignedBlobHeap(BlobBuilder builder)
   at System.Reflection.Metadata.Ecma335.MetadataBuilder.WriteHeapsTo(BlobBuilder builder, BlobBuilder stringHeap)
   at System.Reflection.Metadata.Ecma335.MetadataRootBuilder.Serialize(BlobBuilder builder, Int32 methodBodyStreamRva, Int32 mappedFieldDataStreamRva)
   at System.Reflection.PortableExecutable.ManagedPEBuilder.SerializeTextSection(SectionLocation location)
   at System.Reflection.PortableExecutable.ManagedPEBuilder.SerializeSection(String name, SectionLocation location)
   at Microsoft.Cci.ExtendedPEBuilder.SerializeSection(String name, SectionLocation location)
   at System.Reflection.PortableExecutable.PEBuilder.SerializeSections()
   at System.Reflection.PortableExecutable.PEBuilder.Serialize(BlobBuilder builder)
   at Microsoft.Cci.PeWriter.WritePeToStream(EmitContext context, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, Boolean metadataOnly, Boolean isDeterministic, Boolean emitTestCoverageData, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.SerializePeToStream(CommonPEModuleBuilder moduleBeingBuilt, DiagnosticBag metadataDiagnostics, CommonMessageProvider messageProvider, Func`1 getPeStream, Func`1 getMetadataPeStreamOpt, Func`1 getPortablePdbStreamOpt, PdbWriter nativePdbWriterOpt, String pdbPathOpt, Boolean metadataOnly, Boolean includePrivateMembers,Boolean isDeterministic, Boolean emitTestCoverageData, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream(CommonPEModuleBuilder moduleBeingBuilt, EmitStreamProviderpeStreamProvider, EmitStreamProvider metadataPEStreamProvider, EmitStreamProvider pdbStreamProvider, Func`1 testSymWriterFactory, DiagnosticBag diagnostics, Boolean metadataOnly, Boolean includePrivateMembers, Boolean emitTestCoverageData,String pePdbFilePath, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CommonCompiler.RunCore(TextWriter consoleOutput, ErrorLogger errorLogger, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CommonCompiler.Run(TextWriter consoleOutput, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.<>c__DisplayClass1_0.<Run>b__0(TextWriter tw)
   at Microsoft.CodeAnalysis.CommandLine.ConsoleUtil.RunWithUtf8Output[T](Func`2 func)
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader)
   at Microsoft.CodeAnalysis.CommandLine.CoreClrBuildClient.RunLocalCompilation(String[] arguments, BuildPaths buildPaths, TextWriter textWriter)
   at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter)
   at Microsoft.CodeAnalysis.CommandLine.CoreClrBuildClient.Run(IEnumerable`1 arguments, RequestLanguage language, CompileFunc compileFunc)
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args)

[cartermp]

cc @tmat for FYI

[13xforever]

I've encountered this bug when compiling this file today: https://gitlab.com/13xforever/advent-of-code-2015/blob/master/Day19.fs

Minimal repro is just a field with a long string: https://gitlab.com/13xforever/advent-of-code-2015/tree/compiler_crash_repro

Looks like it crashes if line exceeds 512 characters

[praeclarum]

I continue to run into this bug. Any chance it will be fixed?

It even happens when disabling debug output. Only happens with optimizations turned on.

[cartermp]

@praeclarum possibly. Would you happen to know which version of F# and/or the compiler you're using? If you're using the mono F# compiler, which IIRC hasn't been updated since F# 4.5, it will be missing this change that changes how ranges are represented in source files to fix numerous issues that used to exist: #4476

I can't say for sure if this affects that, but it might be worth checking against a .NET Core console app to see if the same thing happens?

[KevinRansom]

I will take a look.

[praeclarum]

Sorry I keep forgetting that the compiler is behind on Xamarin. Part of the problem is the VS4Mac F# project settings don't let you control the debug settings, so you have to hack around the project file to disable PDBs. I keep forgetting that and run into this error.

[StrikerRUS]

Hi there!

I see the same error with .NET Core 3.1 on Ubuntu Xenial.

  wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb
  sudo dpkg -i packages-microsoft-prod.deb
  sudo apt-get update
  sudo apt-get install --no-install-recommends -y dotnet-sdk-3.1

source code.txt

Microsoft (R) Build Engine version 16.6.0+5ff7b0c9e for .NET Core

Copyright (C) Microsoft Corporation. All rights reserved.


  Determining projects to restore...
  All projects are up-to-date for restore.
error FS0193 : internal error : Specified argument was out of the range of valid values. (Parameter 'value') [/tmp/tmpn_i9efvi/test_model.fsproj]
error FS0193 : internal error : Specified argument was out of the range of valid values. (Parameter 'value') [/tmp/tmpn_i9efvi/test_model.fsproj]

Build FAILED.

error FS0193 : internal error : Specified argument was out of the range of valid values. (Parameter 'value') [/tmp/tmpn_i9efvi/test_model.fsproj]
error FS0193 : internal error : Specified argument was out of the range of valid values. (Parameter 'value') [/tmp/tmpn_i9efvi/test_model.fsproj]
    0 Warning(s)
    2 Error(s)

[StrikerRUS]

@cartermp @KevinRansom Hey! Is there any news?

[KevinRansom]

I'm going to close this, without a repro there is not much we can do. If anyone knows how to repro this and can provide the repro, I can take a look.

Thanks

Kevin

[KevinRansom]

oh wait, I just noticed, strikers like to source for a repro.

[KevinRansom]

Good grief that really is a long line ... 704330 characters.