Integrate Symbol API etc. of FSharp.Compiler.Service into Visual F# Tools

[dsyme]

FSharp.Compiler.Service is an evolution of the core F# compiler code and especially the src\fsharp\vs... API for the language service. I'm investigating what it takes to continue the process of reconciling FSharp.Compiler.Service and the Visual F# Tools. This would

• improve the Visual F# Tools through various FCS fixes
• ease the process of reconciling the Visual F# Tools and the Visual F# Power Tools, and will put the Visual F# Tools on a more "modern" basis for future work.
• improve test coverage in the Visual F# Tools, since FCS has a lot of testing

A logical next step in this work is contained in this PR: we bring most of the FCS changes relevant to the Visual F# Tools IDE (and the Visual F# Power Tools) back into Microsoft/visualfsharp. Here's the status of the work:

☑️ FSharp.LanguageService.Compiler contains the new FCS API

☑️ The Visual F# IDE tools (vsintegration\src\vs\FsPkgs....) are rejigged to use this API and build

☑️ The compiler tests pass

☑️ The Visual F# IDE unit tests pass (all passing, final checks being run)

☑️ An appropriate subset of the FCS tests for the new FCS API are incorporated and run as part of CI testing in this repo

☑️ Install and use the updated Visual F# Tools

☑️ Check the new tools still work with Visual F# Power Tools

Not all the FCS changes are brought across. Here's the catalogue:

✅ the new Symbol API

✅ the much improved SourceCodeServices API

✅ the reimplemented IncrementalBuilder

✅ not-on-disk cross-project references

✅ the tests for these

❎ providing F# Interactive as a service (which we don't need)

❎ project file cracking as a service (which we don't need in Visual F# Tools)

❎ cross-platform changes from http://github.com/fsharp/fsharp (only relevant on Mono)

❎ building, testing, releasing, documenting FSharp.Compiler.Service (which we don't need yet, for the forseeble future the nuget packagewill still be released from http://github.com/fsharp/FSharp.Compiler.Service).

This PR is an initial part of this investigation, incorporating all code changes and tests from the FCS repo into this repo in one squashed commit (the commit history to the FCS repo is very large and incorporates the entire commit history of github.com/fsharp/fsharp, so we want to squash here)

[dsyme]

This diff is too big for GitHub's diff limit, so I've committed it in multiple commits. Please review them independently

[dsyme]

This is getting close to ready to integrate. Tests are passing. After my talk is over I'll install and use the tools to check things are working ok, then finalize the PR and add detailed comments explaining the most significant diffs.

[KevinRansom]

Nice ... just in time for me to try to put back into coreclr symbol generation. 132 files changed, integrating into dev15 branch will be fine, but I'm guessing coreclr branch
may be more interesting,

[dsyme]

Here's a guide to the changes:

src\absil\il.fs: - Use ConcurrentDictionary to allow symbol information to be accessed from client threads

src\absil\il.fs - A fix to IL metadata rescoping uncovered by the FCS tests

src\absil\ilreflect.fs - FCS adds a "collectible" flag. Always set to false for Visual F# Tools

src/fsharp/CompileOps.fs - GetFSharpCoreReferenceUsedByCompiler and friends now respect --simpleresolution and useFsiAuxLib passed through from language service

src/fsharp/CompileOps.fs - Implementation of cross-project references, added by FCS. Not yet used in Visual F# Tools but good to have them integrated.

src/fsharp/ErrorLogger.fs - FCS adds a value to StopProcessing exceptions to propagate the exception to clients of the FCS API. Visual F# Tools doesn't yet use this information. This adds the information in a non-intrusive way.

src/fsharp/ExtensionTyping.fs - alignment to allow code to compile against .NET 4.5 (FCS compiles against 4.5)

src/fsharp/FSharp.Compiler-proto/FSharp.Compiler-proto.fsproj - Remove INcrementalBuild.fs from the proto compiler, it is not needed

src/fsharp/FSharp.Compiler/FSharp.Compiler.fsproj - Remove the language service files from FSharp.Compiler.dll, they are not needed since that DLL is only used by FSC.EXE and FSI.EXE

src/fsharp/FSharp.LanguageService.Compiler/FSharp.LanguageService.Compiler.fsproj - Add the files for the full FCS implementation into the build of this DLL. There is no point not having the full set of files in this DLL.

src/fsharp/InternalCollections.fs - Fixes to support "requiredToKeep" and "onStrongDiscard"

src/fsharp/LexFilter.fs - Move an inner closure to an outer function definition for a minor performance gain, a change that was present in FCS

src/fsharp/NameResolution.fs - Fixes and functionality related to name resolution reporting from the compiler. This supports the new functionality in the FCS API

src/fsharp/NicePrint.fs - Display .NET attributes, used in the FCS Symbol API

src/fsharp/QuotationTranslator.fsi - add helpers used by the FCS "expression" API to this signature file. While not the "expression" API is not included in this commit, we include this to help align the FCS implementation with Visual F# Tools.

src/fsharp/TastPickle.fs - Allow data to be unpickled from in-memory data

src/fsharp/TypeChecker.fs - Corrections to record extra symbol resolutions, and record get/set symbol resolutions for properties at the property name, with an auxiliary entry at the "get" and "set" keyword. This is under test in the FCS tests and the Visual F# Tools unittests. Also record ranges of symbols for both signature files and implementation riles systematically for all symbol kinds. Also add an "IsCompilerGenerated" flag to override information (used to mark the auto-generated CompareTo and Equals overrides for record and union types)

src/fsharp/fsc.fs - Move the GCSettings adjustment to fscmain.fs

src/fsharp/infos.fs - Helpers to return more information about symbols in the FCS symbol API, under test in the FCS tests added in ProjectSystem.fs

src/fsharp/pars.fsy - Allow expressions and types to be parsed, used by the FCS evaluation API

src/fsharp/range.fs - Helpers to track the use of 0 and 1 based line numbers

src/fsharp/tast.fs - Track both signature and implementation locations systematically for all symbols

src/fsharp/vs/IncrementalBuild.fs, fsi - Cleanup of the incremental builder, a major part of the FCS work. The same build graph is used but extra nodes are added to allow the results of a background build of one project to be used by another project.

src/fsharp/vs/Reactor.fs - Cleanup of the Reactor thread implementation, now simpler with operations matching those in service.fs

src/fsharp/vs/ServiceDeclarations.fs - Adjusted implementations of various operations in the FCS Symbol API. Under test in the extensive Project tests in the FCS tests.

src/fsharp/vs/ServiceParamInfoLocations.fs - Cosmetic cleanup plus return 1-based range/position markers instead of 0-based integer pairs (the former is simpler and less error-prone)

src/fsharp/vs/ServiceParseTreeWalk.fs - Likewise, return 1-based range/position markers instead of 0-based integer pairs (the former is simpler and less error-prone)

src/fsharp/vs/Symbols.fs - the implementation of the FCS symbols API

src/fsharp/vs/Symbols.fsi - the signature of the implementation of the FCS symbols API. Still internal in the Visual F# Tools.

src/fsharp/vs/service.fs - The new FCS API, and modifications stemming from other files.

src/fsharp/vs/service.fsi - The signature of the new FCS API

src/utils/CompilerLocationUtils.fs - Use the correct SDK install location when compiling with VS2015

vsintegration/update-vsintegration.cmd - This script is used to clobber a current install location for the Visual F# Tools when using VS2015. AFAIK it should clobber the 4.0 location which is where the VS2015 tools get installed

packages.config - make the NUnit.Runners version match

src/FSharpSource.Settings.targets - define VS_VERSION_DEV14 when compiling for VS2015

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/BackgroundRequests.fs - splits out the part of servicem.fs dealing with background requests

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/BackgroundRequests.fs - splits out the part of servicem.fs dealing with background requests. Some of the implementation of background requests is still in the C# code.

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/Colorize.fs - Minor updates and renames, and split our the part of servicem.fs that deals with ColorableItem

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/Colorize.fsi - remove this largely pointless signature file

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/Com.fs - remove this helper file and put the small part of it that was being used in Vs.fs

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/Error.fs - remove some old tracing code that no on makes use of any more

FSharp.LanguageService/FSharp.LanguageService.Base/FSharp.LanguageService.Base.csproj - cosmetic change

FSharp.LanguageService/FSharp.LanguageService.Base/LanguageService.cs - some renamings, remove some base implementations of methods that were never called, and move the implementation of a couple of simple VS interfaces out of C~ into F# code. This avoids the very obfuscating F# --> C# --> F# indirections through abstract methods which have the same name as interface methods

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/FSharpLanguageService.fs - the part of servicem.fs completing the implementation of LanguageService

vsintegration/src/vs/FsPkgs/FSharp.LanguageService/FSharpPackage.fs - the part of servicem.fs implementing the package logic for the language service package

FSharp.LanguageService/FSharpSource.fs - renamings, remove unnecessary namespace "open", rename "IdealSource" to "IFSharpSource", rename "DelegatingSource" to "FSharpSourceTestable", rename SourceOverIdealSource to FSharpSource, remove some #if FX_ATLEAST_45, adjust calls to use the new FCS API,

FsPkgs/FSharp.LanguageService/GotoDefinition.fs - use the new FCS API

LanguageService/ProjectSite.fs → ...gs/FSharp.LanguageService/IProjectSite.fs - simple rename, IProvideProjectSite is moved to ProjectSitesAndFiles.fs

FsPkgs/FSharp.LanguageService/NavigationBar.fs - everything from servicem.fs to do with the navigation bar and regions

sPkgs/FSharp.LanguageService/Source.fsi - remove the needless signature file

vs/FsPkgs/FSharp.LanguageService/Servicem.fsi - remove the needless signature file

vs/FsPkgs/FSharp.LanguageService/Project.fsi - remove the needless signature file

LanguageService/ProjectSitesAndFiles.fs - rename from the obscure "Artifacts.fs" and cosmetic cleanup

src/vs/FsPkgs/FSharp.LanguageService/QuickParse.fs - cosmetic cleanup and documentation

FsPkgs/FSharp.LanguageService/Vs.fs - Remove the needless ServiceProvider type

src/Salsa/.... - cleanup of various things in the "Sals" test layer, to make things clearer. Also remove the signature file, the layer is clearer without it.

[enricosada]

well, huge 👍 😄