Cache reference timestamps behind an IVsAsyncFileChangeEx2 watcher for snapshot reuse

docs/ide/file-watching.md

@@ -0,0 +1,66 @@
+# File watching in FSharp.Editor
+
+`vsintegration/src/FSharp.Editor/LanguageService/FileChangeWatcher.fs` is the F# counterpart of
+Roslyn's `FileChangeWatcher` and `ReferenceFileChangeTracker` (both internal to
+`Microsoft.VisualStudio.LanguageServices` and not exposed through `ExternalAccess.FSharp`).
+
+## What the workspace already gives us
+
+Not every on-disk change needs this watcher. A `-r:` that the Roslyn workspace holds as a
+`MetadataReference` is already watched by Roslyn: `ProjectSystemProjectFactory` advises every
+reference path, and when one changes it swaps the reference on the solution, which bumps
+`Project.Version`. `FSharpProjectOptionsReactor` sees that version through `isProjectInvalidated`,
+recomputes, and calls `InvalidateConfiguration` — measured in VS, that path wins the race against
+a watcher subscribed to the same file, because Roslyn batches over 500 ms where this tracker
+additionally debounces for 2 s.
+
+So a second subscription to the same reference set buys nothing. What the workspace does *not*
+cover is everything it has no document or reference for, and every stat FCS still performs
+internally:
+
+- `#load` sources of a script: not documents, not references, invisible to the workspace.
+- `IsReferencesInvalidated` on the incremental builder, which stats every reference on every
+  request.
+- `ReferencesOnDisk` on snapshot reuse, which did the same per comparison — the consumer below.
+
+## Shape
+
+- **Service.** `IVsAsyncFileChangeEx2`, obtained asynchronously; nothing ever blocks on the UI
+  thread. Callbacks arrive through `IVsFreeThreadedFileChangeEvents2` and stay on background
+  threads.
+- **Batching.** Subscribe/unsubscribe operations go through a single-consumer queue with a 500 ms
+  window (Roslyn's empirical value for solution open/close). Consecutive operations of the same
+  kind, and for file watches the same sink, are coalesced into one service call.
+- **Directory watches.** A context starts with recursive `.dll` watches on the places reference
+  assemblies live: `DOTNET_ROOT/packs` and the machine-wide `dotnet/packs`, the .NET Framework
+  reference assemblies, and the NuGet cache (`NUGET_PACKAGES` or `~/.nuget/packages`). A file
+  under one of them costs no cookie of its own. Roslyn does not watch the NuGet cache; a consumer
+  that watches every `-r:` uniformly wants it, since package assemblies are the bulk of them and
+  the alternative is a per-file advise for each.
+- **Per-file watches.** Paths outside those directories (project outputs, loose assemblies) get
+  an individual advise, ref-counted across consumers by `FSharpReferenceChangeTracker`. The
+  tracker also keeps the last-write stamp of each watched path (see Consumers).
+- **Debounce.** A rebuild writes a temp file and renames it, producing several notifications; the
+  tracker fires one callback per path after 2 s of quiet.
+
+## Consumers
+
+**Snapshot reference stamps.** `FSharpProjectOptionsReactor` watches the `-r:` set of every
+project it computes options for, diffed on recompute so an unchanged set touches nothing. The
+tracker keeps each path's last-write stamp inside its watch entry and drops it on the raw change
+notification, before the debounce. The `ReferencesOnDisk` guard in `createProjectSnapshot` reads
+stamps through `IReferenceStamps`, so the comparison that runs for every new `Project` instance is
+a dictionary read per reference instead of a stat. A path nobody watches is stat'd directly. A
+mismatch against the snapshot's own stamps (FCS stats when it builds a snapshot) drops the
+project's stamps, so a missed notification costs one re-stat pass rather than a rebuild per
+`Project` instance. The reactor watch exists for these stamps, not to invalidate the FCS build —
+Roslyn already does that, as the previous section says.
+
+Still to come:
+
+1. Scripts: watch `#load` sources so an edit outside the editor drops the cached options for
+   that document.
+2. `FSharpProjectSnapshot.FromOptions` stats every `-r:` when a snapshot is built from scratch;
+   an overload taking host-supplied stamps lets it read the same cache.
+3. A reference-change notification for the incremental builder on the FCS side, the analogue of
+   `useChangeNotifications` for sources, so `IsReferencesInvalidated` stops stat'ing at all.

docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

@@ -148,6 +148,7 @@
 * IL: fix leaking binary view ([PR #20250](https://github.com/dotnet/fsharp/pull/20250))
 
 ### Added
+* `Internal.Utilities.Library`: `String.StartsWithOrdinalIgnoreCase` extension, the `StartsWith` sibling of `EndsWithOrdinalIgnoreCase`. ([PR #20457](https://github.com/dotnet/fsharp/pull/20457))
 
 * Added a "most concrete" tiebreaker for overload resolution (`--langversion:preview`). ([RFC FS-1340](https://github.com/fsharp/fslang-design/pull/834), [PR #19277](https://github.com/dotnet/fsharp/pull/19277))
 * Added support for `OverloadResolutionPriorityAttribute` in overload resolution (`--langversion:preview`). ([RFC FS-1338](https://github.com/fsharp/fslang-design/pull/828), [PR #19277](https://github.com/dotnet/fsharp/pull/19277))

docs/release-notes/.VisualStudio/18.vNext.md

@@ -4,6 +4,7 @@
 * Expand `<inheritdoc/>` in IDE tooltips, completion, and signature help, inheriting XML documentation from base classes, interfaces, overridden members, and constructors. ([Issue #19175](https://github.com/dotnet/fsharp/issues/19175), [PR #19188](https://github.com/dotnet/fsharp/pull/19188))
 
 ### Fixed
+* Transparent compiler snapshot reuse no longer stats every `-r:` reference on each project change; reference timestamps are cached and invalidated by `IVsAsyncFileChangeEx2` notifications. ([PR #20457](https://github.com/dotnet/fsharp/pull/20457))
 
 * Improve Find All References performance by throttling parallel typechecks. ([PR #20128](https://github.com/dotnet/fsharp/pull/20128))
 * Fixed Rename incorrectly renaming `get` and `set` keywords for properties with explicit accessors. ([Issue #18270](https://github.com/dotnet/fsharp/issues/18270), [PR #19252](https://github.com/dotnet/fsharp/pull/19252))

src/Compiler/Utilities/illib.fs

@@ -97,6 +97,9 @@ module internal PervasiveAutoOpens =
         member inline x.StartsWithOrdinal(value: string) =
             x.StartsWith(value, StringComparison.Ordinal)
 
+        member inline x.StartsWithOrdinalIgnoreCase(value: string) =
+            x.StartsWith(value, StringComparison.OrdinalIgnoreCase)
+
         member inline x.EndsWithOrdinal(value: string) =
             x.EndsWith(value, StringComparison.Ordinal)
 

src/Compiler/Utilities/illib.fsi

@@ -58,6 +58,8 @@ module internal PervasiveAutoOpens =
 
         member inline StartsWithOrdinal: value: string -> bool
 
+        member inline StartsWithOrdinalIgnoreCase: value: string -> bool
+
         member inline EndsWithOrdinal: value: string -> bool
 
         member inline EndsWithOrdinalIgnoreCase: value: string -> bool

vsintegration/src/FSharp.Editor/FSharp.Editor.fsproj

@@ -55,6 +55,7 @@
     <Compile Include="LanguageService\IProjectSite.fs" />
     <Compile Include="LanguageService\ProvideFSharpVersionRegistrationAttribute.fs" />
     <Compile Include="LanguageService\MetadataAsSource.fs" />
+    <Compile Include="LanguageService\FileChangeWatcher.fs" />
     <Compile Include="LanguageService\FSharpProjectOptionsManager.fs" />
     <Compile Include="LanguageService\IFSharpWorkspaceService.fs" />
     <Compile Include="LanguageService\SingleFileWorkspaceMap.fs" />

vsintegration/src/FSharp.Editor/LanguageService/FSharpProjectOptionsManager.fs

@@ -8,20 +8,21 @@ open System.Collections.Concurrent
 open System.Collections.Immutable
 open System.IO
 open System.Linq
-open Microsoft.CodeAnalysis
+open System.Runtime.CompilerServices
+open System.Threading
+open System.Windows
 open FSharp.Compiler
 open FSharp.Compiler.CodeAnalysis
+open FSharp.Compiler.Text
+open Microsoft.CodeAnalysis
+open Microsoft.VisualStudio
 open Microsoft.VisualStudio.FSharp.Editor
-open System.Threading
 open Microsoft.VisualStudio.FSharp.Interactive.Session
-open System.Runtime.CompilerServices
-open CancellableTasks
 open Microsoft.VisualStudio.FSharp.Editor.Extensions
-open System.Windows
-open Microsoft.VisualStudio
-open FSharp.Compiler.Text
 open Microsoft.VisualStudio.TextManager.Interop
 
+open CancellableTasks
+
 #nowarn "57"
 
 [<AutoOpen>]
@@ -113,7 +114,7 @@ type private FSharpProjectOptionsMessage =
     | ClearSingleFileOptionsCache of DocumentId
 
 [<Sealed>]
-type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
+type private FSharpProjectOptionsReactor(checker: FSharpChecker, fileChangeWatcher: IFSharpFileChangeWatcher) =
     let cancellationTokenSource = new CancellationTokenSource()
 
     // Store command line options
@@ -133,6 +134,51 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
 
     let scriptUpdatedEvent = Event<FSharpProjectOptions>()
 
+    // The '-r:' set of each project is watched for the reference stamps the snapshot-reuse
+    // guard reads; invalidating the FCS build on a change is Roslyn's job (it swaps the
+    // MetadataReference and bumps Project.Version, which reaches tryComputeOptions).
+    let referenceChangeTracker =
+        new FSharpReferenceChangeTracker(fileChangeWatcher, ignore)
+
+    let referenceWatches = ConcurrentDictionary<ProjectId, HashSet<string>>()
+
+    let referencePaths (projectOptions: FSharpProjectOptions) =
+        let paths = HashSet<string>(StringComparer.OrdinalIgnoreCase)
+
+        for option in projectOptions.OtherOptions do
+            if option.StartsWith("-r:", StringComparison.Ordinal) then
+                paths.Add(option.Substring "-r:".Length) |> ignore
+
+        paths
+
+    let watchReferenceFiles (projectId: ProjectId) (projectOptions: FSharpProjectOptions) =
+        let paths = referencePaths projectOptions
+
+        match referenceWatches.TryGetValue projectId with
+        | true, previous ->
+            for path in previous do
+                if not (paths.Contains path) then
+                    referenceChangeTracker.StopWatchingReference path
+
+            for path in paths do
+                if not (previous.Contains path) then
+                    referenceChangeTracker.StartWatchingReference path
+        | _ ->
+            for path in paths do
+                referenceChangeTracker.StartWatchingReference path
+
+        if paths.Count > 0 then
+            referenceWatches[projectId] <- paths
+        else
+            referenceWatches.TryRemove projectId |> ignore
+
+    let clearReferenceWatches (projectId: ProjectId) =
+        match referenceWatches.TryRemove projectId with
+        | true, paths ->
+            for path in paths do
+                referenceChangeTracker.StopWatchingReference path
+        | _ -> ()
+
     let createPEReference (referencedProject: Project) (comp: Compilation) =
         let projectId = referencedProject.Id
 
@@ -408,7 +454,9 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
 
                             if not (Seq.isEmpty projectsToClearCache) then
                                 projectsToClearCache
-                                |> Seq.iter (fun pair -> cache.TryRemove pair.Key |> ignore)
+                                |> Seq.iter (fun pair ->
+                                    cache.TryRemove pair.Key |> ignore
+                                    clearReferenceWatches pair.Key)
 
                                 let options =
                                     projectsToClearCache
@@ -428,6 +476,7 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
                             let parsingOptions, _ = checker.GetParsingOptionsFromProjectOptions(projectOptions)
 
                             cache.[projectId] <- (project, parsingOptions, projectOptions)
+                            watchReferenceFiles projectId projectOptions
 
                             return ValueSome(parsingOptions, projectOptions)
 
@@ -514,6 +563,7 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
                     | _ -> ()
 
                     legacyProjectSites.TryRemove(projectId) |> ignore
+                    clearReferenceWatches projectId
                 | FSharpProjectOptionsMessage.ClearSingleFileOptionsCache(documentId) ->
                     match singleFileCache.TryRemove(documentId) with
                     | true, (_, _, _, projectOptions, subscription) ->
@@ -556,18 +606,24 @@ type private FSharpProjectOptionsReactor(checker: FSharpChecker) =
         singleFileCache.Clear()
         lastSuccessfulCompilations.Clear()
 
+        for projectId in referenceWatches.Keys |> Array.ofSeq do
+            clearReferenceWatches projectId
+
     member _.ScriptUpdated = scriptUpdatedEvent.Publish
 
+    member _.ReferenceStamps = referenceChangeTracker :> IReferenceStamps
+
     interface IDisposable with
         member _.Dispose() =
+            referenceChangeTracker.Dispose()
             cancellationTokenSource.Cancel()
             cancellationTokenSource.Dispose()
-            (agent :> IDisposable).Dispose()
+            agent.Dispose()
 
 /// Manages mappings of Roslyn workspace Projects/Documents to FCS.
-type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace) =
+type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Workspace, fileChangeWatcher: IFSharpFileChangeWatcher) =
 
-    let reactor = new FSharpProjectOptionsReactor(checker)
+    let reactor = new FSharpProjectOptionsReactor(checker, fileChangeWatcher)
 
     do
         // We need to listen to this event for lifecycle purposes.
@@ -636,4 +692,6 @@ type internal FSharpProjectOptionsManager(checker: FSharpChecker, workspace: Wor
 
     member _.ClearAllCaches() = reactor.ClearAllCaches()
 
+    member _.ReferenceStamps = reactor.ReferenceStamps
+
     member _.Checker = checker