File content is fetched multiple times

[felixfbecker]

When adding a list of fetched content to RemoteFileSystem.getTextDocumentContent() and logging when content is fetched twice, you can see that apparently every single file is fetched at least twice (without having done any request besides initialize).

My suspicion is that this is because on initialize ensureFilesForWorkspaceSymbols() is called, whose result promise is memoized, but other places in the code (ensureFilesForHoverAndDefinition()? ProjectConfiguration.init()?) fetch them directly again.

Solution

Eventually, these file contents will be added to the InMemoryFileSystem. This should be the source of truth for file contents and whether they are loaded or not. One difficulty I see here is the "dummy" placeholder that is added for some files (ideally this should be removed).

I want InMemoryFileSystem to be able to hold the workspace structure (as returned from workspace/files), but only optionally hold a content for each file.
Additional benefit: RemoteFileSystem.getWorkspaceFiles() is currently memoized, instead should only be used to hydrate the structure of InMemoryFileSystem once (that is, *.ts, tsconfig.json, package.json), contents in the background and then all parts of the codebase should just interact with that (and invalidate / modify it as needed).

TODO

• Extend InMemoryFileSystem to differentiate whether a file does not exist or is just not loaded
• Find out which functions call the remote file system to fetch files directly
• Adapt them
• Remove memoization in RemoteFileSystem and hydrate the structure once on initialization
• Make InMemoryFileSystem available to TypeScriptService and the build handler, remove direct usage of RemoteFileSystem (for other purposes than hydration)

Difficulties

• ProjectManager's, ProjectConfiguration's and InMemoryFileSystem's properties, methods and parameters are undocumented and I have a hard time understanding the semantics of each one
• InMemoryFileSystem.entries is just any. It's hard for me to understand how the tree is typed.

wdyt @beyang @alexsaveliev

[alexsaveliev]

I think it happens because ensureFiles may be called independently by

• ensureFilesForHoverAndDefinition (hover request)
• ensureFilesForWorkspaceSymbol (workspace symbol request)
  there is a fetched set which supposed to deal with duplicates but it being updated only in get text document content callback thus allowing two (or more) requests run in parallel until one of them will finish and update fetched set.
  See
  https://sourcegraph.com/github.com/sourcegraph/javascript-typescript-langserver/-/blob/src/project-manager.ts#L360:1-368:1