Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Optimized find all references and reduced memory usage in VS #8339
The aim of this PR is to optimize find all references and reduce overall memory usage in VS, or potentially other editors depending on their needs.
To accomplish this, we need to stop storing full symbol information for every single file in every project in incremental builder. This will break find all references and rename refactor though, so we have to do a bit of work.
The solution I propose here is as follows:
Both the storage of symbol keys and semantic classification in incremental builder will be disabled by default.
This design isn't perfect; I would rather not store
I will be porting over a lot of work that was done in a prototype and this PR will be the real thing.
The prototype showed significant memory reduction in VS, even without calling find all references, due to lack of storing full symbol information in-memory. Find all references's performance also significantly improved for large solutions.
This is ready for the most part.
I added a new public lexing API and marked it as experimental; we really really need a better API for lexing. My hope is this will improve over time. This was really needed for find all references for syntactic classification so I didn't have to use the other one. I only wanted to get classification for a small span of text; though it isn't perfect and will not be accurate for tokens that span multiple lines.
dsyme left a comment •
The code looks great
My question is really about the memory-mapped file. You say "it uses memory but not the process's memory". I don't understand this. My impression of memory mapped files is that they are "mapped into the process's address space" and my mental model is that if a mmap is 1GB big then 1GB of process address space is used. The actual contents of the file may or may not be in physical memory but that's true for anything from the process address space - the contents are only brought into physical memory as needed but the memory mapping does consume virtual address space, which is 4GB limited for VS.
So if that's correct then this MemoryMappedFile burns VS devenv.exe address space? I thought if you wanted to get the data out of the process address space then you'd have to use an actual file on disk, like a temporary file??
Now it could be that the above statement is somehow wrong for ViewStream over mmap files. If so could you include a link to definitive documentation about that? Or a sample that shows that you can create, say, 10x1GB mmap streams (using the combination of calls we are using here to create them) in a 32 bit process, and have them all live and accessible?
This is what we are doing, with the exception of using it for IPC. I don't think it is possible to even share the information by IPC because we give the MMF name a "null" value. Though, as found by @baronfel, Mono, unfortunately, does not allow a "null" name in their MMF impl, but Desktop and Core do. So, we might need to special case that here.
This is the API we use:
Regarding memory use, MMF uses virtual memory which could come from RAM or pages; it should not be using memory from the process's (devenv.exe) private memory but will in the address space. While this will use memory, it lowers the memory pressure for the actual process.
I think this isn't wrong.
open System open System.IO.MemoryMappedFiles let create1GB () = let size = 1024 * 1024 * 1024 // 1gb let mmf = MemoryMappedFile.CreateNew(null, int64 size) let view = mmf.CreateViewStream() (mmf, view) [<EntryPoint>] let main argv = let tenMMF = Array.init 10 (fun _ -> create1GB()) Console.ReadLine() |> ignore Console.WriteLine(tenMMF) 0
This will explode because of the address space for MMF regarding "views" because it is over 2GB in address space.