Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow startup opening a large workspace over the network #70045

Closed
DaviMedrade opened this issue Mar 8, 2019 · 43 comments
Closed

Slow startup opening a large workspace over the network #70045

DaviMedrade opened this issue Mar 8, 2019 · 43 comments
Assignees
Labels
candidate Issue identified as probable candidate for fixing in the next release file-explorer Explorer widget issues freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues important Issue identified as high-priority verified Verification succeeded

Comments

@DaviMedrade
Copy link

Issue Type: Performance Issue

My workspace directory sits on a server in the network, which I access using a mapped network drive. It has about 470,000 files (~36GB) pertaining to about 500 websites. Each website is contained within a folder in the workspace top level, e.g. “T:\vm_sites\example.com", "T:\vm_sites\otherwebsite.org", and so on.

I open the parent folder (“T:\vm_sites”) instead of just the folder of the website I'll be working on because then I don't have to copy workspace settings (including FTP settings for upload) to each of the hundreds of website folders.

Earlier today I started VS Code as usual, by right-clicking on the folder containing all the websites on the network drive and selecting “Open with Code”. VS Code started as usual and, as I was about to start working, I saw a blue mark over the gear icon on the bottom left. Upon hovering it, a tooltip appeared: “Restart to update”, or something to that effect. I clicked on it, VS Code closed, updated to version 1.32.1, and restarted.

Once VS Code started, the file browser pane stayed empty, with a blue bar scrolling horizontally above it (which I believe is a “loading” indicator). Only after a few minutes did the file pane get populated. From then on, VS Code was back to normal.

Just to be sure, once the files pane was populated, I opened a .cshtml file, closed VS Code (leaving the file open), and opened VS Code again. The exact same thing happened (took minutes to load the file pane), but syntax highlighting and the effects of the “indent-rainbow” extension didn't appear in the editor pane of the opened file until after the files pane was populated.

The issue still happens even if VS Code is started from the command line with “--disable-extensions”.

It seems to me that VS Code is now recursively (and, up to a point, synchronously) scanning the whole workspace directory before populating the files pane and loading/running extensions, which I believe is a change from its behavior up until now. With so many files, over the network, it takes a while. And, while it happens, the files pane can't be used to open files.

Now, this wouldn't be as much of an issue if VS Code could open just a subfolder of the workspace and still detect the workspace settings. Then I could just open the folder of the website I'll be working on instead of the parent folder with all websites, without having to make each and every website a separate workspace—which would bring on the hassle of keeping the settings of hundreds of workspaces in sync.

VS Code version: Code 1.32.1 (05f146c, 2019-03-08T00:51:49.278Z)
OS version: Windows_NT x64 10.0.17763

System Info
Item Value
CPUs Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz (12 x 3696)
GPU Status 2d_canvas: enabled
checker_imaging: disabled_off
flash_3d: enabled
flash_stage3d: enabled
flash_stage3d_baseline: enabled
gpu_compositing: enabled
multiple_raster_threads: enabled_on
native_gpu_memory_buffers: disabled_software
rasterization: enabled
surface_synchronization: enabled_on
video_decode: enabled
webgl: enabled
webgl2: enabled
Memory (System) 15.94GB (9.28GB free)
Process Argv
Screen Reader no
VM 0%
Process Info
CPU %	Mem MB	   PID	Process
    0	   119	 17004	code main
    0	   236	  5416	   window (● _Layout.cshtml - vm_sites - Visual Studio Code)
    0	    11	  1696	     watcherService 
    0	    11	 16064	       console-window-host (Windows internal process)
    0	    13	 12536	     electron-crash-reporter
    0	    60	 16820	     extensionHost
    0	    79	 13884	   gpu-process
    0	    67	 16672	   window (Issue Reporter)
    0	    65	 19136	   shared-process
Workspace Info
|  Window (● _Layout.cshtml - vm_sites - Visual Studio Code)
|    Folder (vm_sites): more than 64723 files
|      File types: jpg(15098) js(14499) dll(12581) css(5861) png(5081)
|                  config(1580) map(1513) pdb(1214) xml(791) mno(791)
|      Conf files: csproj(3) sln(2) package.json(2) settings.json(1)
|                  tsconfig.json(1) tslint.json(1);
Extensions (8)
Extension Author (truncated) Version
pascal ale 8.0.0
pascal-formatter ale 2.1.0
haml kar 1.1.0
ftp-sync luk 0.3.9
vscode-apache mrm 1.2.0
indent-rainbow ode 7.2.4
ruby reb 0.22.3
sass-indented rob 1.5.1
@vscodebot vscodebot bot added the new release label Mar 8, 2019
@donaggio
Copy link

donaggio commented Mar 8, 2019

The very same thing as described by the OP is happening to me under a similar scenario: Ubuntu Linux using a workspace with some remote directories mounted locally using gvfs. Except that VS Code becomes unresponsive and I'm forced to kill its process to close it.
If I start a clean session and I create a new workspace adding back the same directories from the old one, I can work just fine, but as soon as I exit VS Code and I relaunch it, it happens again.

VS Code version 1.32.1 (.deb version 1.32.0-1551898681 from http://packages.microsoft.com/repos/vscode stable/main)

OS: Ubuntu Linux 18.10 X86_64

@citrus-thunder
Copy link

I'm experiencing the same issue here:

  • VS Code Version 1.32.1
  • Ubuntu 18.10 X86_64

When attempting to load a mapped drive mounted through sshfs the explorer window never even populates with folders -- I've been working with this large filespace in VS Code for two years now and this is the first version that showed any slowness at all loading it up, let alone being seemingly unable to load it entirely.

To test I even mounted a personal webserver's /var/www/ directory to the same drive and attempted to open it with VS Code and while the folders did eventually appear in the explorer this time, it took several minutes, and this is a server with about five small personal websites on it.

I've ended up reinstalling version 1.31 and I'm able to continue working.

@bpasero
Copy link
Member

bpasero commented Mar 11, 2019

According to #70143 this is not only from the explorer but also opening normal files.

People hitting this, are you all on the same OS?

@isidorn
Copy link
Contributor

isidorn commented Mar 11, 2019

@bpasero lookint at the comments I see two people on Ubuntu and one person on Windows_NT

@bpasero
Copy link
Member

bpasero commented Mar 11, 2019

Can people try to reproduce with our insiders builds: https://code.visualstudio.com/insiders/

@DaviMedrade
Copy link
Author

Still happens (on Win10 x64) with insiders build 1.33.0 (d77d8d5).

@psilonmarko
Copy link

psilonmarko commented Mar 11, 2019

Running debian 9. Purged 1.31 and installed 1.33 (insiders). No extensions.
Opened folder from mount via VPN. Populating file list (left pane explorer) takes about 2-3 minutes - just to list files in the root folder. Additional minutes to open any subfolder.
Clicking on a file and waiting to show in editor... too long. And I had to kill it because it stopped responding.

@bpasero
Copy link
Member

bpasero commented Mar 11, 2019

Can people just check if node.js 10.2.0 shows the same issue when trying to e.g. fs.readdir a remote folder?

@DaviMedrade
Copy link
Author

DaviMedrade commented Mar 11, 2019

I used this:

{
    let start = process.hrtime();
    fs.readdir("T:\\vm_sites\\", function(e, f) {
        console.log("Callback: ", process.hrtime(start), "Files: ", f.length);
    });
    process.hrtime(start);
}

The results were always around 40 µs for fs.readdir to return and around 36 ms for the callback to be called, with a list of 493 entries (the folder in question contains 487 directories and 6 files at that level).

Node readdir

Neither {withFileTypes: true} nor using fs.readdirSync changes the times significantly (± a couple ms).

Notice that T: is a network mapped drive.

@bpasero
Copy link
Member

bpasero commented Mar 11, 2019

@davidsantos-br thanks for testing, I forgot to ask people to try out our exploration builds that come with a much newer Electron/node.js version to see if it could help:

@psilonmarko
Copy link

@bpasero tried code-exploration, same problem(s)

@mightypenguin
Copy link

mightypenguin commented Mar 11, 2019

Mac OS host to SSHFS on Linux. Same issue.
What's weird is that this doesn't happen if I remove and create a new workspace (right after creating the new workspace).
It only has the constantly moving blue bar and fails to open files if I reopen a workspace to a large SSHFS filesystem.
It won't even load locally stored files after I close and relaunch the workspace.

Correction: it does eventually open files it just takes a long time.

@bpasero
Copy link
Member

bpasero commented Mar 12, 2019

Maybe about time that we get some profiling information to see where most time is spend.
Can you follow this wiki and post the results back here to understand more about why performance is so bad for you: https://github.com/Microsoft/vscode/wiki/Performance-Issues

Thanks!

@martbean
Copy link

Just to add I am also experiencing this issue on Windows 10 Home connecting via OpenVPN to a remote server running Ubuntu.

I've downgraded to VS Code 1.31.1 for now and that's working fine.

@psilonmarko
Copy link

I tried one more thing today in code-insiders 1.33 - opening mounted folder on another server over cisco vpnc connection and it took some time to index all items (file explorer), but not nearly as long as over openvpn. Clicking on any file or subfolder opened it immediately.
Looks like there is a problem with openvpn

@m33ts4k0z
Copy link

Hello,

Trying to reproduce the issue and upload the text file requested, produced empty files. I'm also suffering from this issue that appears to be VPN related since accessing the same folder from a computer on the same LAN works without delays. However I don't have 500 projects as the OP so I'm certain the root of the problem is still there.

Using either SoftEther VPN or OpenVPN makes things really REALLY slow without an obvious reason.

@bpasero bpasero added the info-needed Issue requires more information from poster label Mar 14, 2019
@bpasero
Copy link
Member

bpasero commented Mar 14, 2019

People that see this issue, are you opening the network folder directly (e.g. via File > Open Folder), or is it part of a multi-root workspace file you open? Does it reproduce if you use the UNC on localhost scheme, e.g. when opening a path such as \\localhost\c$\my_dir?

@bpasero bpasero mentioned this issue Mar 14, 2019
@sherryyshi
Copy link
Member

sherryyshi commented Mar 14, 2019

This also happens with large repos using GVFS . In case this is helpful - when the virtualization service is disabled (i.e. when GVFS is unmounted), the issue does not repro. However, when the virtualization service is active, it always repros, even on subsequent launches of VS Code.

EDIT: This is on VSCode 1.32 and 1.33.0-insider, and Windows 10

@GabeDeBacker
Copy link
Contributor

I was just debugging VSCode and expanded folder "X" in the explorer pane, and it appears that VSCode is enumerating files well outside of folder "X" (meaning not even within it).

For example:
I expanded the folder
"e:\repos\osNew\src\minkernel"
and vscode is enumerating
"e:\repos\osNew\src\base\PerfTest\MemRe\AppLifecycle"

@bpasero
Copy link
Member

bpasero commented Mar 14, 2019

i have an idea and will ask people to try a new build once its ready

@GabeDeBacker
Copy link
Contributor

Looking through the commits and debugging VSCode, I am wondering if these strict null checks changes somehow changed the decision of the file service on whether to enumerate all directories or just the one the explorer view is asking for:

a6c498b#diff-a63388ce79427f16181b4686359ff685

Debugging through VSCode certainly shows that when the explorer view asks to expand a directory, the entire workspace contents are enumerated.

@bpasero
Copy link
Member

bpasero commented Mar 14, 2019

Can people please try with this build if it still reproduces:

This build removes the changes from #67269 that for me put a lot of pressure right on startup due to extensive file scanning.

@sherryyshi
Copy link
Member

@bpasero the build seems to have fixed the issue for me

@psilonmarko
Copy link

@bpasero this latest build (code-insider-1552583659, debian9) works for me as well

@bpasero
Copy link
Member

bpasero commented Mar 14, 2019

Awesome thanks for the quick turn around!

@m33ts4k0z
Copy link

@bpasero This build greatly improved things. Opening the folder in a workspace or "open folder" makes eveyrthing open almost instantly even over VPN. I have a folder with over 1000 subfolders and this build needed about 20 seconds to open it which was great improvement since before I had to wait about 5 minutes.

Generally, the experience is still not the same as the local one but its a great improvement in comparison to other builds.

@bpasero bpasero assigned kieferrm and unassigned isidorn Mar 14, 2019
@bpasero bpasero added important Issue identified as high-priority candidate Issue identified as probable candidate for fixing in the next release and removed info-needed Issue requires more information from poster labels Mar 14, 2019
@octref octref added this to the February 2019 Recovery 3 milestone Mar 14, 2019
roblourens added a commit that referenced this issue Mar 14, 2019
roblourens added a commit that referenced this issue Mar 14, 2019
@roblourens
Copy link
Member

We verified it by the following steps:

  • Open task manager
  • Open \\localhost\c$ on windows using the latest Insiders that has the bug
  • See that disk/CPU usage spikes a ton. Try to kill vscode, it won't die

Repeat the same steps with 1.32.3, see that disk/cpu doesn't spike and it's responsive very soon after starting.

@roblourens roblourens added the verified Verification succeeded label Mar 15, 2019
@donaggio
Copy link

Confirming that the latest insider's build from @bpasero resolved the issue under Linux, too.

@martbean
Copy link

Looks good to me on Windows 10 as well. Thanks for sorting this!

@mightypenguin
Copy link

1.32.3 appears to have fixed it for me!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
candidate Issue identified as probable candidate for fixing in the next release file-explorer Explorer widget issues freeze-slow-crash-leak VS Code crashing, performance, freeze and memory leak issues important Issue identified as high-priority verified Verification succeeded
Projects
None yet
Development

No branches or pull requests