Some trouble with the sidebar again

[deathaxe]

Summary

I realized three issues with the sidebar of ST 3132 since yesterday which may all be related and maybe caused by the same reason.

1. Some files existing on the filesystem are not displayed in the project tree.
2. A folder existing on the filesystem is displayed but can't be expanded / shows as empty even if it contains files. (The folder icon seemed a little different than usual. It contained a kind of infinite icon.)
3. Sublime Text suddenly crashes and closes when clicking on a file (Keywords.sublime-completions).

Issue 1: missing file

I made some screenshots about this issue, yesterday.

Expected behavior

This is the state after restarting sublime. The file is displayed in the sidebar again.

Actual behavior

As you can see the open file Keywords.sublime-completions is not part of the project tree on the left even though it exists on disk.

Steps to reproduce

Unfortunately I found no way to reproduce with 100% chance but all of them happened after rebasing the underlying repository. I sometimes do so with GitExtensions or with GitSavvy's git: stage current file and fixup command. In all cases the file Keywords.sublime-completions was temporarily removed from disk and spawned again by git rebase/cherry-pick.

Environment

• Operating system and version:
  • Windows 10 x64
  • Mac OS ...
  • Linux ...
• Sublime Text:
  • Build 3132 x64

[FichteFoll]

Possibly related: #310

[keith-hall]

Number 2 could be related to #1548

[deathaxe]

1. None of the entries is highlighted (incorrectly).
   Edit: Or do you mean the row of Parameters.sublime-snippets ?
2. None of the folders or files is a junction or symbolic link.

[keith-hall]

Maybe check your (icon) theme to see under what circumstances it uses the icon that is a folder with an infinite symbol then.

[deathaxe]

I use Adaptive Theme with A File Icon package installed. A File Icon seems not to provide folder icons thus I had a look into Adaptive Theme folder.

There are several accountable folder icons but I am quite sure it was symlink.png which I saw. Very strange as the folders definitely were real folders but no symlinks or junctions. Maybe ST accidentally recognized them as those? I had some trouble with symlink handling in GitGutter, too, as python's _getfinalpathname() which is an OS function was a bit tricky to handle.

[wbond]

@deathaxe The infinite symbol is a chain link, which means the file showed up as a file that has been scanned before, which usually implies a symlink. You can't expand symlinks since it could easily lead to an infinite list of files to popular in Goto Anything and the indexer.

[deathaxe]

But it is/was no symlink at all. That's the strange thing. Or does git create temporary symlinks while rebase/cherry pick?

[wbond]

It isn't strictly for symlinks, it will do it for any file that appears to be a duplicate (based on inode). It may be that git is moving things around in such a way that it was (properly) detected as a duplicate. When the other copy/reference was removed, you were stuck with a "duplicate" copy.

That is my hunch, anyway.

[deathaxe]

I realized those issues when calling "git: stage current file and fixup" command provided by GitSavvy. I didn't investigate how this command works in detail, but I already got error messages like "I can't cherry pick" from it in the past. Sidebar indeed changes much during that operation.

[wbond]

@deathaxe If you see this again, can you check in your console for ReadDirectoryChangesW failed with ERROR_ACCESS_DENIED?

[deathaxe]

Is this a new massage added in recent dev builts? Can't remember to have seen something like that.

[wbond]

No, it's been around a long time. It is just the only way I've been able to find issues with the sidebar similar to what you describe here is to do a git operation that either adds or removes thousands of files from one of the folders in the sidebar. Each time the sidebar gets out-of-whack, I see that error in my console.

[deathaxe]

I recently struggled a lot with #1750 when rebasing package resources especially tmPreferences. The dozens of error dialogs may prevent the sidebar issue for me at the moment. But I'll try with another repo.

[deathaxe]

Ah, just realize the repo I had this issue is the one which triggers #1750 since 3133. Therefore I think I'll fail reproducing this issue at the moment.

[deathaxe]

The documentation of ReadDirectoryChangesW warns about buffer overflows on heavy directory modifications. I considered something like that as the reason for the three issues mentioned in my first post as tree only got out of sync on git operations in the most recent builts.

Maybe it is a good idea to add some idle time, when realizing such an error and restart watching the directory and updating the tree some seconds later. The documentation suggests to manually enumerate the directories to read changes, if ReadDirectoryChangesW fails, but this makes sense only if the file structure doesn't change anymore.

The sidebar relies only on information got from ReadDirectoryChangesW at the moment? I ask because I have one project with a folder which is located on an external drive. If I open the project with the drive not being connected, the folder shows as empty. After connecting the drive the folder is never updated by ST until I reopen the project. I would expect ST to try to "mount" the folder if I expand it.

[wbond]

@deathaxe Yes, we already handle the various documented error conditions of ReadDirectoryChangesW. From the codebase I can see that the ERROR_ACCESS_DENIED typically happens (to an existing watch) when the directory being watched is deleted. I am going to be working on handling this relatively odd, undocumented failure mode for situations where some process apparently overwhelms the Windows API with notifications. This should hopefully be in the next build.

[deathaxe]

Sounds good for us users, but like a kind of nightmare for you. 😄

I can at least confirm directory deletions were part of the git operations which lead to unsync sidebar. So you are on track from my point of view.

[wbond]

This should (hopefully) be improved/fixed in 3137. We found that ReadDirectoryChangesW sometimes drops events under high CPU load, so we now detect heavy FS activity and set a timeout to rescan the folder for changes that aren't picked up.

[wbond]

@deathaxe Have you seen this at all with 3139?

[deathaxe]

Not yet. Did some checkouts, external rebase and some fixups using GitSavvy on several months old commits to see what happens on such heavy operations. Sidebar stops updating on some point and goes after a little idle time. After all it's up to date again.

Seems ok now.

[wbond]

Ok, that seems to indicate the issue was that ReadDirectoryChangesW was dropping some events and never sending them to us. Thanks for the confirmation!