Access denied error in latest nightly on Windows #3601
Comments
|
Update - can confirm the issue isn't present in stale 0.11.4, if that helps narrow down when it was introduced. |
|
Thanks a lot for reporting! The very low-level error is probably a result of #3582, which is intended as with help of the logs we should be able to see where it is coming from. If you could find such errors in the logs and post them here, we could identify what's happening and tune the errors to be more descriptive. The root cause here is probably that reads and writes happen in parallel, which isn't allowed on Windows and GitButler has to get better at coordinating these tasks. |
Byron
added
the
feedback requested
|
I’m encountering a similar issue. It seems to have surfaced in the latest 2-3 nightly builds. Based on the logs, this issue first occurred with app version 0.5.428. Latest logs (0.5.429) : |
Byron
removed
the
feedback requested
|
Thanks for the logs, everyone! It seems like the error chain is very short, and that it feels like GitButler is stepping onto its own feet. I will see to land some patches that might improve the situation in the interim, as truly fixing them requires more time. |
…p#3601) By increasing the window size for collecting filesystem events, knowing that each event is processed in parallel, we might be lucky and that already reduces the likelyhood of clashes. It's an experiment though.
People can't copy it without the terminal-escape code, making it harder to read.
…p#3601) By increasing the window size for collecting filesystem events, knowing that each event is processed in parallel, we might be lucky and that already reduces the likelyhood of clashes. It's an experiment though. On Unix, run with: `LOG_LEVEL=debug pnpm tauri dev --features adapt-to-windows`
People can't copy it without the terminal-escape code, making it harder to read.
|
A PR is now available which will make log-excerpts more readable, while trying to reduce the parallelism of the application. This will make it appear more laggy, but might also help reduce the probability of access-denied errors. Fingers crossed. |
This probably means the raciness is not due concurrency introduced by filesystem events, which leads me to think that having great 'disk-IO-hygiene` should improve things.'
|
In conclusion to the failed experiment I can probably say that the issue is indeed caused by disk accesses within serial, unthreaded code. In the experiment, the filesystem monitor was set to only kick in after collecting events for 2 seconds, giving any write operation a long time to finish before reads (and possibly writes, due to lack of IO hygiene) would come in. Since that was ineffective, it must be serial code which is causing this, very reliably so as well. As an interesting note: button-spamming can make it work after 20-or-so tries. |
|
I managed to create a local Windows build (see #3505 for the tales of this adventure), and (unfortunately) could add a project without problems. 
I tried a few bits of GItButler functionality, but wouldn't run into any access-denied issues. Could it be that there is some supervisor-process that meddles with files of applications? Admittedly, I disabled all of these as well as I could, and it's impossible to say if my Parallels filesystem truly replicates typical Windows filesystems semantics. I also couldn't reproduce the issue with GitButler from the release channel. Hence, it might be worth trying again on a truly real Windows machine. CC @krlvi |
|
It took me a while to realize that this issue is most likely related to the introduction of tempfile here: 6665bc9 . There seem to be two writes in short succession, sometimes they succeed, but sometimes one of them fails. Both writes seem to originate in GitButler itself. @daniilS will check if other processes, like Virus Scanners, are interfering, but even if that is the case there was no issue with this previously before tempfiles were introduced. As an immediate fix, one could selectively disable tempfiles on Windows as it seems to be a problem with consecutive writes, maybe due to some laziness in the filesystem? |
|
Think I've identified the root cause: tempfile's implementation of To use tempfiles on Windows, either the implementation of Also worth checking tempfile's implementations of |
|
Thanks so much for this investigation - very helpful, and much appreciated. It seems like the semantics of Looking at the code in Git it becomes clear that it certainly is more advanced on Windows and But this also means, that in the interim, on Windows, tempfiles shouldn't be used anymore until this is fixed upstream. |
|
Re: the mingw code, it seems to account for some cases where it doesn't have permissions to modify the file. If it does, but still gets an access denied error, it assumes that another process is using the file and just retries after a bit. In our case, I think it's reasonable to assume that gb is the only process using Agreed that an upstream fix would be ideal - but until then, would it work to call |
I absolutely agree with that - GitButler can avoid racing against itself.
|
…rapp#3601) Until `perist()` is truly atomic also on Windows, and modelled [after Git](https://github.com/git/git/blob/master/compat/mingw.c#L2159-L2209), sans the retries, it seems to work to just perform ordinary writes. Before going there though, we try to call `sync_all()` on Windows to truly finish the operation. On Unix, a rename is already supposed to be atomic.
…rapp#3601) Until `perist()` is truly atomic also on Windows, and modelled [after Git](https://github.com/git/git/blob/master/compat/mingw.c#L2159-L2209), sans the retries, it seems to work to just perform ordinary writes. Before going there though, we try to call `sync_all()` on Windows to truly finish the operation. On Unix, a rename is already supposed to be atomic.
…rapp#3601) Until `perist()` is truly atomic also on Windows, and modelled [after Git](https://github.com/git/git/blob/master/compat/mingw.c#L2159-L2209), sans the retries, it seems to work to just perform ordinary writes. Before going there though, we try to call `sync_all()` on Windows to truly finish the operation. On Unix, a rename is already supposed to be atomic.
…rapp#3601) Until `perist()` is truly atomic also on Windows, and modelled [after Git](https://github.com/git/git/blob/master/compat/mingw.c#L2159-L2209), sans the retries, it seems to work to just perform ordinary writes. Before going there though, we try to call `sync_all()` on Windows to truly finish the operation. On Unix, a rename is already supposed to be atomic.
The sync-all didn't do anything, and one would still run into locked files.
The sync-all didn't do anything, and one would still run into locked files.
|
Finally I was able to create the test-build which contains some It will run on x86-64 Windows systems. gitbutler-tauri-2024-05-25-01.exe.zip How this binary is different?The build uses a locally modified Resources
Next stepsThe idea is that if this does indeed work, I will produce another binary without the More NotesWhen opening source-code in the RustRover IDE on Windows, it popped up a message saying that Windows Defender was enabled and the project path should be excluded from it to avoid severe performance degradation. It then offers to run an admin-script to make these exclusions for you. It would probably be too much for GitButler to offer doing so as well, but it might be something to try out doing by hand - do the access-denied problems go away once Defender doesn't interfere? |
While this is probably good for users to do for performance reasons, it is also something that GitButler shouldn't do - we want to solve locking bugs, not hide them! |
I meant it as a test to see if Defender is a common reason for these errors, and not to say that should be the (only) way GitButler can work correctly on Windows. |
|
Just for reference, I still receive a lot of access denied errors on windows 11 with the latest gitbutler v0.12.2 Here's a recent one:
and the log: |
|
Thanks everyone for your help with testing this, and for bearing with us. I have compiled another executable based on 0.12.2 (c6b22a7 specifically) which should in theory have less trouble with access-denied errors. It's an updated version of the executable from the earlier comment and if it proves to be better/more stable/more usable than the current stable or nightly I think we can go into further testing and finally, try to upstream the change I made to the Windows version of the Finally, after all that pre-amble, here is the new executable. |
|
Hey @Byron, Thanks for looking into this. I played around with that exe a bit. Unfortunately I'm not sure performance improved much (or at all?), and I still hit access denied errors. initial openFirst, I was simply trying to switch back to the gitbutler/integration branch and got
I can DM someone the logs without redacted paths, but every instance of was the exact same path, and the .py file was actually deleted. updating workspaceI moved on from that by completely deleting and re-adding the project. I was able to commit a few changes and apply a few different virtual branches successfully. Unfortunately, after merging a PR/deleting a branch in github and then clicking the red "Update" workspace button in gitbutler, I encountered another access denied error:
Clicking the red Update button a second time seemed to work. applying branchThen, I tried applying an old virtual branch with a lot more files and changes in it, and got another error: It did completely delete a couple files, as well as leave gitbutler in a weird state where it thinks that branch is applied but with 52 changes and deleted files because it wasn't actually able to update the filesystem. Once gitbutler is at this point I can't unapply that branch or consistently revert changes. Sometimes I can use the new snapshotting feature to get me back somewhere usable but more often than not when I play around with it too much I end up having to delete the project and re-adding it. Let me know if there's anything else I can test or if you need any more information. |
|
Thanks for the detailed account of what happens here, it's much appreciated and very useful! It seems there are two different kinds of errors here - one is about failure to remove a directory during checkout, and the other one is a bit unclear as it only says 'access denied' happened in And that is the error that interests me most (the one from |
Byron
removed
the
feedback requested
|
after updating to 0.12.21 on win11 I started getting: e.g. when clicking on the refresh button to fetch from remote. Dev console log entries below:
makes this unusable now for me |
This should help in cases the binary can't be found nonetheless, for yet to be clarified reason.
This should help in cases the binary can't be found nonetheless, for yet to be clarified reason.
|
Thanks for reporting! I don't think anything change in that department, but I have added more logging to make clear which executable it would try to use. While this trickles through to nightly, do you think you could share logs with us? They should contain more details, maybe even the path to If that would just be Thanks a lot. |
|
Funnily enough, the nightly works as expected again. No problems running the git command. I guess I'll wait for the next stable release and see what happens then. |
|
0.12.22 works again. So from my perspective it's resolved for now. I wonder if there's a difference in configuration/environment settings that get stored depending on how the application is installed. The last one (that didn't work) I did not install via the in-app "new release" popup because the windows elevation prompt just didn't show up and the installation wouldn't continue (that I assume is my local windows problem not of the Gitbutler app although I haven't noticed this behaviour with any other application). |
|
I am glad to hear it works!
To me that seems unlikely as both use the same data, but maybe there is a difference in what Windows Defender or other AntiVirus programs do depending on its location or installation type.
If that's reproducible to you, I think it's worth opening a new issue about it - auto-update should always work or fail loudly (not silently). |
|
The git command started failing again with 0.12.25: For reference: the start of the last run with 0.12.24 (which didn't have any problems): Is there a way to debug the environment in which the application is run in windows? The PATH looks ok
Possible cause... I tried uninstall-installing the application and during the installation I got a popup notification from CrowdStrike about quarantining a file as it was found malicious. The CS event log doesn't really tell me which file it was. And I don't have access to the Falcon console for further investigation :( |
|
Thanks so much for reporting! Indeed, as the log suggests it would try to launch Git using I would think that if GitButler would be opened from a terminal, it would inherit its Given the presence of CrowdStrike I'd think there might be outside interference here, especially since GitButler didn't change between the versions, or at least not enough to do more than trying to find I am looking forward to reading what you find out, thanks in advance for your help. |
|
Indeed we found the issue now. The |
|
Thanks a lot, I am glad we figured this out! |
Since updating to 0.5.427 (sadly not sure what my previously installed version was exactly, I only know that I installed it at some point last week), I keep getting "Access is denied. (os error 5)" errors on Windows, usually with "Failed to fetch" or "Failed load remote branches". This happens when first opening Gitbutler, or trying to switch or add a new project. Sometimes closing and reopening Gitbutler will load the project and just display the toast, other times it won't load the project at all. When a project does load, actions like creating or deleting a virtual branch sometimes have no effect, and work fine at other times. I've tried running Gitbutler with admin privileges, which didn't solve the problem.
The text was updated successfully, but these errors were encountered: