memory leak & high CPU usage on windows 10 #4256
Comments
|
I'm very much inclined to think this is an environmental or git issue than GE-specific. We've had a number of issues of slowness on W10 but most were related to either W10 components (like speech recognition #3292) or some antivirus (#2907) or bad configs (#3968). |
|
@RussKie
If I skip step 2, gitextenstions works fine. no any 'git.exe' will be stuck and leaks memory.
Besides, I tried to all issues you mentioned. |
0xffff00
changed the title
|
@gerhardo you mean the command log taking longest time? |
|
Thank you for the follow up and the info. This may be useful when attemp to
repro.
Can you share your npm install script?
I presume the npm packages all in a public domain....
…On 21/12/2017 1:37 PM, "0xffff00" ***@***.***> wrote:
@RussKie <https://github.com/russkie>
I found this problem may be related to huge file amount of local git repo.
Now I have a git repo of a front-end web project using NodeJS.
It contains <100 files and 'node_modules/*' is written in .gitignore
But after running npm install, the file amount will increased to 19000+
files.
Here is my actions:
1. clone a git repository to local by git bash.
2. run command in git bash: npm install.
when done you can see a directory node_modules with about 20000 files
is created and git status is clean since they are all git-igonored.
3. open this local repo by gitextenstions.
4. open Windows Task Manager, watch for several minutes.
If I skip step 2, gitextenstions works fine. no any 'git.exe' will be
stuck and leaks memory.
However, if no above steps skipped, this problem will be reproduced.
Besides, I tried to all issues you mentioned.
I have disabled speech reocgnition ,disabled firewall and windows
defender, run 'net use * /delete' to clear remote share folder links. All
these do not work.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4256 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEMyXrrgwWeFpRTyqL6Ga81-eLBtxQklks5tCcRugaJpZM4RH1Pf>
.
|
|
@RussKie
then, GitExt Browse it. |
|
Cool thanks.
Are there any tools you run on the background which are watching for
changes?
I can't promise anything soon though with the end of the year, xmas and the
new release demanding attention.
…On 21/12/2017 3:34 PM, "0xffff00" ***@***.***> wrote:
@RussKie <https://github.com/russkie>
Yes, all npm packages are public. Here is a demo git repo.
just run:
cd gitextensions-issues-4256-reprod
npm install
gitextensions-issues-4256-reprod.zip
<https://github.com/gitextensions/gitextensions/files/1577921/gitextensions-issues-4256-reprod.zip>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4256 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEMyXpaAJ3WtvYF6M8H4KTKsB3-GT4RNks5tCd_vgaJpZM4RH1Pf>
.
|
|
@RussKie xmas |
|
As there were no other long running commands, this should be a git command that is started but never finished. Check the command line argument to the running git processes. https://superuser.com/questions/415360/how-do-i-find-out-command-line-arguments-of-a-running-program |
|
@gerhardol thanks for help |
|
Turn of update of commit button to get rid of this: related to the issues I mentioned. This case with these long running times will not solve the issue though. |
|
@gerhardol thanks for your suggestion to avoid this issue. git ls-files -o -i --exclude-standard
git ls-files -oBut these are quite fast, may be useful to fetch count for commit button: #list some ignored files
git ls-files -i --exclude-standard
#list untracked files
git ls-files -o --exclude-standard |
|
@0xffff00 could you please share what settings you have set/unset under
Settings > Git Extensions tab (similar to Gerhard's screenshot above).
…On 23 December 2017 at 15:27, 0xffff00 ***@***.***> wrote:
@gerhardol <https://github.com/gerhardol> thanks for your suggestion to
avoid this issue.
I tried more commands.
There will be stuck for long running time:
git ls-files -o -i --exclude-standard
git ls-files -o
But these are quite fast, may be useful to fetch count for commit button:
#list some ignored files
git ls-files -i --exclude-standard #list untracked files
git ls-files -o --exclude-standard
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4256 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEMyXsBdkD65AAq1tvPOKR4--vqoRiEAks5tDIE5gaJpZM4RH1Pf>
.
|
|
I assume the settings were changed after the problem was reported? GE uses the list of ignored files to ignore changes that the file system reports. Without the filtering "git status" would run more often that normally would give a lot more load. Edit: I do not find a better way for the normal case where ignored files are 'static'. The include/exclude rules are complex to reimplement in the tool. So deactivating Commit count is probably the sane solution. |
Reported in gitextensions#4069 Based on changes in gitextensions#4209 GE can in some situations use a lot of CPU and disk load, as it runs "git status --untracked-files" at changes in the filesystem If a status command is not done before next update is detaected, the next start immediately after. "git status" can therefore run continuously. (This is more visible with certain viruses like Symantec and McAfee.) This PR limits the checks in two ways: * First run is started after 1000 ms + timer (average 250ms) instead of 500+timer, to collect more changes (probably not essential) * Next command is started minimum 5000ms after the previous command. So normal one shot changes are detected about the same as now, but continuous changes are not updating so frequently. A few other changes like that the "next run" time could be overwritten by a time in the future and that retrieved "git status" data were not used if more file system changes were detected. If "git status" requires more than 5s to run, the behavior is not improved. The counter in the Commit button should not be used then. Configurable time could improve (like longest of 5s and twice the time to run git status), configurable time is not going to be used. There are situations where the "ignored files detection" is insufficient. See gitextensions#4256 for a discussion.
|
I still propose no change, but summarize some more findings. The current "ignore files" detection is insufficient in this situation. But I see no better solution than to deactivate commit count. A consequence of the current detection of ignored files is that GE may be slower if starting up in a clean repo (without ignored), then building. When ignored is refreshed after 10 min, the number of git status decreases too. Parsing the git ignore rules is probably not an option, there are too many specific rules. Instead of using "ls-files -o -i --exclude-standard" to detect all ignored files, "git status --ignored" could be used to detect directories, so new ignored files can be ignored and memory usage decreases. The parsing of a file the OS reports is changed will have to be redone, but the list will be smaller as directories are listed. But the "startup" problem remains, as well as a part of git exclusion rules in git ignore (this last part can probably be ignored). I also have to mention "git check-ignore -v", to check each and every file (not realistic). @0xffff00 Can you try git status --ignored? |
Reported in gitextensions#4069 Based on changes in gitextensions#4209 GE can in some situations use a lot of CPU and disk load, as it runs "git status --untracked-files" at changes in the filesystem If a status command is not done before next update is detaected, the next start immediately after. "git status" can therefore run continuously. (This is more visible with certain viruses like Symantec and McAfee.) This PR limits the checks in two ways: * First run is started after 1000 ms + timer (average 250ms) instead of 500+timer, to collect more changes (probably not essential) * Next command is started minimum 5000ms after the previous command. So normal one shot changes are detected about the same as now, but continuous changes are not updating so frequently. A few other changes like that the "next run" time could be overwritten by a time in the future and that retrieved "git status" data were not used if more file system changes were detected. If "git status" requires more than 5s to run, the behavior is not improved. The counter in the Commit button should not be used then. Configurable time could improve (like longest of 5s and twice the time to run git status), configurable time is not going to be used. There are situations where the "ignored files detection" is insufficient. See gitextensions#4256 for a discussion.
|
@gerhardol |
Reported in gitextensions#4069 GE can in some situations use a lot of CPU and disk load, as it runs "git status --untracked-files" at changes in the filesystem If a status command is not done before next update is detected, the next start immediately after. "git status" can therefore run continuously. (This is more visible with certain viruses like Symantec and McAfee.) This PR limits the checks in two ways: * First run is started after 1000ms + timer (fires every 500ms so adds on average 250ms) instead of 500ms +timer, to collect more changes (probably not essential) * Next command is started minimum 3000ms after the previous command. If 3*time to run the status update is longer than 3000ms, the interval time is dynamically increased. So normal one shot changes are detected about the same as now, but continuous changes are not updating so frequently, "git status" should not run continuously. A few other changes like that the "next run" time could be overwritten by a time in the future and that retrieved "git status" data were not used if more file system changes were detected. If "git status" requires more than 5s to run, the behavior is not improved. The counter in the Commit button should not be used then. Configurable time could improve (like longest of 5s and twice the time to run git status), configurable time is not going to be used. There are situations where the "ignored files detection" is insufficient. See gitextensions#4256 for a discussion.
Issue gitextensions#4256 Listing ignored files with a separate "ls-files -o -i --exclude-standard" seem to never exit in some situations, see gitextensions#4256 This reuses the git-status call with --ignored as an option to get the ignored files. (The option is only added when ls-files were called previously.) This call did not seem to hang with many ignored files and the command runtime does not increase much. The solution is not optional though. It still lists all ignored files in memory. The time to find ignored files is faster, but a lot of memory is needed. gitextensions#4256 has some discussions. Also, processing all output to GitStatusItem first before creating the HashSet will require some more memory (temporary) than required.
|
Please test that changes I propose is working better |
Issue gitextensions#4256 Listing ignored files with a separate "ls-files -o -i --exclude-standard" seem to never exit in some situations, see gitextensions#4256 This reuses the git-status call with --ignored as an option to get the ignored files. (The option is only added when ls-files were called previously.) This call did not seem to hang with many ignored files and the command runtime does not increase much. The solution is not optional though. It still lists all ignored files in memory. The time to find ignored files is faster, but a lot of memory is needed. gitextensions#4256 has some discussions. Also, processing all output to GitStatusItem first before creating the HashSet will require some more memory (temporary) than required.
Reported in gitextensions#4069 GE can in some situations use a lot of CPU and disk load, as it runs "git status --untracked-files" at changes in the filesystem If a status command is not done before next update is detected, the next start immediately after. "git status" can therefore run continuously. (This is more visible with certain viruses like Symantec and McAfee.) This PR limits the checks in two ways: * First run is started after 1000ms + timer (fires every 500ms so adds on average 250ms) instead of 500ms +timer, to collect more changes (probably not essential) * Next command is started minimum 3000ms after the previous command. If 3*time to run the status update is longer than 3000ms, the interval time is dynamically increased. So normal one shot changes are detected about the same as now, but continuous changes are not updating so frequently, "git status" should not run continuously. A few other changes like that the "next run" time could be overwritten by a time in the future and that retrieved "git status" data were not used if more file system changes were detected. If "git status" requires more than 5s to run, the behavior is not improved. The counter in the Commit button should not be used then. Configurable time could improve (like longest of 5s and twice the time to run git status), configurable time is not going to be used. There are situations where the "ignored files detection" is insufficient. See gitextensions#4256 for a discussion.
* Limit how often GitStatus polls for changes Reported in #4069 GE can in some situations use a lot of CPU and disk load, as it runs "git status --untracked-files" at changes in the filesystem If a status command is not done before next update is detected, the next start immediately after. "git status" can therefore run continuously. (This is more visible with certain viruses like Symantec and McAfee.) This PR limits the checks in two ways: * First run is started after 1000ms + timer (fires every 500ms so adds on average 250ms) instead of 500ms +timer, to collect more changes (probably not essential) * Next command is started minimum 3000ms after the previous command. If 3*time to run the status update is longer than 3000ms, the interval time is dynamically increased. So normal one shot changes are detected about the same as now, but continuous changes are not updating so frequently, "git status" should not run continuously. A few other changes like that the "next run" time could be overwritten by a time in the future and that retrieved "git status" data were not used if more file system changes were detected. If "git status" requires more than 5s to run, the behavior is not improved. The counter in the Commit button should not be used then. Configurable time could improve (like longest of 5s and twice the time to run git status), configurable time is not going to be used. There are situations where the "ignored files detection" is insufficient. See #4256 for a discussion.
Issue gitextensions#4256 Listing ignored files with a separate "ls-files -o -i --exclude-standard" seem to never exit in some situations, see gitextensions#4256 This reuses the git-status call with --ignored as an option to get the ignored files. (The option is only added when ls-files were called previously.) This call did not seem to hang with many ignored files and the command runtime does not increase much. The solution is not optional though. It still lists all ignored files in memory. The time to find ignored files is faster, but a lot of memory is needed. gitextensions#4256 has some discussions. Also, processing all output to GitStatusItem first before creating the HashSet will require some more memory (temporary) than required.
Issue gitextensions#4256 Listing ignored files with a separate "ls-files -o -i --exclude-standard" seem to never exit in some situations, see gitextensions#4256 This reuses the git-status call with --ignored as an option to get the ignored files. (The option is only added when ls-files were called previously.) This call did not seem to hang with many ignored files and the command runtime does not increase much. The solution is not optional though. It still lists all ignored files in memory. The time to find ignored files is faster, but a lot of memory is needed. gitextensions#4256 has some discussions. Also, processing all output to GitStatusItem first before creating the HashSet will require some more memory (temporary) than required.
Motivated by issue reported in gitextensions#4256. Relates to gitextensions#4753.
Motivated by issue reported in gitextensions#4256. Relates to gitextensions#4753.
fixes gitextensions#5439 fixes gitextensions#4256 partly based on gitextensions#5429 The handling was improved in gitextensions#4327, gitextensions#4328, fixing the regression Those PRs limited the max load for git-status Changes: React faster on changing directory (1.0s->0.2s) Some tweaks to the checks, do not check if update is required if an update is already scheduled Do not read ignored files due to two problems: * Reading ignored takes longer times. Since this was done the first time, the first update was delayed. (Reading ignored could be done later, but this will still delay the next update). * Saving all seen ignored files require substantial memory, see gitextensions#4256. The list could be limited (see the issue), but that will increase the test time for ignored files As ignored files will now always trigger git-status may cause some more runs. However, the ignored file list was never complete, the list was better over time, so GE must be able to handle this load anyway.
fixes gitextensions#5439 fixes gitextensions#4256 partly based on gitextensions#5429 The handling was improved in gitextensions#4327, gitextensions#4328, fixing the regression Those PRs limited the max load for git-status Changes: React faster on changing directory (1.0s->0.2s) Some tweaks to the checks, do not check if update is required if an update is already scheduled Do not read ignored files due to two problems: * Reading ignored can takes significant time. Since this was done the first time, the first update was delayed. (Reading ignored could be done later, but this will still delay the next update). * Saving all seen ignored files require substantial memory, see gitextensions#4256. The list could be limited (see the issue), but that will increase the test time for ignored files As ignored files will now always trigger git-status may cause some more runs. However, the ignored file list was never complete, the list was better over time, so GE must be able to handle this load anyway.
fixes gitextensions#5439 fixes gitextensions#4256 partly based on gitextensions#5429 The handling was improved in gitextensions#4327, gitextensions#4328, fixing the regression Those PRs limited the max load for git-status Changes: React faster on changing directory (1.0s->0.2s) Some tweaks to the checks, do not check if update is required if an update is already scheduled Do not read ignored files due to two problems: * Reading ignored can takes significant time. Since this was done the first time, the first update was delayed. (Reading ignored could be done later, but this will still delay the next update). * Saving all seen ignored files require substantial memory, see gitextensions#4256. The list could be limited (see the issue), but that will increase the test time for ignored files As ignored files will now always trigger git-status may cause some more runs. However, the ignored file list was never complete, the list was better over time, so GE must be able to handle this load anyway.
fixes gitextensions#5439 fixes gitextensions#4256 partly based on gitextensions#5429 The handling was improved in gitextensions#4327, gitextensions#4328, fixing the regression Those PRs limited the max load for git-status Changes: React faster on changing directory (1.0s->0.2s) Some tweaks to the checks, do not check if update is required if an update is already scheduled Do not read ignored files due to two problems: * Reading ignored can takes significant time. Since this was done the first time, the first update was delayed. (Reading ignored could be done later, but this will still delay the next update). * Saving all seen ignored files require substantial memory, see gitextensions#4256. The list could be limited (see the issue), but that will increase the test time for ignored files As ignored files will now always trigger git-status may cause some more runs. However, the ignored file list was never complete, the list was better over time, so GE must be able to handle this load anyway.
fixes gitextensions#5439 fixes gitextensions#4256 partly based on gitextensions#5429 The handling was improved in gitextensions#4327, gitextensions#4328, fixing the regression Those PRs limited the max load for git-status Changes: React faster on changing directory (1.0s->0.2s) Some tweaks to the checks, do not check if update is required if an update is already scheduled Do not read ignored files due to two problems: * Reading ignored can takes significant time. Since this was done the first time, the first update was delayed. (Reading ignored could be done later, but this will still delay the next update). * Saving all seen ignored files require substantial memory, see gitextensions#4256. The list could be limited (see the issue), but that will increase the test time for ignored files As ignored files will now always trigger git-status may cause some more runs. However, the ignored file list was never complete, the list was better over time, so GE must be able to handle this load anyway.
fixes gitextensions#5439 fixes gitextensions#4256 partly based on gitextensions#5429 The handling was improved in gitextensions#4327, gitextensions#4328, fixing the regression Those PRs limited the max load for git-status Changes: React faster on changing directory (1.0s->0.2s) Some tweaks to the checks, disable file system monitoring temporary if an update is already scheduled Do not read ignored files due to two problems: * Reading ignored can takes significant time. Since this was done the first time, the first update was delayed. (Reading ignored could be done later, but this will still delay the next update). * Saving all seen ignored files require substantial memory, see gitextensions#4256. The list could be limited (see the issue), but that will increase the test time for ignored files As ignored files will now always trigger git-status may cause some more runs. However, the ignored file list was never complete, the list was better over time, so GE must be able to handle this load anyway.
|
I have similar issue on: i feel like issues happens when i have this issue https://stackoverflow.com/questions/26046698/git-refname-origin-master-is-ambiguous/26047558 . and I have seen it first time ever at same time I start getting above. not sure if that is enough to create new issue. so I see status |
|
Please open a new issue with a repro. Though if it's something user-originated via the command line (like the referenced SO issue) I'm not sure what we can do... |
|
This is updated in master, in some situations where git-status was slow several parallel instances could start running. You would see several running git processes. |
|
yes, i did weird things with cli before trying to fix these with gitext. will try to get repro. i have seen enough gits running slowing down my PC with 16 physical theads. |
Do you want to request a feature or report a bug?
bug
What is the current behavior?
A memory leak & high CPU usage always happens to me. it causes nearly 100% memory usage & 100% single-thread CPU usage.
I have closed console window, it happens also.
I have tryed an older version - 2.49, it happens also.
I am sure that git bash has no problem. If I only use git bash, it works fine and memory used will never exceed 2M.
Here is a comparation:
20171221 Updated
A error dialog will be displayed in v2.49(may be fixed already ,i don't know) :
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.
Environment you encounter the issue:
Did this work in previous version of GitExtensions (which)?
no
The text was updated successfully, but these errors were encountered: