Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

All tools from shell are very slow #138

Open
mmozeiko opened this issue Dec 8, 2014 · 44 comments

Comments

@mmozeiko
Copy link

commented Dec 8, 2014

Any tool I run in msys2 shell (ls, mv, ...) are very slow. Simply ls takes something like 3 seconds.

Running "strace ls" I see following output:

   15   39327 [main] ls 7460 DLL build:    2014-11-21 06:38
   16   39343 [main] ls 7460 dtable::extend: size 32, fds 0x1802FB488
  381   39724 [main] ls 7460 transport_layer_pipes::connect: Try to connect to named pipe: \\.\pipe\msys-07c9b4f1c823d951-lpc
   40   39764 [main] ls 7460 transport_layer_pipes::connect: Error opening the pipe (2)
   31   39795 [main] ls 7460 client_request::make_request: cygserver un-available
--- Process 7460, exception 00000000 at 000007FEFD22940D
1804450 1844245 [main] ls 7460 reg_key::build_reg: failed to create key ServicesForNFS in the registry
 2011 1846256 [ldap_init] ls 7460 cygthread::stub: thread 'ldap_init', id 0x2410, stack_ptr 0x2FCAC90
310497 2156753 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <...USERNAME...:*:1599881:1049089:....SOME_REGISTRY_KEY...:/home/...USERNAME...:/usr/bin/bash>
 6571 2163324 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Domain Users:S-1-5-21-1915207013-2615040368-3076929458-513:1049089:>
  768 2164092 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Performance Log Users:S-1-5-32-559:559:>
  402 2164494 [main] ls 7460 pwdgrp::fetch_account_from_windows: line: <Users:S-1-5-32-545:545:>

Could slowness be related to this? I am on non-admin user on this machine..
Can that be avoided?

@maoueh

This comment has been minimized.

Copy link

commented Dec 8, 2014

Are you using LDAP to connect to your computer? If yes, this is similarly related to this issue, but I'm not 100% sure.

To test that you are facing the same issue, you could create a /etc/passwd with your account information so they are not fetch from Windows directly. This should be fairly simple to test it out, however, I never did it myself yet.

Information in this thread or on cygwin's nsswitch.conf could prove useful. Maybe others have a fairly simple cookbook to follow.

@mmozeiko

This comment has been minimized.

Copy link
Author

commented Dec 8, 2014

I using Windows by logging in with domain user.

I created /etc/passwd and /etc/group files using mkgroup and mkpasswd utilities. They include all groups and usernames that were in "strace ls" output (and more, but that's irrelevant). But running "ls" is still slow. "strace ls" now show this:

   12   43607 [main] ls 14816 dtable::extend: size 32, fds 0x1802FB3E8
  236   43843 [main] ls 14816 transport_layer_pipes::connect: Try to connect to named pipe: \\.\pipe\msys-07c9b4f1c823d951-lpc
   39   43882 [main] ls 14816 transport_layer_pipes::connect: Error opening the pipe (2)
   21   43903 [main] ls 14816 client_request::make_request: cygserver un-available
--- Process 14816, exception 00000000 at 000007FEFD22940D
1459351 1503254 [main] ls 14816 reg_key::build_reg: failed to create key ServicesForNFS in the registry
966780 2470034 [main] ls 14816 cygheap_user::ontherange: what 2, pw 0x1802FB6B8
   41 2470075 [main] ls 14816 cygheap_user::ontherange: HOME is already in the environment /home/USERNAME
   96 2470171 [main] ls 14816 build_argv: argv[0] = 'ls'
10469 2480640 [main] ls 14816 build_argv: argc 1

As you can see it has no more "pwdgrp::fetch_account_from_windows" lines. But there are still errors:

  1. something about connecting to named pipe + cygserver
  2. something about failure to create registry key + ServicesForNFS
@jmichae3

This comment has been minimized.

Copy link

commented Dec 12, 2014

  • if it's running anything over a network like a network share, you can expect significant lag
  • if it's 32-bit code running on a 64-bit box, you can expect the virtualization (WOW64) to slow the program down. but hopefully not that much on a local box, especially if it's in the OS's internal disk cache in RAM. first access will be slow, subsequent accesses should be much faster due to disk cache, but the virtualization thing is another hurdle, however it doesn't slow things down too much I should think.
@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Jan 14, 2015

my case is slightly different; for me msys is slow on starting and on bash completion; but "ls" is fast

see http://sourceforge.net/p/msys2/discussion/general/thread/87925e6f/

i also tried to create passwd/group; "mkgroup -d" doesn't output group from domain

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Jan 21, 2015

i "solved" disabling db in nsswitch.conf

@Alexpux

This comment has been minimized.

Copy link
Member

commented Jan 21, 2015

ok.

@mmozeiko

This comment has been minimized.

Copy link
Author

commented Jan 21, 2015

This is great. Commenting out "db" from nsswitch.conf helps. Now all operations are fast for me.

@legends2k

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2015

Disabling db from both passwd and group fixed the issue but led to my username being unrecognised (bash prompt showed me as Unknown+User). However, the "fix" was to just disable db from group and leave passwd as it was, and this sped up MSYS2 utilities like diff, which, ls and also the startup of MSYS2 significantly.

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2015

unfortunately now some commands are slower than some time ago; for example: ls, git status/diff, configure (autoreconf), gitk

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2015

re-enabling db in nsswitch.conf makes these commands again fast; but it slows bash completion (TAB); the start of msys2_shell.bat is quite normal

@legends2k

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2015

@KrullKorg Bash completion being slow has got nothing to do with MSYS2 itself. See here for an explanation.

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2015

as i wrote bash completion isn't slow with db disabled in nsswitch.conf and slow if db is enabled; tomorrow i'll try again, to be sure

of course I'm talking about a pc under ms (samba) domain; at home it works perfectly

and yes i know that msys2 derived from cygwin

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2015

i just tried under latest cygwin (1.7.5) and it works perfectly

@elieux

This comment has been minimized.

Copy link
Member

commented Apr 21, 2015

Notable differences between MSYS2 and Cygwin:

  • the current MSYS2 runtime is based on Cygwin trunk from April 10th, not on Cygwin release
  • MSYS2 doesn't include cygserver and other Cygwin system integration tools

Can you look at differences between your Cygwin and MSYS2 installs? Namely /etc/passwd, /etc/group, /etc/nsswitch.conf, /etc/fstab, output of mount, a cygserver process running. MSYS2 should take advantage the same caching mechanisms as Cygwin, at least within a process tree.

I'm sorry I'm dumping all the debugging on you, but I don't currently have any domain to join here.

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2015

i don't have /etc/passwd, /etc/group and cygserver in both msys2 and cygwin

FSTAB - CYGWIN

# /etc/fstab
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.  For a description
#    see https://cygwin.com/cygwin-ug-net/using.html#mount-table

# This is default anyway:
none /cygdrive cygdrive binary,posix=0,user 0 0

FSTAB - MSYS2

# For a description of the file format, see the Users Guide
# http://cygwin.com/cygwin-ug-net/using.html#mount-table

# DO NOT REMOVE NEXT LINE. It remove cygdrive prefix from path
none / cygdrive binary,posix=0,noacl,user 0 0

NSSWITCH - CYGWIN

# /etc/nsswitch.conf
#
#    This file is read once by the first process in a Cygwin process tree.
#    To pick up changes, restart all Cygwin processes.  For a description
#    see https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-mapping-nsswitch
#
# Defaults:
# passwd:   files db
# group:    files db
# db_home:  cygwin desc
# db_shell: cygwin desc
# db_gecos: cygwin desc

NSSWITCH - MSYS2

# Begin /etc/nsswitch.conf

passwd: files db
group: files db

db_enum: cache builtin

db_home: cygwin desc
db_shell: cygwin desc
db_gecos: cygwin desc

# End /etc/nsswitch.conf

MOUNT - CYGWIN

C:/cygwin64/bin on /usr/bin type ntfs (binary,auto)
C:/cygwin64/lib on /usr/lib type ntfs (binary,auto)
C:/cygwin64 on / type ntfs (binary,auto)
C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)
I: on /cygdrive/i type smbfs (binary,posix=0,user,noumount,auto)
K: on /cygdrive/k type ntfs (binary,posix=0,user,noumount,auto)
O: on /cygdrive/o type smbfs (binary,posix=0,user,noumount,auto)
V: on /cygdrive/v type smbfs (binary,posix=0,user,noumount,auto)
X: on /cygdrive/x type smbfs (binary,posix=0,user,noumount,auto)
Z: on /cygdrive/z type sshvfs (binary,posix=0,user,noumount,auto)

MOUNT - MSYS2

C:/msys64 on / type ntfs (binary,noacl,auto)
C:/msys64/usr/bin on /bin type ntfs (binary,noacl,auto)
C: on /c type ntfs (binary,noacl,posix=0,user,noumount,auto)
I: on /i type smbfs (binary,noacl,posix=0,user,noumount,auto)
K: on /k type ntfs (binary,noacl,posix=0,user,noumount,auto)
O: on /o type smbfs (binary,noacl,posix=0,user,noumount,auto)
V: on /v type smbfs (binary,noacl,posix=0,user,noumount,auto)
X: on /x type smbfs (binary,noacl,posix=0,user,noumount,auto)
Z: on /z type sshvfs (binary,noacl,posix=0,user,noumount,auto)

PS: i just tried to remove db_enum from nsswitch.conf in msys2
PPS: don't worry about ask info; i'm happy to help msys2

@elieux

This comment has been minimized.

Copy link
Member

commented Apr 21, 2015

Can you try to mount cygdrive in Cygwin with noacl? I don't see why noacl would make things slower, but it's worth a shot.

I'm assuming you're trying ls somewhere inside /c/ (/cygdrive/c/) and not inside /home/ or somewhere like that.

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2015

i changed /etc/fstab in cygwin but i got the same behavior adding noacl: it's fast

and same behavior both in /home/ and in /c/

i also tried to remove noacl in msys2, but without improvements

@elieux

This comment has been minimized.

Copy link
Member

commented Apr 21, 2015

I'm out of ideas at this point. I don't know enough about Cygwin internals to suggest a good test for you. You can try searching the Cygwin mailing list, maybe some of the information there will apply to your situation.

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented Apr 22, 2015

i just tried on a relative new pc (virtual machine) with windows 10 tp (under the same domain) and it seems to work...

i installed msys2 from exe; then i upgraded with pacman -Syu

so i'll try to use a clean windows 8.1 vm... maybe the problem is the windows version... or my pc

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented May 5, 2015

i really think it's the fault of my pc; i tried also with windows 8 and it works

i found on strace executed on my pc this error (that isn't present in virtual machines)

2388701 3040772 [main] ls 7032 pwdgrp::fetch_account_from_windows: LookupAccountSid(S-1-5-21-726227932-2052316878-829958588-513), Win32 error 1789

@elieux

This comment has been minimized.

Copy link
Member

commented May 5, 2015

Interesting. KB976494 says the error is ERROR_TRUSTED_RELATIONSHIP_FAILURE "The trust relationship between this workstation and the primary domain failed.", so this could be a bug in Windows, or your domain connection is borked.

Can you try to leave and re-join the domain?

@KrullBorg

This comment has been minimized.

Copy link
Contributor

commented May 5, 2015

yes i arrived at the same conclusion, and it worked

@k-takata

This comment has been minimized.

Copy link
Contributor

commented May 13, 2015

I also faced the same issue with KrullKorg and legends2k's case.
I found that expansion of ~* was very slow.
I had to update /etc/passwd and /etc/group, then disable db in nsswitch.conf.
More detail: https://gist.github.com/k-takata/9b8d143f0f3fef5abdab

@elieux

This comment has been minimized.

Copy link
Member

commented May 14, 2015

Very nice summary, @k-takata.

@kjeremy

This comment has been minimized.

Copy link

commented Jun 15, 2015

@k-takata your workaround gist works great until I disconnect from the domain and hop onto another network... then it's slow again.

@virtuald

This comment has been minimized.

Copy link

commented Sep 26, 2016

I found that pacman installs are really slow until I applied @k-takata 's workaround. I wonder why it's so slow for it to do lookups on the domain?

@Globik

This comment has been minimized.

Copy link

commented Nov 23, 2016

And why no cygserver? Cannot run now postgresql. Error like function is not implemented.

@riban-bw

This comment has been minimized.

Copy link

commented Mar 26, 2017

Even after changing nsswitch.conf things are still a bit slow after dropping off an AD network. I found that disabling network and then re-enabling (using my laptop hardware button) made things responsive again.

@i2000s

This comment has been minimized.

Copy link

commented Jun 29, 2017

This may not be related to this issue, but Google led me here. My MSYS has been extremely slow (ran 12 hours not to finish a ./configure which could be finished in less than 1 min on Linux) using MinGW on Windows 10. All anti-virus software has been turned off, and I couldn't find the nsswitch.conf file on my installation directory. Any idea?

@legends2k

This comment has been minimized.

Copy link
Contributor

commented Jun 30, 2017

@i2000s Are you using MSYS and not MSYS 2? They're very different! I'm pretty sure that every installation of MSYS 2 I've seen had /etc/nsswitch.conf.

@i2000s

This comment has been minimized.

Copy link

commented Jun 30, 2017

Yeah, eventually I found the file and commented out '#db' for the:'group'. But that doesn't help. Everything is extremely slow.

@i2000s

This comment has been minimized.

Copy link

commented Jun 30, 2017

@legends2k I think I have found the problem: my Anti-Virus software has been dragging every single one of my commands. Disabling it doesn't change the game. Once I completely removed the anti-virus software, everything started working again. Just to let you know. Thanks.

@legends2k

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2017

@i2000s Glad it worked out. Thanks for letting us know!

@sskras

This comment has been minimized.

Copy link

commented Jul 17, 2017

@i2000s it would be useful for readers to know the name of the AV software you removed.

@i2000s

This comment has been minimized.

Copy link

commented Jul 17, 2017

I think the AV is called Avast. I have uninstalled it.

@Arnavion

This comment has been minimized.

Copy link

commented Jun 12, 2018

With a new msys2 installation today, I observed every process, the initial bash.exe spawned by mintty.exe as well as every process spawned by bash, taking ~1s to ~3s to start. strace bash 2>&1 | ts revealed all the time was spent before the first line out of output from strace, suggesting it was part of the process startup itself.

I tried the nsswitch.conf technique in the comments above but it did not change anything. (I believe Cygwin had got a patch to make the workaround unnecessary, and perhaps msys2 has had that same patch as well.)

So I traced bash.exe through procmon and observed it spending ~500ms on every startup enumerating ~8k files that match C:\Windows\System32\catroot\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\*.cat, each with a callstack containing appid.sys (the AppLocker driver). That led me to this page, and indeed my machine has AppLocker policies enabled (*). After code-signing all binaries under C:\msys64\usr\bin as that page suggests (cd C:\msys64\usr\bin\; gci -re *.exe | %{ &'C:\Program Files (x86)\Windows Kits\8.1\bin\x64\signtool.exe' sign /a /uw $_.FullName }), all msys2 process start nearly instantaneously.

(*) secpol.msc actually says there are no AppLocker policies, but Get-AppLockerPolicy -Effective clearly lists several.

@legends2k

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2018

Thanks for the info. Does this mean every time an executable under /usr/bin (, /mingw64/bin or /mingw32/bin) changes or new ones get added, the signtool has to be rerun?

@Arnavion

This comment has been minimized.

Copy link

commented Jun 13, 2018

Yes. Signing adds a signature to the executable, so if the executable changes it'll need to be signed again.

@RonaldinhoL

This comment has been minimized.

Copy link

commented Aug 31, 2018

@i2000s
i try many ways and it has no help, then i try to close my AV, evething works!
3ks!

@StarWolf3000

This comment has been minimized.

Copy link

commented Aug 31, 2018

@i2000s
i try many ways and it has no help, then i try to close my AV, evething works!
3ks!

@RonaldinhoL
Yes, your AV scans most of the utilities every time, if they're not signed and in an internal database with checksums, to make sure they don't harm your system. You should be able to exclude your MSYS2 installation from scanning. AV software and similar security applications can be a nightmare for developers.

@jwiegley

This comment has been minimized.

Copy link

commented Sep 26, 2018

I've been encountering this exact same slowdown, but the nsswitch.conf changes have not helped. What else may be tried?

@jwiegley

This comment has been minimized.

Copy link

commented Sep 27, 2018

I was able to reproduce this in a Windows 7 virtual machine. After doing an upgrade with pacman -Syu followed by restarting the shell and running pacman -Su, solved the problem for me.

@jwiegley

This comment has been minimized.

Copy link

commented Sep 27, 2018

On my friend's Windows 10 machine, updating did not help.

@jmichae3

This comment has been minimized.

Copy link

commented Sep 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.