-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Masterless gitfs performance #30100
Comments
Pretty interesting stuff, @armooo . Thanks for the research here. I'm going to ping @terminalmage here to see what feedback he might have. |
Is there a workaround for this? We use a lot of formulas in git for local development in vagrant and it gets worse each time we add a formula. For context, we currently have 10 formulas and a no-op highstate does 390 gitfs fetches. |
I made (with my lame bash skills) a workaround script, that synces repos locally on demand. It sucks, but it's better than nothing. https://gist.github.com/tomasfejfar/e3607569e6a38424e7cf2d9d56187815 |
I've been looking into this. We store the fileclient object in a dictionary that should live for the entirety of a Salt run, to prevent re-initialization (and thus a fileserver update) from being run more than once during a masterless salt-call. However, it appears that (for some reason) this is no longer the case. |
I've opened #34048 as a potential fix for this issue. I know that it works to resolve the issue of multiple fileserver updates per masterless run, but there may be a more elegant way to resolve this. In the course of my investigation I found that the method of using the |
Fix was merged into the 2015.8 release branch, closing this issue. It will take a few days for this change to be merged into the 2016.3 release branch and the develop branch, I'll update this comment thread once the fix has propagated to those branches for those who would like to test. |
The fix has propagated to the 2016.3 release branch, and should be in develop in the next few days. |
Actually, the fix is now in the develop branch as well, so anyone intrepid enough to test from git can do so from the 2015.8, 2016.3, or develop branches, and the fix will be there. |
I updated to latest version and the bug is gone. Good job. Cheers! |
When using masterless salt with the gitfs fileserver I observed that performance was scaling with the number of sls files * number of git repos configured in my minion config. With 12 state files and 13 pillar files and 9 git repos a noop highstate run was taking ~3 minutes to complete.
I was able to track this down to the time updating the git repos multiple times in a single highstate run. It seems that a salt master periodically updates, it seems by default once a minute. On the other hand when running masterless the construction of a
FSChan
causes the git repos to be updated.I was able to test this by changing
get_file_client
to cache theFSChan
instance, in a hacked up way complete ignoring the value ofopts
. After this a noop highstate run was finishing in ~30 seconds, most of which is my fault with one slow virtualenv.managed state, and the debug logs show that the repos were only updated once.I don't have the context to propose the correct solution but would be happy to help with a patch with some guidance.
The text was updated successfully, but these errors were encountered: