Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Perfy McPerferton #58400

Open
wants to merge 28 commits into
base: devel
from

Conversation

Projects
None yet
1 participant
@sivel
Copy link
Member

commented Jun 26, 2019

SUMMARY

This PR aims to further advance performance improvements with inventories consisting of thousands of hosts.

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

many

ADDITIONAL INFORMATION

Most perf testing was done using a set of 35k hosts (localhost aliases), 1-10 ping tasks, and 25 forks.

cProfile provided much of the inspiration of fixes. Many are micro performance improvements that you will only see with a huge number of hosts.

A few things this PR does:

  1. Many small micro performance improvements on comparisons
  2. Use sets where possible
  3. Define regex patterns globally, instead of re-defining on method call
  4. Don't convert types unnecessarily, such as stringifying for dict keys
  5. Remove SharedPluginLoaderObj to avoid repeated instantiation
  6. Deal with hostnames where possible, to avoid expensive attribute lookups on host objects
  7. Probably the most important here: Don't recalculate get_hosts so frequently, cache the results and manipulate the cache for the lifetime of the Strategy plugin
  8. Along with the item above, pass this cache to get_vars and used by _get_magic_variables so that we aren't recalculating get_hosts repeatedly in that code path when invoked from the Strategy (Can be extended further later if needed)
@sivel

This comment has been minimized.

Copy link
Member Author

commented Jun 26, 2019

bot_skip

@sivel

This comment has been minimized.

Copy link
Member Author

commented Jun 26, 2019

Waiting on the results of another perf run to validate some changes.

@sivel

This comment has been minimized.

Copy link
Member Author

commented Jun 26, 2019

Perf comparison between this PR and devel:

devel:
real    533m9.880s
user    1275m2.398s
sys     1216m14.454s

PR:
real    338m5.147s
user    1264m11.409s
sys     1423m4.057s

This is 35k hosts, 10 ping tasks, 25 forks.

With a single ping task:

devel:
real    46m55.181s
user    122m55.709s
sys     72m4.257s

PR:
real    26m50.820s
user    129m13.340s
sys     81m49.809s

@sivel sivel requested review from bcoca, jimi-c and nitzmahone Jul 1, 2019

sivel added some commits Jun 25, 2019

@sivel sivel force-pushed the sivel:perfy-mc-perferton branch from 7caffb6 to 93de5bf Jul 1, 2019

@sivel

This comment has been minimized.

Copy link
Member Author

commented Jul 1, 2019

Leaving this in WIP right now, but it's ready for review.

@sivel sivel requested a review from mkrizek Jul 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.