-
-
Notifications
You must be signed in to change notification settings - Fork 233
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pmdaprometheus performance improvement ideas #434
Comments
|
It turns out we need this badly. When targeting a prometheus server, we exceed the 5s timeout at something between 40-60 urls during the initial startup. That means a pmda restart, a new initial startup, which means no progress. |
OK, will add PM_ERR_PMDANOTREADY / PM_ERR_PMDAREADY notifications whenever there are more than a few new sources to process (not just during startup, though that would be when it's most needed obviously). |
I expect to have PM_ERR_PMDANOTREADY / PM_ERR_PMDAREADY working in pmdaprometheus later today |
GH performancecopilot#434 pmdaprometheus now starts in the "notready" state, meaning it can take as long as it likes to initialize and startup before PMCD will send it any requests - i.e. very large initial configs (1000's of URLs) can be set up without timing out on startup. Once startup has completed and all the source class instances have been created and initialized, pmdaprometheus now tells PMCD that it is ready using the python binding for the new pmdaSendError() function. Exercised by QA 1287 with 500 URLs and log checks to confirm the initialization is now done in the pmda notready state. This feature requires the changes to the Install script, and the previous commits for pmdaSendError() and the python bindings. modified: src/pmdas/prometheus/Install modified: src/pmdas/prometheus/pmdaprometheus.1 modified: src/pmdas/prometheus/pmdaprometheus.python
@fche, have been experimenting with pmdaprometheus - your patch (as fpasted) is substantially slower that current master branch, but I think it's heading in the right direction because we need to limit the number of threads, sockets and file descriptors. With O(1000) URLs, we run out of FDs, etc., and 1000 threads is just plain silly, so limiting that to min(100,len(clusters)) is the way to go for sure. I'm still trying to figure out why it's so much slower .. might be the queuing or something - will continue and come up with a refined patch. Similar queuing is needed to stagger the processing of large config changes too, as already discussed. Another simple experiment I did was to replace the requests module with pycurl. The patch is almost a drop in replacement, and approximately doubles the performance for the fetch (http get), i.e. halves the fetch latency. The parse time is of course about the same (no changes made there).
Above, each URL end-point has two metrics, and is to a remote host running apache over gig-E. The WALLTIME is milliseconds to fetch the URLs (averaged over 10 fetches to all metrics). FETCHTIME is from the control.fetch_time[0] metric (averaged over 10 repeats), PARSETIME is the parse_time metric and TOTALTIME is the sum of fetch+parse.
|
pycurl also has pycurl.CurlMulti(), which can launch a swag of async http gets, and then harvest them in a select loop. This could fit in pretty well with our existing code - i.e. launch all the fetches needed for a refresh in parallel, then process/parse them sequentially as the responses come back. I think it's worth pursuing. |
Can you please elaborate? |
Some quick performance tests today indicate we have some limitations well short of the PMID-limited 1000 prometheus sources. Some ideas:
parse_lines
them all in the main threadPM_ERR_PMDANOTREADY
facility should be used if necessaryThe text was updated successfully, but these errors were encountered: