Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python.d.plugin: use separate process for initial module checking #5552

Merged
merged 17 commits into from Mar 7, 2019

Conversation

Projects
None yet
4 participants
@ilyam8
Copy link
Member

ilyam8 commented Mar 5, 2019

Summary

This PR adds (major) changes only to python.d.plugin file.

Fixes: #5525

pyhton.d.plugin imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram.


Memory utilization comparing before/after the PR (one job example module, py3.7.2):

21.1 => 8.8 MiB

screenshot_20190305_111837

Component Name

collectors/python.d.plugin

Additional Information

This PR adds separate process for initial module checking.

Logic:

  • main process spawns checker process
  • checker process loads every module, loads module config, creates jobs and runs job.check() for every job, if check success it adds the job to the list.
  • checker process returns list of modules and jobs.
  • main process loads only active modules, etc.

ilyam8 added some commits Mar 5, 2019

@netdata netdata deleted a comment from codacy-bot Mar 5, 2019

@netdata netdata deleted a comment from codacy-bot Mar 5, 2019

@netdata netdata deleted a comment from codacy-bot Mar 5, 2019

@ilyam8

This comment has been minimized.

Copy link
Member Author

ilyam8 commented Mar 5, 2019

ok, i tested it with on Manjaro (py 3.7.2) and on Centos6(py 2.6.6), and it appears to be working correctly

@ilyam8 ilyam8 changed the title [wip] python.d.plugin: use separate process for initial module checking python.d.plugin: use separate process for initial module checking Mar 5, 2019

@ilyam8

This comment has been minimized.

Copy link
Member Author

ilyam8 commented Mar 5, 2019

Hey @Ferroin if you could find a time and test the PR a bit it would be very nice

@netdata netdata deleted a comment from codacy-bot Mar 6, 2019

@netdata netdata deleted a comment from codacy-bot Mar 6, 2019

@ilyam8

This comment has been minimized.

Copy link
Member Author

ilyam8 commented Mar 6, 2019

ok, i don't think we need to run a job if job.check() returns False in main. It can be error prone, instead we can treat all jobs in main as autodetection jobs, so if a job's check(), for some reason, succeed in child process and failed in main we will retry check again and again

@netdata netdata deleted a comment from codacy-bot Mar 6, 2019

@ilyam8

This comment has been minimized.

Copy link
Member Author

ilyam8 commented Mar 6, 2019

@ktsaou if you are ok with the changes this is ready.

@codacy-bot

This comment has been minimized.

Copy link

codacy-bot commented Mar 6, 2019

Codacy Here is an overview of what got changed by this pull request:

Clones added
============
- collectors/python.d.plugin/powerdns/powerdns.chart.py  1
         

See the complete overview on Codacy

@netdata netdata deleted a comment from codacy-bot Mar 6, 2019

@ilyam8

This comment has been minimized.

Copy link
Member Author

ilyam8 commented Mar 6, 2019

@cakrit please approve

@cakrit

cakrit approved these changes Mar 7, 2019

@ilyam8 ilyam8 merged commit 2175673 into netdata:master Mar 7, 2019

6 checks passed

Codacy/PR Quality Review Up to standards. A positive pull request.
Details
LGTM analysis: C/C++ No code changes detected
Details
LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python No new or fixed alerts
Details
license/cla Contributor License Agreement is signed.
Details
netlify/netdata/deploy-preview Deploy preview ready!
Details
@ilyam8

This comment has been minimized.

Copy link
Member Author

ilyam8 commented Mar 7, 2019

ok, here we go, i am on a high alert

@ilyam8 ilyam8 deleted the ilyam8:pythond_plugin_refactor branch Mar 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.