Anomalies collector

[andrewm4894]

Summary

ML based anomaly detection python collector built on top of PyOD.

Component Name

/collectors/python.d.plugin/anomalies

Test Plan

Additional Information

Additional details included in the README.md for the collector.

[ilyam8]

@andrewm4894 i made it draft to make it clear it is wip

[andrewm4894]

@andrewm4894 i made it draft to make it clear it is wip

Yep sorry, was a little trigger happy with the pr.

[thiagoftsm]

Hey @andrewm4894 , please, take a look at here, it looks like something simple to fix.

[thiagoftsm]

@andrewm4894 when I ran the command

# pip3 install requests netdata-pandas numba==0.50.1 scikit-learn pyod

I received the following warning:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

netdata-pandas 0.0.23 requires requests==2.23.0, but you'll have requests 2.24.0 which is incompatible.

should we add an information related to this to our documentation?

[andrewm4894]

@andrewm4894 when I ran the command

# pip3 install requests netdata-pandas numba==0.50.1 scikit-learn pyod

I received the following warning:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

netdata-pandas 0.0.23 requires requests==2.23.0, but you'll have requests 2.24.0 which is incompatible.

should we add an information related to this to our documentation?

Thanks for that, I'll pin version ranges to the readme and/or add more flexible version ranges to netdata-pandas requirements too and that should cover this.

[vlvkobal]

@andrewm4894 please do squash and force push. The PR is clattered with commits. I even can't open the page with commits.

[ilyam8]

@andrewm4894 i assume you have tested the collector and it works w/o problems/errors.

Code, makefiles ,etc looks good to me.

[andrewm4894]

@andrewm4894 i assume you have tested the collector and it works w/o problems/errors.

Code, makefiles ,etc looks good to me.

Yep, have been running it for last few weeks on some of my nodes.

[vlvkobal]

2020-12-02 15:37:53: python.d WARNING: plugin[main] : [anomalies] error on loading source : SyntaxError('invalid syntax', ('/usr/libexec/netdata/python.d/anomalies.chart.py', 82, 139, "        self.charts_in_scope = list(filter(self.charts_regex.match, [c for c in requests.get(f'{self.protocol}://{self.host}/api/v1/charts').json()['charts'].keys()]))\n")), skipping it
2020-12-02 15:37:53: python.d INFO: plugin[main] : no jobs to run

[andrewm4894]

2020-12-02 15:37:53: python.d WARNING: plugin[main] : [anomalies] error on loading source : SyntaxError('invalid syntax', ('/usr/libexec/netdata/python.d/anomalies.chart.py', 82, 139, "        self.charts_in_scope = list(filter(self.charts_regex.match, [c for c in requests.get(f'{self.protocol}://{self.host}/api/v1/charts').json()['charts'].keys()]))\n")), skipping it
2020-12-02 15:37:53: python.d INFO: plugin[main] : no jobs to run

Hmm - is that the f-string breaking a lower version of python perhaps?

[ilyam8]

Hmm - is that the f-string breaking a lower version of python perhaps?

It was added relatively recently, so likely it is

[andrewm4894]

Hmm - is that the f-string breaking a lower version of python perhaps?

It was added relatively recently, so likely it is

I think would be better for me to replace all f strings as this would break anyone <3.6 which i guess is not great.

[vlvkobal]

netdata@ubuntu-20-04:/home/vlad/test/netdata$ python3 --version
Python 3.8.5

[andrewm4894]

netdata@ubuntu-20-04:/home/vlad/test/netdata$ python3 --version
Python 3.8.5

I have broken out the offending line onto two separate lines. No idea what it might be assuming 3.8.5. I will try and make a test machine using ubunutu 20 04 and 3.8.5 to see if i can recreate

[andrewm4894]

I created a ubuntu-2004-focal-v20201201 e2-medium (2 vCPUs, 4 GB memory) instance on GCP and installed and configured the collector and it is working for me so not sure what might be going on with that error.

[netdata-community-bot]

This pull request has been mentioned on Netdata Community. There might be relevant details there:

https://community.netdata.cloud/t/collector-anomalies-error-on-loading-source/581/4