add IPFS plugin

[davidak]

with Bandwidth and Peers chart

[davidak]

@paulfantom is this the right way to query multiple URLs?

anything else to improve?

[ktsaou]

For python guide lines @paulfantom will review.
For sure it needs a conf file with proper defaults, like the rest of the python modules.
Also, if there something to configure on the ipfs side to allow or enable this, please give in the conf file proper instructions.

[davidak]

• why is the name ipfs_local and postfix: local different? i think it's cleaner with :
• url from config don't get passed through to the plugin, it uses default

[paulfantom]

Those lines won't have any meaning since self.url will be overridden when python.d.plugin executes check method (from UrlService).

[paulfantom]

why is the name ipfs_local and postfix: local different? i think it's cleaner with :

This is something to be done in dashboard.html. If you want I can change it later.

url from config don't get passed through to the plugin, it uses default

That's because you don't use url, you use baseurl. Configuration is loaded in check method, not in constructor (__init__).
I think you could add something like this to Service class:

def check(self):
    if UrlService.check(self):
        return True
    if len(self.url) == 0:
        self.baseurl = "http://localhost:5001/"
     else:
        self.baseurl = self.url
    test = self._get_data()
    if test is None or len(test) == 0:
        return False
    else:
        return True

Some quick explanation, how it works (since documentation is non-existent):
Module needs to have check, create, and update methods, since they are executed by python.d.plugin.
Those 3 methods are provided by SimpleService class, which also implements many other useful things (like threading or wrappers on chart creation).
Other *Service classes are children of SimpleService class, and they just provide some helper functions (like HTTP basic auth in UrlService or handling socket connections in SocketService).

And every SimpleService child class needs to have:

1. constructor with:
   • ParentService.__init__(self, configuration=configuration, name=name)
   • self.definitions - this stores all chart definitions (usually defined in global variable CHARTS)
   • self.order - order in which charts are displayed (usually in global variable ORDER)
2. _get_data method which returns dictionary where key is dimension name and value is a number to show on graph.

And if you want to get something from configuration, you need to parse configuration variable which stores everything that is defined in yaml file. You can do it either in constructor or in custom check method (preferred).
But remember that by default every service parses it's configuration file not in constructor, but in check method.

[paulfantom]

I hope my explanations are good enough.

[paulfantom]

This shouldn't return None as dictionary value. If dimension has None value it shouldn't be returned.

[davidak]

should it just return None then?

[paulfantom]

It should check if peers and bandwidths are None.
If peers is None, but bandwidths have some data, it should return {'in': bandwidth_in, 'out': bandwidth_out}. if bandwidth_in is None but other have some data it should return {'peers': peers, 'out': bandwidth_out}. And so on.
None should be returned if there is no data in any dimension.

[paulfantom]

Also this will be the first service to query multiple URLs.

[davidak]

Thanks for the explanations!

I added your check() code.

    if UrlService.check(self):
        return True

leads to ipfs_localhost cannot find check() function. so i deleted that lines.

i added a print to see if i get the correct url from configuration, but it's empty!

    def check(self):
        print(self.url)
        if len(self.url) == 0:

so that don't work :(

[paulfantom]

def check(self):
        print(self.url)
        if len(self.url) == 0:

with this you overrode gathering data from configuration, so object won't have self.url.

[paulfantom]

My proposed changes:

• from constructor, remove:

if len(self.url) == 0:
    self.baseurl = "http://localhost:5001/"
else:
    self.baseurl = self.url

• add my previous check method
• wrap in try..except block this code:

peers = self._get_peers()
bandwidth_in, bandwidth_out = self._get_bandwidth()

• url taken from configuration shouldn't have / at the end
• there should be a possibility to return just a fraction of available dimensions

Basically it should look something like this:

# -*- coding: utf-8 -*-
# Description: IPFS netdata python.d module
# Authors: Pawel Krupa (paulfantom), davidak

from base import UrlService
import json

# default module values (can be overridden per job in `config`)
# update_every = 2
priority = 60000
retries = 60

# default job configuration (overridden by python.d.plugin)
# config = {'local': {
#     'update_every': update_every,
#     'retries': retries,
#     'priority': priority,
#     'url': 'http://localhost:5001'
# }}

# charts order (can be overridden if you want less charts, or different order)
ORDER = ['bandwidth', 'peers']

CHARTS = {
    'bandwidth': {
        'options': [None, 'IPFS Bandwidth', 'kbits/s', 'Bandwidth', 'ipfs.bandwidth', 'line'],
        'lines': [
            ["in", None, "absolute", 8, 1000],
            ["out", None, "absolute", -8, 1000]
        ]},
    'peers': {
        'options': [None, 'IPFS Peers', 'peers', 'Peers', 'ipfs.peers', 'line'],
        'lines': [
            ["peers", None, 'absolute']
        ]}
}


class Service(UrlService):
    def __init__(self, configuration=None, name=None):
        UrlService.__init__(self, configuration=configuration, name=name)
        self.order = ORDER
        self.definitions = CHARTS

    def _get_bandwidth(self):
        """
        Format data received from http request
        :return: int, int
        """
        self.url = self.baseurl + "/api/v0/stats/bw"
        try:
            raw = self._get_raw_data()
        except AttributeError:
            return None

        try:
            parsed = json.loads(raw)
            bw_in = int(parsed['RateIn'])
            bw_out = int(parsed['RateOut'])
        except:
            return None

        return bw_in, bw_out

    def _get_peers(self):
        """
        Format data received from http request
        :return: int
        """
        self.url = self.baseurl + "/api/v0/swarm/peers"
        try:
            raw = self._get_raw_data()
        except AttributeError:
            return None

        try:
            parsed = json.loads(raw)
            peers = len(parsed['Strings'])
        except:
            return None

        return peers

    def _get_data(self):
        """
        Get data from API
        :return: dict
        """
        try:
            peers = self._get_peers()
            bandwidth_in, bandwidth_out = self._get_bandwidth()
        except:
            return None
        data = {}
        if peers is None:
            data['peers'] = peers
        if bandwidth_in is not None and bandwidth_out is not None:
            data['in'] = bandwidth_in
            data['out'] = bandwidth_out

        if len(data) == 0:
            return None
        return data

    def check(self):
        if UrlService.check(self):
            return True
        if len(self.url) == 0:
            self.baseurl = "http://localhost:5001"
        else:
            self.baseurl = self.url

        test = self._get_data()
        if test is None or len(test) == 0:
            return False
        else:
            return True

or like this:

# -*- coding: utf-8 -*-
# Description: IPFS netdata python.d module
# Authors: Pawel Krupa (paulfantom), davidak

from base import UrlService
import json

# default module values (can be overridden per job in `config`)
# update_every = 2
priority = 60000
retries = 60

# default job configuration (overridden by python.d.plugin)
# config = {'local': {
#     'update_every': update_every,
#     'retries': retries,
#     'priority': priority,
#     'url': 'http://localhost:5001'
# }}

# charts order (can be overridden if you want less charts, or different order)
ORDER = ['bandwidth', 'peers']

CHARTS = {
    'bandwidth': {
        'options': [None, 'IPFS Bandwidth', 'kbits/s', 'Bandwidth', 'ipfs.bandwidth', 'line'],
        'lines': [
            ["in", None, "absolute", 8, 1000],
            ["out", None, "absolute", -8, 1000]
        ]},
    'peers': {
        'options': [None, 'IPFS Peers', 'peers', 'Peers', 'ipfs.peers', 'line'],
        'lines': [
            ["peers", None, 'absolute']
        ]}
}


class Service(UrlService):
    def __init__(self, configuration=None, name=None):
        UrlService.__init__(self, configuration=configuration, name=name)
        try:
            self.baseurl = str(self.configuration['url'])
        except (KeyError, TypeError):
            self.baseurl = "http://localhost:5001"
        self.order = ORDER
        self.definitions = CHARTS

    def _get_bandwidth(self):
        """
        Format data received from http request
        :return: int, int
        """
        self.url = self.baseurl + "/api/v0/stats/bw"
        try:
            raw = self._get_raw_data()
        except AttributeError:
            return None

        try:
            parsed = json.loads(raw)
            bw_in = int(parsed['RateIn'])
            bw_out = int(parsed['RateOut'])
        except:
            return None

        return bw_in, bw_out

    def _get_peers(self):
        """
        Format data received from http request
        :return: int
        """
        self.url = self.baseurl + "/api/v0/swarm/peers"
        try:
            raw = self._get_raw_data()
        except AttributeError:
            return None

        try:
            parsed = json.loads(raw)
            peers = len(parsed['Strings'])
        except:
            return None

        return peers

    def _get_data(self):
        """
        Get data from API
        :return: dict
        """
        try:
            peers = self._get_peers()
            bandwidth_in, bandwidth_out = self._get_bandwidth()
        except:
            return None
        data = {}
        if peers is None:
            data['peers'] = peers
        if bandwidth_in is not None and bandwidth_out is not None:
            data['in'] = bandwidth_in
            data['out'] = bandwidth_out

        if len(data) == 0:
            return None
        return data

Both are valid.

Also what is returned by ipfs when you request localhost:5001/api/v0/stats ?

[davidak]

the second makes it clearer where the url comes from, eg. self.configuration['url'].

now i can configure multiple checks!

Also what is returned by ipfs when you request localhost:5001/api/v0/stats ?

curl http://localhost:5001/api/v0/stats
{"Message":"This command can't be called directly. Try one of its subcommands.","Code":1}

that is no command.

[davidak]

last thing is the name in the webui menu.

it would be nice to have postfix: local only when there are actually multiple instances, else only postfix or the instance name other than local.

also all names should be displayed as postfix: local and not postfix_local.

i tried something like this for ipfs in index.html, also restartet netdata but nothing changed.

be51843#diff-d23fe9001e474c96419c8f2519cbef67

@paulfantom can you do that?

[paulfantom]

To display ipfs: local instead of ipfs_local on dashboard you need to add two things to index.html:
first is:

case 'postfix':
case 'ipfs':
case 'redis':

and the second:

'redis': {
    title: 'Redis',
    info: undefined
},
'ipfs': {
    title: 'ipfs',
    info: undefined
},
'phpfpm': {
    title: 'PHP-FPM',
    info: undefined,
},

And that's all. Remember about refreshing page with Ctrl + F5.

[davidak]

i have done exactly that, but in /root/netdata/web/index.html, not in /usr/share/netdata/web/index.html.

so it's ready to merge i thing. :)

@paulfantom thanks again for your help!

[paulfantom]

Ok, I think it is good.
Merging.