Registry enhancements

[ktsaou]

To find what is the registry and how it works, check: https://github.com/firehol/netdata/wiki/mynetdata-menu-item

The original discussion for the implementation of the registry is #310 .

These are new features for the registry (my-netdata related tasks), we could implement:

Cross server dashboards

• Cross server charts comparison.

The dashboards could allow us to select / pin one or more charts. All selected charts are given to the registry and follow us from dashboard to dashboard. On each dashboard there will be button to show the selected charts from all dashboards on a single page, one below the other, so that cross server comparison of charts, is possible ! (I really like this one 👍 )

• Dynamic cross servers dashboards.

  There could be a menu item in my-netdata menu to dynamically create cross server dashboards. This page could work as follows:

  1. The page is showing a search box where you can search the context of the charts. The context is the attribute that defines the template of a chart. For example, if we have 2 mysql servers, we have charts mysql_A.net and mysql_B.net but both have context = mysql.net.
  2. Once we search for a context the dashboard will walk through all servers to find all their charts in this context and present a gauge or easypie chart for each server (one next to the other) and below them a full width chart for the selected server (this full width chart will allow hover).
  3. We should be able to add as many such blocks as needed.
  4. The URL of the dashboard (after the hash) should change to allow recreating the same page by just hitting F5.
  5. A button should exit to save this dynamic dashboard at the my-netdata menu.

• Verify my-netdata servers are accessible

When the user presses the my-netdata dropdown menu, we could start sending HELLO messages to all the servers listed to check if they are available or not.

For servers that netdata has multiple URLs, this is required to decide which URL is the proper one for the current location of the user (for example netdata1 has been accessed as http://localhost:19999 and http://monitor.my.network/server1 - which URL should the menu item pick?)

While this happens, we could also render sparklines of the CPU of each server on the menu.

• Search box at my-netdata.

  Add a search box at the my-netdata menu, to allow us find a server. This is required if we have several dozens (or even hundreds) of netdata servers.

• Grouping servers at my-netdata.

  Allow us to tag netdata servers so that they can be grouped. Ideally, multiple tags per server should be supported and filtering based on tag should be provided.

---

Use the registry for cross server personal storage

• Dashboard settings saved at the registry

  Allow saving at the registry, limited dashboard settings per person:

  • online help: on/off
  • dashboard theme: slate/white

  This will allow dashboard settings to be propagated from server to server automatically.

• Save visible URL and API URL.

  Today the registry stores the base URL of a netdata API. This means that custom dashboards cannot be bookmarked by the registry. We need to find a way of storing the visible url too and also exposing them at the my-netdata menu.

---

Better personal identity management

• Password protect the persons identities

  Instead of sharing person GUIDs we could store a HMAC hash with each peetrson. Users willing to password protect their identity will simply enter their email and password at the dashboard, an HMAC signature will be generated that will be stored at the registry for the person (the registry will not store any clear text password - only the HMAC signature).

  The user to recover his/her person GUID, he would be able to give his/her email and password on another dashboard, which will be checked by the registry. If the HMAC hashes match, the registry will automatically impersonate the user (assign to him his/her correct person GUID).

  The user could also press "logout" to make the registry clear the person GUID from this web browser.

• Impersonate with QR.

  Allow impersonate with a QR code. So a user, to quickly link his/her mobile phone and his/her desktop, could just scan on his/her mobile phone the QR code.

• Sharing my-netdata menu without sharing identity.

  Allow us to share our netdata servers list, without sharing our identity, so that we can give our list of servers to someone else, without exposing our personal GUID.

[team4music]

Hi, i sadly can't get it to work yust nothing happens or i get the error message "failed to contact"..

My Master Server (registry) is using

    enabled = yes
    registry db directory = /var/lib/netdata/registry
    netdata unique id file = /var/lib/netdata/registry/netdata.public.unique.id
    registry db file = /var/lib/netdata/registry/registry.db
    registry log file = /var/lib/netdata/registry/registry-log.db
    registry save db every new entries = 1000000
    registry expire idle persons days = 365
    registry to announce = http://system.gnitter.de:19999

Sadly they do not want to communicate with each other. Is there something i missed?
My Slave is using

[registry]
    enabled = no
    # registry db directory = /var/lib/netdata/registry
    # netdata unique id file = /var/lib/netdata/registry/netdata.public.unique.id
    # registry db file = /var/lib/netdata/registry/registry.db
    # registry log file = /var/lib/netdata/registry/registry-log.db
    # registry save db every new entries = 1000000
    # registry expire idle persons days = 365
    #registry domain = http://system.gnitter.de:19999
    registry to announce = http://system.gnitter.de:19999
    # registry hostname = NbI-1704
    # max URL length = 1024
    # max URL name length = 50

PS I noticed a double slash Bug within the API Hello. But this will likely not be the source of the problem

[ktsaou]

I just talked to you on twitter. Your registry http://system.gnitter.de:19999/api/v1/registry?action=access&machine=274f7812-2065-11e6-b19a-931ed2ff508c&name=NbI-1704&url=http%3A%2F%2F31.185.111.69%3A19999%2F&_=1463986806350 is not reachable. This is what the browser logs on its console:

jquery-1.12.0.min.js:4 GET http://system.gnitter.de:19999/api/v1/registry?action=access&machine=274f78…8c&name=NbI-1704&url=http%3A%2F%2F31.185.111.69%3A19999%2F&_=1463986806350 net::ERR_CONNECTION_TIMED_OUTsend @ jquery-1.12.0.min.js:4n.extend.ajax @ jquery-1.12.0.min.js:4NETDATA.registry.access @ dashboard.js?v37:5602(anonymous function) @ dashboard.js?v37:5561(anonymous function) @ dashboard.js?v37:5586i @ jquery-1.12.0.min.js:2j.fireWith @ jquery-1.12.0.min.js:2y @ jquery-1.12.0.min.js:4c @ jquery-1.12.0.min.js:4
dashboard.js?v37:516 ERROR 410: Netdata registry ACCESS failed: http://system.gnitter.de:19999

Your browser talks to the registry.

Your machine lack disk charts. Please open an issue to discuss this.

[jurgenhaas]

Unfortunately the EFF Privacy Badger recognizes registry.my-netdata.io as a tracking unit and blocks access by default. Then that feels as if the registry doesn't work and as a user you can put that url on a white list with that browser plugin.

It would be great if you could contact the EFF so that they are aware of your service here and hopefully build in an exception to their rules.

[ktsaou]

I opened the ticket EFForg/privacybadger#850 to their project, according to the instructions on their site.

Thanks for reporting this.

[shenjichenai]

How should I set the registry?Have the default configuration registry in the netdata.conf?If have ,I din't find. Should I changed the default configuration registry or add a new registry configuration in netdata.conf?Can you give me a template? thank you!

[simonnagl]

Sorry for the short answer. This should not seem unfriendly:
Have you've seen this? https://github.com/firehol/netdata/wiki/mynetdata-menu-item

If there are any questions left please ask!

[stale]

Currently netdata team doesn't have enough capacity to work on this issue. We will be more than glad to accept a pull request with a solution to problem described here. This issue will be closed after another 60 days of inactivity.

[ktsaou]

We are already working on a few of these.
@gmosx you may want to cherry pick features from this. This is generally on our roadmap...

[cakrit]

Updated list, linking with #3990 and based on the current roadmap:

Unique TODOs in this issue

• Cross server charts comparison - TBD in Cloud

  The dashboards could allow us to select / pin one or more charts. All selected charts are given to the registry and follow us from dashboard to dashboard. On each dashboard there will be button to show the selected charts from all dashboards on a single page, one below the other, so that cross server comparison of charts, is possible ! (I really like this one 👍 )

• Dynamic cross servers dashboards. - TBD in Cloud

  There could be a menu item in my-netdata menu to dynamically create cross server dashboards. This page could work as follows:

  1. The page is showing a search box where you can search the context of the charts. The context is the attribute that defines the template of a chart. For example, if we have 2 mysql servers, we have charts mysql_A.net and mysql_B.net but both have context = mysql.net.
  2. Once we search for a context the dashboard will walk through all servers to find all their charts in this context and present a gauge or easypie chart for each server (one next to the other) and below them a full width chart for the selected server (this full width chart will allow hover).
  3. We should be able to add as many such blocks as needed.
  4. The URL of the dashboard (after the hash) should change to allow recreating the same page by just hitting F5.
  5. A button should exit to save this dynamic dashboard at the my-netdata menu.

• Verify my-netdata servers are accessible - TBD

  When the user presses the my-netdata dropdown menu, we could start sending HELLO messages to all the servers listed to check if they are available or not.

  For servers that netdata has multiple URLs, this is required to decide which URL is the proper one for the current location of the user (for example netdata1 has been accessed as http://localhost:19999 and http://monitor.my.network/server1 - which URL should the menu item pick?)

  While this happens, we could also render sparklines of the CPU of each server on the menu.

• Search box at my-netdata. - Provided in Cloud Sign-In #5095

  Add a search box at the my-netdata menu, to allow us find a server. This is required if we have several dozens (or even hundreds) of netdata servers.

• Grouping servers at my-netdata. - TBD in cloud

  Allow us to tag netdata servers so that they can be grouped. Ideally, multiple tags per server should be supported and filtering based on tag should be provided.

Features from #3990

• Decouple the registry from the dashboard - In progress, part of the new console UI.
• Live server status at the registry menu - Will only show the status, if the server has alarms or not. The rest will be in the network view.
• Last visited servers
• Searchable list of servers - Provided in Cloud Sign-In #5095
• Renaming of servers and groups - TBD in cloud
• Login to enable the registry - Provided in Cloud Sign-In #5095
• Dashboard settings registry - UI settings persisted in the cloud and in local storage
• Server groups - TBD in cloud
• Sharing of server groups - TBD in cloud
• Claiming servers - Claiming nodes for netdata cloud and easier addition of known agents #6318

---

To be discussed

• Save visible URL and API URL.

  Today the registry stores the base URL of a netdata API. This means that custom dashboards cannot be bookmarked by the registry. We need to find a way of storing the visible url too and also exposing them at the my-netdata menu.

• Password protect the persons identities

  Instead of sharing person GUIDs we could store a HMAC hash with each peetrson. Users willing to password protect their identity will simply enter their email and password at the dashboard, an HMAC signature will be generated that will be stored at the registry for the person (the registry will not store any clear text password - only the HMAC signature).

  The user to recover his/her person GUID, he would be able to give his/her email and password on another dashboard, which will be checked by the registry. If the HMAC hashes match, the registry will automatically impersonate the user (assign to him his/her correct person GUID).

  The user could also press "logout" to make the registry clear the person GUID from this web browser.

Obsolete

• Impersonate with QR. - Obsolete via SSO and Authorization/Authentication

  Allow impersonate with a QR code. So a user, to quickly link his/her mobile phone and his/her desktop, could just scan on his/her mobile phone the QR code.

• Sharing my-netdata menu without sharing identity. - Obsolete via SSO and Authorization/Authentication

  Allow us to share our netdata servers list, without sharing our identity, so that we can give our list of servers to someone else, without exposing our personal GUID.

[cakrit]

We are no longer maintaining netdata.cloud features here, closing this.