Plan migration to go.d plugin for common collectors

[cakrit]

Feature idea summary

Related to #5006

We need to plan the next step for an assisted migration of users from the old plugins to go.d, where we have reimplemented a collector.

Expected behavior

Come up with a design of what we will implement within v1.13

[Ferroin]

I think from a longer-term perspective, we need to have a way to prioritize implementations of a collector module. Our current approach of blacklisting the old module in the original collector works fine for migrating stuff to C (because everybody has C support implicitly), but I don't see it working well for migrating from one optional language to another. What we really need is a unified namespace for collection sources (we kind of already have that), and a prioritized list of which implementation of a given collector to use (so, C would be at the top of the list, followed by Go, then Python, then Node (until we drop it), then Bash, then everything else).

My first thought is to change the plugin API to provide some way to negotiate what collectors will run in each plugin. Making the API bidirectional and having plugins ask for configuration from the core during startup would make this pretty easy (because the core could just report the modules that shouldn't run as disabled in the config), and would also simplify unifying the configuration across plugins. If we take this approach, we could have each plugin report a 'weight'for the collectors it implements, and wouldn't need to bake-in the above-mentioned list (and could make it easy for people to override the stock implementations with their own).

[ilyam8]

Plan migration to go.d plugin for common modules Plan migration to go.d plugin for common collectors

We need to agree on definitions, i see it's still confusing. And I think it doesn't matter if module word has special meaning in go.

Netdata currently use: collector, plugin, module, orchestrator.

collector is a data collection plugin (collector == (data collection) plugin).

orchestrator is a special type of collector. It doesn't collect any data by themselve. It contains data collection modules.

We need strictly define them.

[ilyam8]

Our current approach of blacklisting the old module in the original collector works fine for migrating stuff to C (because everybody has C support implicitly)

It was easy because those modules were modules without jobs/configuration. Otherwise it wouldn't be so easy.

My first thought is to change the plugin API to provide some way to negotiate what collectors will run in each plugin. Making the API bidirectional and having plugins ask for configuration from the core

This. We discussed it (central configuration service)

[ilyam8]

We need to plan the next step for an assisted migration of users from the old plugins to go.d

• disable python module by default
• enable corresponding go module by default
• add additional configuration translation logic to the installer (python script?)

Python module can be disabled:

• in python.d.plugin (can't be enabled via conf, completely disabled)
• in module, using disable_by_default = True (can be enabled again via conf)

Tranlator script logic depends on the disabling approach.

[ilyam8]

Configuration translation is very depends on module. Lets take mysql:

Python version configuration types:

• my.cnf
• socket
• tcp

mycnf2:
  name     : 'local'
  'my.cnf' : '/etc/mysql/my.cnf'

socket3:
  name     : 'local'
  # user     : ''
  # pass     : ''
  socket   : '/var/lib/mysql/mysql.sock'

tcpipv4:
  name     : 'local'
  # user     : ''
  # pass     : ''
  host     : '127.0.0.1'
  port     : '3306'

Go version uses dsn syntax:

[username[:password]@][protocol[(address)]]/dbname[?param1=value1&...&paramN=valueN]

jobs:
  - name: local
    dsn: netdata@tcp(127.0.0.1:3306)/

  - name: local
    dsn: root@/

  - name: local
    dsn: root@unix(/var/run/mysqld/mysqld.sock)/

  - name: local
    dsn: root@unix(/usr/local/var/mysql/mysql.sock)/

__

So it is not a trivial task, it is easy to make a mistake.

[cakrit]

I went through apache and freeradius as well.
My idea is the following:

• We have the daemon detect that a user is currently using one or more obsolete modules and alert him/her via the UI.
• We have an interactive command-line tool that will do what it can automatically or semi-automatically (i.e. with confirmations) and prompt the user to provide info for the rest (e.g. for the mysql DSN, if we can't reconstruct it). We could even have a non-interactive mode that only enables the go modules that are safe to do so, because we can predictably build the new config from the existing one (e.g. apache).

[ilyam8]

I like the idea to have a notification in the UI. I think we can live with it until we have a Central Configuration in place.

[cakrit]

We will proceed like this then.

[stale]

This issue has been inactive for 30 days. It will be closed in one week, unless it is updated.

[ilyam8]

Ok, back to this.

We need to mirgate module in lang A to module in lang B.

Unfortunately there is no way to completely avoid breaking changes: there is inconsistency between new and old modules, we rethink things doing rewrite, it is a good chance to fix poor naming/grouping/etc.

Natural way for me:

• Define deprecation schedule (like obsolete N modules every 3 month).
• Warn user about it during that periond (in logs, in GUI - if possible, in docs).
• Module in lang B is disabled by default during this periond.
• Provide a migration utility, it helps user to migrate from module in lang A to module in lang B.
• No ulterior rewriting of files. User should run migration tool on its own.
• After end of obsolete period we completely disable lang A module, enable by default lang B module.

Pros:

• Easy to understand, easy to follow plan, crystal clear.
• Relatively long period for user to prepare for changes. (we can make a plan for half a year, for a year).

[Ferroin]

@ilyam8 Sounds sensible to me, though I'd go by minor releases (when we bump the second number in the version, not the third), not months. That will make things easier to track for users who are using stable releases instead of nightlies (and thus who are less likely to notice things other than release notes).

I would ideally like to get Go support working on Gentoo before we start actively disabling existing modules and enabling the Go modules, but that's not absolutely critical IMO as we aren't even officially providing a Portage overlay for Gentoo for Netdata yet (though hopefully that will change in the near future).

[ilyam8]

Other way is to try to hide migration from the user and make it transparent. All work will be done under the hood - not problems, no bugs, user will notice nothing.

My point that it is not possible and more destructive and error prone then prev approach.

• It will be frustrating for the user. He configure module lang A, it works, then under the hood we migrate it to the module lang B, user is unaware. He want to change/tune config and goes to module lang A config...
• We will need to rewrite user config files (in /etc/) without asking him.
• And the main thing that still there will be "breaking" changes (avoiding them was the main goal of this approach btw) - new metric names, etc.
• Migration tool can fail badly - disable module lang A and not enable module lang B and other fancy bugs.

---

As a user i would like to have a clear deprecation schedule, where i could see all upcoming changes and dates and prepare my enviroment.

[stale]

This issue has been inactive for 30 days. It will be closed in one week, unless it is updated.

[stale]

This issue has been inactive for 30 days. It will be closed in one week, unless it is updated.

[stale]

This issue has been automatically closed due to extended period of inactivity. Please reopen if it is still valid. Thank you for your contributions.