Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a gRPC based plugin system #3813

Open
danielnelson opened this issue Feb 20, 2018 · 8 comments

Comments

Projects
None yet
5 participants
@danielnelson
Copy link
Contributor

commented Feb 20, 2018

Feature Request

Proposal:

Add a plugin system, similar to #1717 but using gRPC.

Current behavior:

Plugins must be compiled into Telegraf directly. The only option for external code is the exec plugin.

Desired behavior:

Be able to create plugins that can be compiled and maintained separately from Telegraf, and have improved performance with support for event driven ServiceInput's.

Use case:

Some plugins cannot be submitted to, is not general purpose enough for, or for other reasons cannot be included in the main Telegraf build. Allowing the plugin to be compiled separately and loaded at runtime into any supporting Telegraf version will reduce the overhead of maintaining these plugins.

@kerams

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2018

Do you eventually want to move all current "plugins" out so as to keep the binary size to minimum and let users manually drop in what they need, a la the config directory?

@phemmer

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2018

2 questions to start off with, more to follow depending on the answers:

  1. Does this mean that each plugin is going to have to spin up an HTTP/2 server (to be able to receive calls from telegraf), as well as telegraf running an HTTP/2 server itself (for service inputs).
  2. Will telegraf launch the plugins itself, or will they be standalone services that the admin has to manage?
@danielnelson

This comment has been minimized.

Copy link
Contributor Author

commented Feb 21, 2018

We would probably move some of the plugins into an optional package, perhaps on Debian we would have multiple packages: telegraf.deb and telegraf-extras.deb.

I think that only the gRPC server needs to run a HTTP/2 server while the client just makes HTTP/2 calls. Due to the size overhead, we probably will want to allow including multiple plugins to be in each gRPC process.

I would sure like it if Telegraf launched the plugins and managed their lifecycles itself.

We could also investigate basing it on https://github.com/hashicorp/go-plugin

@phemmer

This comment has been minimized.

Copy link
Contributor

commented Feb 21, 2018

I think that only the gRPC server needs to run a HTTP/2 server while the client just makes HTTP/2 calls.

Do you mean that telegraf isn't going to do the scheduled polling of each plugin? It'll be up to the plugin to push its metrics on the configured interval?

@danielnelson

This comment has been minimized.

Copy link
Contributor Author

commented Feb 21, 2018

That's a good point, we would want to trigger the polling. It seems like switching around the server and client roles wouldn't entirely help us either because service plugins need to send on their own schedule.

@bclermont

This comment has been minimized.

Copy link

commented Feb 22, 2018

there is a standard API call for healthcheck https://github.com/grpc/grpc/blob/master/doc/health-checking.md

@danielnelson danielnelson added the rfc label Mar 13, 2018

@m82labs

This comment has been minimized.

Copy link
Contributor

commented Mar 23, 2018

Do we need to "trigger" the polling? Or could the plugins manage it themselves and just check in with telegraf on start to get the polling interval? I really like the general idea of this method of managing plugins.

@danielnelson

This comment has been minimized.

Copy link
Contributor Author

commented Mar 23, 2018

I don't see why the plugin couldn't handle the interval itself. However, one goal we should have is to keep grpc plugin writing as simple as possible. The more we logic we push off to the client the more complicated it will be, and while we could provide a helper library, this somewhat reduces the ease of writing in unsupported languages. It does seems pretty likely we would need a helper library anyway, so maybe its not a big concern.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.