Support External Plugins using Go 1.8 plugin feature #1717
Comments
sparrc
added
the
feat
sparrc
changed the title
|
we may have experimental support for this in Telegraf 1.2, but since Go 1.8 will not be officially released at that point we will probably only see full support in Telegraf 1.3. |
|
@sparrc I have to do some reading on Go 1.8's feature, but have you seen or evaluated this as an alternative? https://github.com/hashicorp/go-plugin |
|
yes, that has been evaluated, IIRC that library doesn't give us anything we don't already have (plugins statically-compiled into a single binary) |
|
I believe you can use it to load plugins as external binaries that communicate over network RPC. That's how Terraform used to work (as separate binaries per plugin) before they changed to bundling everything into a single binary. I believe external plugins are still loaded via this mechanism. |
|
We could do external binaries, but there really is no need in that case for a plugin framework, IMHO, why not just communicate over plain grpc or text at that point? the decision to go with dynamic libraries was specifically to avoid managing external processes. |
|
Sounds good. Just wanted to offer up some options since plugins would be super beneficial for me. +1 to this plan. :) |
|
Intel snap telemetry project also uses external processes via grpc. |
this is for supporting external plugins. external plugins will depend on a few telegraf interface types, as well as a commong telegraf registry. because the Go 'plugin' package asserts that commit hashes of shared dependencies must match, we are breaking these shared dependencies into a separate repo that should very rarely change, which will allow cross-version plugins to work. see #1717
this is for supporting external plugins. external plugins will depend on a few telegraf interface types, as well as a commong telegraf registry. because the Go 'plugin' package asserts that commit hashes of shared dependencies must match, we are breaking these shared dependencies into a separate repo that should very rarely change, which will allow cross-version plugins to work. see #1717
this is for supporting external plugins. external plugins will depend on a few telegraf interface types, as well as a commong telegraf registry. because the Go 'plugin' package asserts that commit hashes of shared dependencies must match, we are breaking these shared dependencies into a separate repo that should very rarely change, which will allow cross-version plugins to work. see #1717
|
Would this supersede the exec plugin moving forward? Or would that stick around? |
|
|
|
Does this pr will allow dynamic loading of parser and serializer plugins ? (we are using custom metrics format in our company with custom serializer and parser) |
|
Answers based on #2373. It is still up in the air if we will merge this, though at this time it is on hold due to issues with shared object support in go.
No.
Only the ones that are specifically made to be external.
Probably all in one repo, but in theory yes you could do this. |
Why not ? It can be very usefull for us... |
Because this is more of a POC and it would also be significantly more complex given the way that parser and serializer plugins are loaded and configured in telegraf. |
|
Such feature would be very useful for us. For some internal projects we had to develop very custom and specific input/output plugins. We currently have the burden to make our own internal build of telegarf and container image and keep everything in sync with the upstream project. |
|
We have the same problem as mazebuhu. |
|
I think we will try to add support for gRPC plugins due to usability issues with the Go plugins. I think everyone will be happier with this solution in the end. |
|
I certainly would. I really like Go as a language but I'm far from proficient in it. gRPC would enable us to do our custom plugins in languages that we are more familiar with. Would also enable us to fairly easily port our existing monitoring tools. What issues/PRs should I track to stay up to date regarding external plugins? |
|
I just opened #3813 to track the gRPC idea. |
|
Both approaches to extension have their merits; possibly in the long-run both mechanisms should be implemented. One tradeoff the gRPC process-based plugin approach brings is increased resource utilization. In trying snaptel for example, I found each plugin enabled added roughly a dozen MiB of resident memory utilization to the overall installation. There each plugin is hosting its own isolated language runtime, gRPC server, etc. That adds up quickly with even a few 'basic' plugins enabled and can become untenable on smaller virtual servers. Also data flowing between the core agent process and plugins must be marshalled multiple times along the pipeline, incurring CPU overhead. I've found Telegraf's lighter footprint to be one of its distinguishing features and believe .so based extensions would help better maintain that. |
|
Is this patch the latest work that has been done on this front? |
|
Yes, no work has been done on this for several years. |
|
I'm currently evaluating if it's possible if this feature would enable us to use other languages (namely Rust) to collect metrics. In our use case that would mean our .so Go plugin would have a statically linked in Rust code that can be called over the C ABI. We don't want to use gRPC because we are targeting devices that have low space requirements. This feature would be better on our space usage. The major question I have is what data can be handed back to Go. The |
|
I got a toy example put together that works, but would the project be willing to use the C ABI for the shared library? It would mean we wouldn't have to maintain any shim Go code (which I would currently have to do since the change as proposed so far expects something implementing the It would necessitate writing some unsafe Go code, but with the |
|
I don't think that would be something we would merge as it would require cgo enabled, and I don't think this is an API we want to maintain officially. Wouldn't mind seeing what you have though if you can link to the code. The best methods for integrating other languages currently are either the exec plugin (for polling data), or for event driven data or for efficiency by running an external process sending to the socket_listener plugin. You can send in any of the supported data formats but InfluxDB line protocol has the best support. |
|
Ok. I'm going to add the Go plugin feature then and keep that CGo shim out of tree. We are going to publish it in a Rust crate so if you're interested I can link it here once it's complete. |
Original patch written by sparrc, brought up to date by timidger. Original from 92d8a2e message follows: support for the Go 1.8 shared object feature of loading external plugins. this support relies on the developer defining a `Plugin` symbol in their .so file that is a telegraf plugin interface. So instead of the plugin developer "Adding" their own plugin to the telegraf registry, telegraf loads the .so, looks up the Plugin symbol, and then adds it if it finds it. The name of the plugin is determined by telegraf, and is namespaced based on the filename and path. see influxdata#1717
Original patch written by sparrc, brought up to date by timidger. Original from 92d8a2e message follows: support for the Go 1.8 shared object feature of loading external plugins. this support relies on the developer defining a `Plugin` symbol in their .so file that is a telegraf plugin interface. So instead of the plugin developer "Adding" their own plugin to the telegraf registry, telegraf loads the .so, looks up the Plugin symbol, and then adds it if it finds it. The name of the plugin is determined by telegraf, and is namespaced based on the filename and path. see influxdata#1717
Original patch written by timidger. Original from 55af621 message follows: support for the Go 1.8 shared object feature of loading external plugins. this support relies on the developer defining a `Plugin` symbol in their .so file that is a telegraf plugin interface. So instead of the plugin developer "Adding" their own plugin to the telegraf registry, telegraf loads the .so, looks up the Plugin symbol, and then adds it if it finds it. The name of the plugin is determined by telegraf, and is namespaced based on the filename and path. see influxdata#1717
Original patch written by sparrc, brought up to date by timidger. Original from 92d8a2e message follows: support for the Go 1.8 shared object feature of loading external plugins. this support relies on the developer defining a `Plugin` symbol in their .so file that is a telegraf plugin interface. So instead of the plugin developer "Adding" their own plugin to the telegraf registry, telegraf loads the .so, looks up the Plugin symbol, and then adds it if it finds it. The name of the plugin is determined by telegraf, and is namespaced based on the filename and path. see influxdata#1717
Original patch written by sparrc, brought up to date by timidger. Original from 92d8a2e message follows: support for the Go 1.8 shared object feature of loading external plugins. this support relies on the developer defining a `Plugin` symbol in their .so file that is a telegraf plugin interface. So instead of the plugin developer "Adding" their own plugin to the telegraf registry, telegraf loads the .so, looks up the Plugin symbol, and then adds it if it finds it. The name of the plugin is determined by telegraf, and is namespaced based on the filename and path. see influxdata#1717
|
Closed in #6024 |
We should support building and loading of external plugins as dynamic libraries, using Go's new "plugin" feature, available in Go 1.8: https://tip.golang.org/pkg/plugin
For this we will likely want to maintain an external repository where we can aggregate "external" plugins which are not part of the telegraf statically-linked build.
Telegraf will be able to load all ".so" plugins from a directory, adding them to the regular plugin registry before the config file is loaded.
The text was updated successfully, but these errors were encountered: