Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Add native plugins support to the Collector #1050

Closed
wants to merge 8 commits into from

Conversation

ledor473
Copy link
Member

@ledor473 ledor473 commented Sep 5, 2018

Which problem is this PR solving?

This PR provides the foundation to support plugins in Jaeger using native Go plugins (pkg/plugin). Resolves #422

Supported plugins:

  • Storage (spanstore + dependencystore)
  • Span Processor (Sanitizer, PreProcessor, PreSave, SpanFilter)

Short description of the changes

The current PR is working and this comment #1050 (comment) includes an example of a storage plugin that seems to work so far.

What is missing

  • Configuration file to order the plugins
  • Unit/Integration tests
  • More examples
  • Documentation
  • Support for that in the Dockerfile

How to run it

  1. Compile the plugins
cd examples/collector/plugins/
go build  -buildmode=plugin sanitizer_logger.go
go build  -buildmode=plugin pre_processor_logger.go
go build  -buildmode=plugin pre_save_logger.go
go build  -buildmode=plugin span_filter_logger.go
  1. Run the collector with the proper flag
SPAN_STORAGE_TYPE=memory LOG_LEVEL=debug PLUGINS_DIRECTORY=`pwd`/examples/collector/plugins/ go run cmd/collector/main.go
  1. Send traces and look at the logs, it should display something like:
PreProcessSpans...  4 spans
SpanFilter... TraceID=66333d0d4a3452c SpanID=ef9dbde43426f82f OperationName=truncated
SpanFilter... TraceID=66333d0d4a3452c SpanID=ef9dbde43426f82f OperationName=truncated
SpanFilter... TraceID=66333d0d4a3452c SpanID=7c8453653cab6cbf OperationName=truncated
SpanFilter... TraceID=66333d0d4a3452c SpanID=7c8453653cab6cbf OperationName=truncated
SpanFilter... TraceID=66333d0d4a3452c SpanID=8de1edfd89f8e32e OperationName=truncated
SpanFilter... TraceID=66333d0d4a3452c SpanID=8de1edfd89f8e32e OperationName=truncated
SpanFilter... TraceID=66333d0d4a3452c SpanID=66333d0d4a3452c OperationName=truncated
SpanFilter... TraceID=66333d0d4a3452c SpanID=66333d0d4a3452c OperationName=truncated
Sanitizer... TraceID=66333d0d4a3452c SpanID=ef9dbde43426f82f OperationName=truncated
Sanitizer... TraceID=66333d0d4a3452c SpanID=7c8453653cab6cbf OperationName=truncated
Sanitizer... TraceID=66333d0d4a3452c SpanID=66333d0d4a3452c OperationName=truncated
Sanitizer... TraceID=66333d0d4a3452c SpanID=8de1edfd89f8e32e OperationName=truncated
PreSave... TraceID=66333d0d4a3452c SpanID=66333d0d4a3452c OperationName=truncated
PreSave... TraceID=66333d0d4a3452c SpanID=8de1edfd89f8e32e OperationName=truncated
PreSave... TraceID=66333d0d4a3452c SpanID=ef9dbde43426f82f OperationName=truncated
PreSave... TraceID=66333d0d4a3452c SpanID=7c8453653cab6cbf OperationName=truncated

Gotchas

  • Mac& Linux only
  • If you want to debug on macOS using delve, you need to compile Go from source to get this fix golang/go@9c83383
    • If needed, something like this could be added to allow someone to debug the rest of the codebase

@yurishkuro
Copy link
Member

yurishkuro commented Sep 6, 2018

Great! Looks very promising.

Side note: I didn't even know we had presave postsave... Whatever for?

A few comments on the design:

  • I think this should be generic capability, not collector specific. That means, besides going into pkg/plugins module, a global cli flag for plugins dir, similar to metrics/logging. Btw, you mentioned env var: that should come auto from viper, as long as you have a flag. However, ultimately we will need this to behave similar to span-storage-type env var, because I can envision some plugins needing access to cli flags.
  • perhaps plugin factory can be something like Get(name string, type reflect.Type) ([]interface{}, error), ie factory initially loads all plugin libs and allows to search for symbols of specific name and type, returning all found. This way we don't hardcode specific interfaces inside the factory. The caller is responsible for chaining or whatever is appropriate. The callers may also deal with things that implement Initializable/Configurable, already introduced in storage.Factory.
  • one important consideration for things like filters/sanitizers is the order. Not sure how to enforce it; one option is instead of walking a directory we take a file that lists plugin files in user-defined order and we respect it. That seems cleaner and more explicit than the other approaches I can think of, yet does not address the case of needing to inject a plugin in the middle of standard sanitizer list (we can dismiss the insert use case "as depressing").
  • lastly, you're right that we need to demonstrate using plugins with docker, ideally having plugins themselves also distributed as docker images.

You asked for examples, I think the ultimate example would be exposing one of the existing storage impls as a real plugin (despite them being also statically linked), to demo the handling of configuration.

@jpkrohling
Copy link
Contributor

Stupid question, but doesn't pkg/plugin has the drawback of requiring the plugin to be build with the exact same "vendor" dependencies as the main binary? This would mean that people can't just build and ship their own plugin without forking Jaeger at the specific target version and building their plugin based on the fork, right?

@yurishkuro
Copy link
Member

Yes afaik

@ledor473
Copy link
Member Author

ledor473 commented Sep 6, 2018

Yes I think the handling of dependencies in a plugin is not fully working as the issues mentioned here are still open.

The main reason I started with the filters/processors is that a storage implementation would likely require additional libraries. With the current bugs, I'm not sure if it's even feasible to implemement a storage plugins without using hashicorp/go-plugin...

That being said, I'm not sure neither if using hashicorp/go-plugin for filters/processors would be a good idea performance wise.

I could work on the following changes:

  • Move to a more generic approach
  • Plugins access to the CLI
  • Plugins access to the logger
  • Ordering of the plugins

But if the dependency issue is a deal breaker for filters/processors (as it is for storage), I would start exploring hashicorp/go-plugins instead.

@jpkrohling
Copy link
Contributor

I'm not sure it's a deal breaker, but it might change how we support/document this. If we are going to support only the plugins we build as part of the regular binary, then I think it's OK to have it like this. But if we are going to say that we support people building their own plugins, then I think we need to sort out the dependency problem.

@ledor473
Copy link
Member Author

It turns out much harder than expected to do the proposed changes but I feel (even if it's not fully done yet) that it's a much cleaner implementation!

So here's what I added in the last commits:

  • Moved the core features to pkg/plugin and the rest to cmd/collector/app/plugin
  • Moved the --collector.plugins-directory to the PLUGINS_DIRECTORY environment variable because we need to be able to load the plugins before using Viper
  • Configurable plugins via CLI args:
SPAN_STORAGE_TYPE=memory LOG_LEVEL=info PLUGINS_DIRECTORY="/Users/ledor473/go/src/github.com/jaegertracing/jaeger/examples/collector/plugins/" go run cmd/collector/main.go -h

Usage:
...

Available Commands:
...

Flags:
...
      --pre-save-log-interval int        How often should the PreSave Logger should print (default 30)
...
  • Ability to provide the logger to the plugins:
{"level":"info","ts":1537905084.1992269,"caller":"plugins/pre_save_logger.go:58","msg":"PreSave Logger","rate":0,"count":0,"interval":3}
  • Ability to add metrics (untested)

Things to do:

  • Discuss a way to have a logger before the application logger is created (something like a "bootstrap logger" that would be discarded afterward?). This is to replace the usage to os.Stderr
  • Plugin ordering

@yurishkuro
Copy link
Member

Discuss a way to have a logger before the application logger is created

do we need to? the storage factory facade is already able to pass loggers to the underlying factories.

@ledor473
Copy link
Member Author

So the logger problem is mainly because it's getting created here (in RunE) https://github.com/jaegertracing/jaeger/pull/1050/files#diff-03e42ee625fab9eaa63d9d9daa235f8eR90
But I would like to use it here: https://github.com/jaegertracing/jaeger/pull/1050/files#diff-03e42ee625fab9eaa63d9d9daa235f8eR202

The usage is mainly in that method: https://github.com/jaegertracing/jaeger/pull/1050/files#diff-e6ce03f3742f3a6c8b3e443d8e1a53daR58
The debug logs indicate plugin detection and attempt to find symbol.
It's also used in case there's a symbol that is found but is not of the proper type.

A similar thing that would benefit from having a bootstrap logger is this method: https://github.com/jaegertracing/jaeger/pull/1050/files#diff-03e42ee625fab9eaa63d9d9daa235f8eR62 that currently use os.Stderr directly

@yurishkuro
Copy link
Member

maybe punt on the logger for now, use io.Writer same as that last example.

@yurishkuro yurishkuro mentioned this pull request Nov 24, 2018
8 tasks
@ledor473
Copy link
Member Author

ledor473 commented Dec 1, 2018

I've looked at the other PR for hashicorp/go-plugins and it look great so far, but I still started to implement the storage support to experiement with it as well.

Now that this is pushed, I'm trying to implement a working storage plugin with dependencies that are not in the main project. So far I'm having the same issue as we expected:

SPAN_STORAGE_TYPE=plugin PLUGINS_DIRECTORY="/Users/ledor473/go/src/github.com/ledor473/jaeger-storage-dynamodb" LOG_LEVEL=debug go run cmd/collector/main.go -h

{"level":"info","ts":1543646585.109542,"caller":"pluginloader/factory.go:122","msg":"Collector plugin found","path":"/Users/ledor473/go/src/github.com/ledor473/jaeger-storage-dynamodb/jaeger-storage-dynamodb.so"}
panic: /debug/requests is already registered. You may have two independent copies of golang.org/x/net/trace in your binary, trying to maintain separate state. This may involve a vendored copy of golang.org/x/net/trace.

goroutine 1 [running]:
github.com/ledor473/jaeger-storage-dynamodb/vendor/golang.org/x/net/trace.init.0()
	/Users/ledor473/go/src/github.com/ledor473/jaeger-storage-dynamodb/vendor/golang.org/x/net/trace/trace.go:116 +0x16f
plugin.open(0xc0002ee300, 0x5a, 0xbef8a93e46877a70, 0x27b793c, 0x5535a00)
	/Users/ledor473/bin/go/src/plugin/plugin_dlopen.go:103 +0xabf
plugin.Open(0xc0002ee300, 0x5d, 0x0, 0x0, 0x0)
	/Users/ledor473/bin/go/src/plugin/plugin.go:32 +0x35
github.com/jaegertracing/jaeger/pkg/pluginloader.(*factory).loadPlugin(0xc00023c000, 0xc0002ee300, 0x5d, 0xc0001e0000, 0x1)
	/Users/ledor473/go/src/github.com/jaegertracing/jaeger/pkg/pluginloader/factory.go:129 +0x4d
github.com/jaegertracing/jaeger/pkg/pluginloader.(*factory).walkpluginDir(0xc00023c000, 0xc0002ee300, 0x5d, 0x4d20060, 0xc000350820, 0x0, 0x0, 0x202, 0xffffffffffffffff)
	/Users/ledor473/go/src/github.com/jaegertracing/jaeger/pkg/pluginloader/factory.go:123 +0x2b6
github.com/jaegertracing/jaeger/pkg/pluginloader.(*factory).walkpluginDir-fm(0xc0002ee300, 0x5d, 0x4d20060, 0xc000350820, 0x0, 0x0, 0x1a, 0xc000627bd8)
	/Users/ledor473/go/src/github.com/jaegertracing/jaeger/pkg/pluginloader/factory.go:51 +0x69
path/filepath.walk(0xc0002ee300, 0x5d, 0x4d20060, 0xc000350820, 0xc000627d88, 0x0, 0x0)
	/Users/ledor473/bin/go/src/path/filepath/path.go:358 +0x41c
path/filepath.walk(0xc000040012, 0x42, 0x4d20060, 0xc0003500d0, 0xc000627d88, 0x0, 0xc00023c000)
	/Users/ledor473/bin/go/src/path/filepath/path.go:382 +0x2fe
path/filepath.Walk(0xc000040012, 0x42, 0xc000627d88, 0x0, 0x0)
	/Users/ledor473/bin/go/src/path/filepath/path.go:404 +0x105
github.com/jaegertracing/jaeger/pkg/pluginloader.(*factory).Load(0xc00023c000, 0x42, 0xc0000a27e0)
	/Users/ledor473/go/src/github.com/jaegertracing/jaeger/pkg/pluginloader/factory.go:51 +0x65
main.main()
	/Users/ledor473/go/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:66 +0x16f
exit status 2

I'll try to workaround it by compiling both in the same Docker container and report back as soon I have have something working or if I run out of idea to make it work 👍

@ledor473
Copy link
Member Author

ledor473 commented Dec 4, 2018

So I was able to make it run. The plugin doesn't do much right now so it's 100% proven but I still wanted to update the PR with more information of what the build looks like.

All the commands will be run inside a Docker, here's my exact run command:

docker run \
-v $GOPATH/src/github.com/ledor473:/go/src/github.com/ledor473 \
-v $GOPATH/src/github.com/jaegertracing:/go/src/github.com/jaegertracing \
-it golang:alpine sh

Some generic installations/variables to make the build work:

apk add --update make git gcc build-base
export GOPATH=/go
export GOOS=linux 

Build Jaeger as we already do:

cd /go/src/github.com/jaegertracing/jaeger/
make install 
make build-collector

Now this gets trickier:

# Install plugins deps in vendor
cd /go/src/github.com/ledor473/jaeger-storage-dynamodb/
glide install

# Go back in Jaeger repo
cd /go/github.com/jaegertracing/jaeger/

# Copy all the vendors from the plugins into Jaeger's vendor (could be an rsync or something that just copy the difference from one side)
cp -r /go/src/github.com/ledor473/jaeger-storage-dynamodb/vendor/ ./

# We also have to copy all Go files of the plugins, but for now it's only 1 file
cp /go/src/github.com/ledor473/jaeger-storage-dynamodb/storage_factory.go ./jaeger-storage-dynamodb.go

# Compile the plugin into a .so file
go build -buildmode=plugin ./jaeger-storage-dynamodb.go

With that done, you can run it as follow:

SPAN_STORAGE_TYPE=plugin PLUGINS_DIRECTORY=`pwd` LOG_LEVEL=debug ./cmd/collector/collector-linux -h
{"level":"info","ts":1543955396.7755852,"caller":"pluginloader/factory.go:122","msg":"Collector plugin found","path":"/go/src/github.com/jaegertracing/jaeger/jaeger-storage-dynamodb.so"}
Jaeger collector receives traces from Jaeger agents and runs them through a processing pipeline.

Usage:
  jaeger-collector [flags]
  jaeger-collector [command]

Available Commands:
...

@yurishkuro
Copy link
Member

yurishkuro commented Dec 5, 2018

Have you tried pulling Jaeger as a vendored dependency of the plugin repo? Because all these machinations remind me of the classic transitive vendor problem caused by relying on a dependency from GOPATH that has its own vendor dir. In our internal builds at Uber we always build against an empty GOPATH, to avoid these issues and have the dep manager fully responsible for deps.

@olivierboucher
Copy link

I am kin to help with this since the gRPC plugin version has shown to have a huge performance impact

@ledor473
Copy link
Member Author

@olivierboucher even though I've worked on this, I think I prefer the gRPC approach anyway.

Seems like the recent results you posted are promising too!

@ledor473
Copy link
Member Author

So I've tried it a bit more and it seems that Go Modules (added in Go 1.11) helped with the issue I've mentioned above. Thanks to this comment golang/go#28983 (comment), I was able to make it run successfully!!

Here's the recipe so far:

docker run \
-v $GOPATH/src/github.com/ledor473:/go/src/github.com/ledor473 \
-v $GOPATH/src/github.com/jaegertracing:/go/src/github.com/jaegertracing \
-it golang:alpine sh

apk add --update make git gcc build-base
export GOPATH=/go
export GOOS=linux 
export GO111MODULE=on

cd /go/src/github.com/jaegertracing/jaeger/
go build -buildmode=plugin jaeger-storage-dynamodb

SPAN_STORAGE_TYPE=plugin PLUGINS_DIRECTORY=`pwd` LOG_LEVEL=debug go run cmd/collector/main.go -h

Which produce this:

{"level":"info","ts":1547857495.5187895,"caller":"pluginloader/factory.go:122","msg":"Collector plugin found","path":"/go/src/github.com/jaegertracing/jaeger/jaeger-storage-dynamodb.so"}
{"level":"info","ts":1547857505.2657535,"caller":"pluginloader/factory.go:73","msg":"Plugin was successfully loaded","plugin":"jaeger-storage-dynamodb.so","symbol":"Configurable"}
Jaeger collector receives traces from Jaeger agents and runs them through a processing pipeline.

Usage:
  jaeger-collector [flags]
...

Note: I run it in Docker to ensure that there's no "hack" on my computer and that it's a repeatable process.

Louis-Etienne Dorval added 5 commits January 21, 2019 10:01
Signed-off-by: Louis-Etienne Dorval <louis-etienne.dorval@ticketmaster.com>
…t of the Collector code

Signed-off-by: Louis-Etienne Dorval <louis-etienne.dorval@ticketmaster.com>
Signed-off-by: Louis-Etienne Dorval <louis-etienne.dorval@ticketmaster.com>
…ush metrics

Signed-off-by: Louis-Etienne Dorval <louis-etienne.dorval@ticketmaster.com>
@ledor473
Copy link
Member Author

@yurishkuro So this is looking pretty good!
In order to try it out, I've copy-pasted the unmerged Badger implementation (from @burmanm) and turned it into a plugin: https://github.com/ledor473/jaeger-storage-badger

The only real modification (except from moving to the main package and changing the repo) was to add this file to properly export the plugin: https://github.com/ledor473/jaeger-storage-badger/blob/master/plugin.go

With the same recipe as the last comment, I now get the following help:

$ SPAN_STORAGE_TYPE=plugin PLUGINS_DIRECTORY="`pwd`" LOG_LEVEL=debug go run cmd/collector/main.go -h
{"level":"info","ts":1548087122.6742558,"caller":"pluginloader/factory.go:122","msg":"Collector plugin found","path":"/Users/ledor473/go/src/github.com/jaegertracing/jaeger/jaeger-storage-badger.so"}
{"level":"info","ts":1548087122.751847,"caller":"pluginloader/factory.go:73","msg":"Plugin was successfully loaded","plugin":"jaeger-storage-badger.so","symbol":"Configurable"}
Jaeger collector receives traces from Jaeger agents and runs them through a processing pipeline.

....

Flags:
      --badger.consistency                     If all writes should be synced immediately. This will greatly reduce write performance.
      --badger.directory-key string            Path to store the keys (indexes), this directory should reside in SSD disk. Set ephemeral to false if you want to define this setting. (default "/var/folders/vs/znnjv22531n2cn1gs4txh4nc0000gn/T/go-build211534391/b001/exe/data/keys")
      --badger.directory-value string          Path to store the values (spans). Set ephemeral to false if you want to define this setting. (default "/var/folders/vs/znnjv22531n2cn1gs4txh4nc0000gn/T/go-build211534391/b001/exe/data/values")
      --badger.ephemeral                       Mark this storage ephemeral, data is stored in tmpfs. (default true)
      --badger.maintenance-interval duration   How often the maintenance thread for values is ran. Format is time.Duration (https://golang.org/pkg/time/#Duration) (default 5m0s)
      --badger.span-store-ttl duration         How long to store the data. Format is time.Duration (https://golang.org/pkg/time/#Duration) (default 72h0m0s)
      --collector.grpc-port int                (experimental) The gRPC port for the collector service (default 14250)
      --collector.http-port int                The HTTP port for the collector service (default 14268)
      --collector.num-workers int              The number of workers pulling items from the queue (default 50)
      --collector.plugins-directory string     The directory used to dynamically load collector plugins
      --collector.port int                     The TChannel port for the collector service (default 14267)
      --collector.queue-size int               The queue size of the collector (default 2000)
      --collector.zipkin.http-port int         The HTTP port for the Zipkin collector service e.g. 9411
      --config-file string                     Configuration file in JSON, TOML, YAML, HCL, or Java properties formats (default none). See spf13/viper for precedence.
      --health-check-http-port int             The http port for the health check service (default 14269)
  -h, --help                                   help for jaeger-collector
      --log-level string                       Minimal allowed log Level. For more levels see https://github.com/uber-go/zap (default "info")
      --metrics-backend string                 Defines which metrics backend to use for metrics reporting: expvar, prometheus, none (default "prometheus")
      --metrics-http-route string              Defines the route of HTTP endpoint for metrics backends that support scraping (default "/metrics")
      --sampling.strategies-file string        The path for the sampling strategies file in JSON format. See sampling documentation to see format of the file
      --span-storage.type string               Deprecated; please use SPAN_STORAGE_TYPE environment variable. Run this binary with "env" command for help.

And it's not just the help that is working:

SPAN_STORAGE_TYPE=plugin PLUGINS_DIRECTORY="`pwd`" go run cmd/collector/main.go
{"level":"info","ts":1548087222.572639,"caller":"pluginloader/factory.go:122","msg":"Collector plugin found","path":"/Users/ledor473/go/src/github.com/jaegertracing/jaeger/jaeger-storage-badger.so"}
{"level":"info","ts":1548087222.646497,"caller":"pluginloader/factory.go:73","msg":"Plugin was successfully loaded","plugin":"jaeger-storage-badger.so","symbol":"Configurable"}
{"level":"info","ts":1548087222.647129,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
{"level":"info","ts":1548087222.647357,"caller":"pluginloader/factory.go:73","msg":"Plugin was successfully loaded","plugin":"jaeger-storage-badger.so","symbol":"Configurable"}
{"level":"info","ts":1548087222.647667,"caller":"pluginloader/factory.go:73","msg":"Plugin was successfully loaded","plugin":"jaeger-storage-badger.so","symbol":"StorageFactory"}
2019/01/21 11:13:42 Replaying from value pointer: {Fid:0 Len:0 Offset:0}
2019/01/21 11:13:42 Iterating file id: 0
2019/01/21 11:13:42 Iteration took: 16.367µs
{"level":"info","ts":1548087222.72284,"caller":"jaeger-storage-badger/factory.go:120","msg":"Badger storage configuration","configuration":{"Dir":"/var/folders/vs/znnjv22531n2cn1gs4txh4nc0000gn/T/badger011673015","ValueDir":"/var/folders/vs/znnjv22531n2cn1gs4txh4nc0000gn/T/badger011673015","SyncWrites":false,"TableLoadingMode":1,"ValueLogLoadingMode":2,"NumVersionsToKeep":1,"MaxTableSize":67108864,"LevelSizeMultiplier":10,"MaxLevels":7,"ValueThreshold":32,"NumMemtables":5,"NumLevelZeroTables":5,"NumLevelZeroTablesStall":10,"LevelOneSize":268435456,"ValueLogFileSize":1073741823,"ValueLogMaxEntries":1000000,"NumCompactors":3,"DoNotCompact":false,"ReadOnly":false,"Truncate":false}}
{"level":"info","ts":1548087222.723698,"caller":"static/strategy_store.go:79","msg":"No sampling strategies provided, using defaults"}
{"level":"info","ts":1548087222.723899,"caller":"collector/main.go:157","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1548087222.723972,"caller":"grpcserver/grpc_server.go:64","msg":"Starting jaeger-collector gRPC server","grpc-port":"14250"}
{"level":"info","ts":1548087222.724033,"caller":"collector/main.go:171","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
{"level":"info","ts":1548087222.724065,"caller":"collector/main.go:180","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1548087222.7240808,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}

@ledor473
Copy link
Member Author

Closing as gRPC plugin is merged now

@ledor473 ledor473 closed this Sep 17, 2020
@yurishkuro yurishkuro changed the title WIP: Add plugins support to the Collector WIP: Add native plugins support to the Collector Sep 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to support plugins
4 participants