Add the ability to use the gRPC storage plugin with a remote endpoint

[cevian]

Requirement - what kind of business use case are you trying to solve?

Ability to have the gRPC storage API be implemented on a remote endpoint (not only locally over a socket).

I'm happy to submit a PR if the community is open to this.

Problem - what in Jaeger blocks you from solving the requirement?

Right now the gRPC storage plugin can only be used via the go-hashicorp plugin (over a socket). I'd like to expose the storage API on another machine, so having a way for the gRPC plugin to use a regular gRPC client instead would be much easier and also more efficient.

In the short term I've solved this with a gRPC plugin that simply proxies the request to a gRPC client with a remote endpoint so the communication is

jaeger --(go-hashicorp-plugin)-->jaeger-query-proxy--(gRPC)--> grpc Endpoint implementing the storage API.

But the jaeger-query-proxy implements the same exact API as the grpc Endpoint implementing the storage API. so it would be much more efficient to just have

jaeger --(gRPC)--> grpc Endpoint implementing the storage API.

Proposal - what do you suggest to solve the problem or improve the existing situation?

I think there would need to be some additional configuration options added to the grpc plugin (here):

• TLS bool yaml:"tls"
• CaFile string yaml:"cafile,omitempty"
• ServerAddr string yaml:"grpc-server"
• ServerHostOverride string yaml:"grpc-server-host-override,omitempty"
• ConnectTimeout time.Duration yaml:"connection-timeout,omitempty"

And then in the plugin Build function we'd need to initialize a regular gRPC client if using a remote connection (instead of plugin.NewClient from the hashicorp plugin).

Of course the existing hashicorp way would still work and not be effected.

I don't think this would be a big change and again I'd be happy to submit a PR.

Any open questions to address

I think in the choice between the plugin or the remote client would be done depending on whether the PluginBinary configuration option is an empty string or not.

I'm hoping that the hashicorp client and the gRPC client would provide an equivalent interface but I'd have to see during implementation.

[yurishkuro]

Yes, that would be awesome, we talked about it many times, because one thing this would allow us to do is to use in-memory storage as a remote storage, which would allow different deployments of collector/query instead of all-in-one.

[albertteoh]

I second that, this would be great! I think this would help with addressing #1221, which in turn addresses the recent flaky cross-dock CI jobs due to slow ES startup.

[yurishkuro]

Duplicate of #1519