Skip to content
This repository has been archived by the owner on Oct 3, 2023. It is now read-only.

Handle the case where grpc client connections are shared #298

Merged
merged 1 commit into from
Oct 29, 2021

Conversation

dashpole
Copy link
Contributor

@dashpole dashpole commented Oct 29, 2021

Follow-up to #295. If using an Insecure client with the OpenTelemetry collector, it will use the WithGRPCConn option. This grpc connection is then used for both the trace and stats clients. When closing the exporter, the second Close() call fails with error(s) closing trace client (<nil>), or metrics client (rpc error: code = Canceled desc = grpc: the client connection is closing), which indicates the connection is already closed. Handle this case by ignoring that error.

This is required to fix unit test errors in open-telemetry/opentelemetry-collector-contrib#5990.

The go.sum changes must be left-over from a previous PR that didn't tidy?

Copy link
Contributor

@punya punya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test for this behavior?

@@ -439,6 +441,12 @@ func (e *Exporter) StopMetricsExporter() {
func (e *Exporter) Close() error {
tErr := e.traceExporter.close()
mErr := e.statsExporter.close()
// If the trace and stats exporter share client connections,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a reasonable local fix, but should the client connection be tracked better? E.g. should there be a reference count?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In practice, this case shouldn't ever really happen, since you normally don't want to dial the same endpoint for metrics and traces. It just happens in unit tests for the collector, from what I can tell. If we did something like that, we would basically just want to only close one connection if the user provides the WithGRPCConn option, since they connections are different otherwise.

But it would have been a bit ugly to check and track the args that are currently just passed through to the monitoring and trace clients. The other option would've been to add a "metrics-only" exporter constructor function, but that also seemed like too much for fixing unit tests.

The other option would be for whoever creates the connection initially to be responsible for closing it. But in most cases, users don't create the connection themselves, and will still need to invoke Shutdown (in the collector) for other reasons.

The test fails without the change to the Close function.
@dashpole
Copy link
Contributor Author

Added a test

@dashpole dashpole merged commit dfce60d into census-ecosystem:master Oct 29, 2021
@dashpole dashpole deleted the fix_reused_client branch October 29, 2021 20:32
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants