tracing: Add spans for outgoing calls made by the memcached client #3148
Conversation
igorwwwwwwwwwwwwwwwwwwww
force-pushed
the
trace-memcached
branch
3 times, most recently
from
c37837c
to
d16cdb8
Compare
Signed-off-by: Igor Wiedler <iwiedler@gitlab.com>
igorwwwwwwwwwwwwwwwwwwww
force-pushed
the
trace-memcached
branch
from
d16cdb8
to
f6473d4
Compare
|
Note that the upstream memcached client does not support opentracing, see bradfitz/gomemcache#84. |
pkg/cacheutil/memcached_client.go
Outdated
| @@ -340,11 +345,16 @@ func (c *memcachedClient) SetAsync(_ context.Context, key string, value []byte, | |||
| start := time.Now() | |||
| c.operations.WithLabelValues(opSet).Inc() | |||
|
|
|||
| span, _ := tracing.StartSpan(ctx, "memcached_client_set") | |||
There was a problem hiding this comment.
Note that by its async nature, this span will appear later in the trace, after its parent has already finished. AFAIK most tracing tools should handle this case ok.
There was a problem hiding this comment.
This looks solid. Thanks a lot for contributing.
I believe in order to truly track the performance issues with the cache, we need to make sure encapsulating actions also have necessary spans. AFAIK, we have spans for caching bucket however I'm not sure if we have spans already in place for index-caching. It'd be great if you could make sure we don't miss any of the spans for both index-cache and caching bucket.
WDYT?
|
I recall the traces being unreadable with thousands on requests against Memcached for single request, but we can enable this if needed - maybe worth to enable when some flag is specified? Otherwise agree with @kakkoyun let's try to unify this for all kind of caching requests 🤔 |
Yes, this was a deal breaker for us (Cortex). Maybe we could add an option to enable these spans, so that we can keep it disabled in Cortex? |
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
igorwwwwwwwwwwwwwwwwwwww
force-pushed
the
trace-memcached
branch
from
4580ee0
to
fd22a8c
Compare
|
@bwplotka I've brought this up-to-date with |
Signed-off-by: Igor Wiedler <iwiedler@gitlab.com>
igorwwwwwwwwwwwwwwwwwwww
force-pushed
the
trace-memcached
branch
from
fd22a8c
to
85871b2
Compare
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Changes
This adds tracing instrumentation to the memcached client. That should help diagnose issues with the cache, be it latency or cache effectiveness.
Verification
Still need to figure out a good way to test this.