Supports spans filtering before reporting

[quaff]

Requirement - what kind of business use case are you trying to solve?

Most short spans is verbose, Tracer should provide a option let application to decide which spans should be simply dropped, and those spans should never be dropped :

1. ctx.isDebug() return true
2. "error" tag is true
3. "sampling.priority" is present

Proposal - what do you suggest to solve the problem or improve the existing situation?

Sample Code:

    spanPredicate = span -> true;

    public Builder withSpanPredicate(Predicate<JaegerSpan> spanPredicate) {
        this.spanPredicate = spanPredicate;
        return this;
    }

  void reportSpan(JaegerSpan span) {
	if(!(span.context().isDebug()
			|| Boolean.TRUE.equals(span.getTags().get(Tags.ERROR.getKey())) 
			|| span.getTags().get(Tags.SAMPLING_PRIORITY.getKey()) != null 
			|| spanPredicate.test(span))) {
		return;
	}
    reporter.report(span);
    metrics.spansFinished.inc(1);
  }

Tracer tracer = builder.withSpanPredicate(span -> span.getDuration() > 5000).build();

I'm not sure the name "spanPredicate" is accurate.

[yurishkuro]

Spans cannot be skipped based on some random condition, because it will result in broken causality links
(child to parent) between them and the other spans that do get captured.

[quaff]

Most of the tiny spans are leafs, I hope it can be skipped if works as expected (fast and without error and debug) to save network traffic and keep trace more cleaner.

[quaff]

Skip base on duration, If parent is skipped then all of children are skipped, no orphan span survives, It's safe, am I right?

[yurishkuro]

The problem is that the way you propose to do it (with a predicate) does not take into account if those spans are leaves or not. So if instrumentation, say, has an internal span around some function, its duration very well may be short, but if it's skipped it will break the rest of the trace subtree.

I think span pruning is a useful feature, but we need to find a better way to implement it. I haven't put too much thought into it so far, but perhaps the following might work:

• spans for a given trace are not sent to the reporter until all of them are finished in the given process
• after all spans within the process are finished, a pruning step runs before sending them to the reporter
• after the pruning step, a certain rewrite logic is run to fix the references, i.e. if the trace had A<-B<-C (as in: parent<-child), and B was pruned, then C is rewritten to point to A as parent, A<-C, as well as indicate somewhere that B was pruned

I think DataDog's clients support something similar, maybe it's worth looking into how they are doing that.

[objectiser]

You would also need to know whether a span's context had been extracted, as an indication that it had been propagated to another service, and therefore the pruning should preserve that span.

[quaff]

@yurishkuro That's what I want, I think it's a little related to my previous #465 .