Headers not always propagated to MDC (with sample)

[anderius]

I want to read header values, and have them available on MDC. One single request works, but not when a test does many requests after each other.

Complete project that reproduces the error every time (on my computer, Java 17). Most often after only 6-7 iterations:

https://github.com/anderius/spring-sleuth-problem-demo

Excerpt from the sample:

@RestController
@SpringBootApplication
class DemoApplication {

    @Bean
    fun sleuthHttpServerRequestParser() = HttpRequestParser { request, context, _ ->
        request.header("customHeader")
            ?.let { BaggageField.create("customField").updateValue(context, it) }
    }

    @Bean
    fun spanHandler(): SpanHandler = SpanHandler.NOOP

    @GetMapping("/mdc")
    fun getMdc(): Map<String, String> = MDC.getCopyOfContextMap()
}

This issue might be related to #2064.

[marcingrzejszczak]

Can you please rewrite this to java?

[anderius]

Sure, tomorrow. :-) I use the Spring Start project to generate the project. Any preference for Maven/Gradle, or Java version?

[marcingrzejszczak]

Maven will be great!

[anderius]

@marcingrzejszczak Maven and Java is ready! :-)

Project is same place: https://github.com/anderius/spring-sleuth-problem-demo

The setup in java:

@RestController
@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }

    @Bean
    HttpRequestParser sleuthHttpServerRequestParser() {
        return (request, context, span) -> {
            String headerValue = request.header("customHeader");
            if (headerValue != null) {
                BaggageField.create("customField").updateValue(context, headerValue);
            }
        };
    }

    @Bean
    SpanHandler spanHandler() {
        return SpanHandler.NOOP;
    }

    @GetMapping("/mdc")
    Map<String, String> getMdc() {
        return MDC.getCopyOfContextMap();
    }
}

[marcingrzejszczak]

It took me a while but I understand why it's not working. So the default sampling strategy is rate based. You're sending requests to fast and after ~8 request sampling is noop so no baggage is set. The tests pass once you set spring.sleuth.sampler.probability=1.0. Then you sample all of them.

[anderius]

@marcingrzejszczak Thank you for looking into this!

That means Spring Cloud Sleuth should not be used for any headers other than the ones used for sampling/tracing, unless we sample everything.

I had the idea the sampling was only how much was sent to zipkin (or similar), but it is also how much is instrumentated.

[marcingrzejszczak]

That's how Brave works, I'd need to double check if OTel works in the same way.

[anderius]

It would be nice to be able to instrument/populate every request onto MDC, and still only send samples.

I will also have to check if this sampling affects propagated headers, as I was expecting that to happen every time as well.

[marcingrzejszczak]

Can you change your code and check if this helps. You can leave the sampling condition as it is but instead of

 @Bean
    SpanHandler spanHandler() {
        return SpanHandler.NOOP;
    }

just create a new instance of a SpanHandler that does nothing (but don't call the NOOP reference).

[anderius]

Same result, I am afraid.

[marcingrzejszczak]

Meh. So it has to be sampled