Manually created baggage fields do not propagate across threads using ObservationAwareSpanThreadLocalAccessor

[wapkch]

A sample is like this, the value of baggage field 'tenant' is expected to propagated correctly. But it's null now.

management.tracing.baggage.remote-fields=tenant

@SpringBootApplication
public class MicrometerTracingDemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(MicrometerTracingDemoApplication.class, args);
    }

    @Bean
    CommandLineRunner commandLineRunner(ObservationRegistry observationRegistry, Tracer tracer) {
        return args -> {
            ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
            executor.setTaskDecorator(runnable ->
                ContextSnapshotFactory.builder().build().captureAll().wrap(runnable));
            executor.initialize();

            ScopedSpan scopedSpan = tracer.startScopedSpan("test");
            try (BaggageInScope baggageInScope = tracer.getBaggage("tenant").makeCurrent("tenant")) {
                executor.submit(() -> {
                    String tenant = tracer.getBaggage("tenant").get();
                    // tenant should not be null
                    System.out.println(tenant);
                });
            } finally {
                scopedSpan.end();
            }
        };
    }

}

@Component
class Registrar implements SmartInitializingSingleton {

    @Autowired
    ObservationRegistry observationRegistry;

    @Autowired
    Tracer tracer;

    @Override
    public void afterSingletonsInstantiated() {
        ContextRegistry.getInstance().registerThreadLocalAccessor(
            new ObservationAwareSpanThreadLocalAccessor(observationRegistry, tracer));
    }
}

The sample can be found here https://github.com/wapkch/micrometer-tracing-demo/tree/master

[krodyrobi]

From playing with the library I noticed that:

• spans get recreated even if Hooks.enableContextPropagation is used and executors are instrumented
• since custom baggages are manually created they won't be refreshed or created when thread switching

All of this comes down to a subtle difference in what is available by default in each context.
Controllers/main schedulers wrap code with Observation.start(..).observe(..) implicitly for us and that without an active observation nothing gets propagated (for instance tests / command line runners don't have this - can check if observationRegistry has a currentObservation).

Tracing ids take the lack of an observation in context as a sign to recreate themselves, while custom baggages can't possibly be re-set as code creating them is not apart of the same library flow.

If you by chance wrap the submit part in the observe scoped lambda do you still see them not propagating?

[wapkch]

Yes, using Observation.start(..).observe(..) and explicitly passing a ReceiverContext could do the trick:

ReceiverContext<Map<String, String>> context = new ReceiverContext<>(Map::get);
            context.setCarrier(Map.of("tenant", "tenant"));
            Observation.createNotStarted("test", () -> context, observationRegistry).observe(() -> {
                executor.submit(() -> {
                    String tenant = tracer.getBaggage("tenant").get();
                    // tenant should not be null
                    System.out.println(tenant);
                });
            });

Because ReceiverContext activates the PropagatingReceiverTracingObservationHandler which propagates the custom baggages.

But in this way, we don't even need a ObservationAwareSpanThreadLocalAccessor.

So, i still think ObservationAwareSpanThreadLocalAccessor should work with the tracer.startScopedSpan(...) style.

[wapkch]

After taking a look into it, i found that when invoking captureFromThreadLocals, the tenant baggage value in the span is exist:

But when invoking setThreadLocals subsequencely, the tenant baggage value is missing in the restored span:

I'm not sure why now.