Misaligned spans with Mongodb (includes sample)

[jiraguha]

Describe the bug

Using Spring Sleuth with Mongodb integration causes issues. We use project-reactor and the issue apprears when a service call another service using flatmap. Both services use mongo:

java.lang.AssertionError: Misalignment: scoped span NoopSpan(3a27c90bb910c469/eef6aa97485a9d38) !=  current span null
	at brave.propagation.ThreadLocalSpan.remove(ThreadLocalSpan.java:164) ~[brave-5.13.2.jar:na]
	at brave.mongodb.TraceMongoCommandListener.commandSucceeded(TraceMongoCommandListener.java:103) ~[brave-instrumentation-mongodb-5.13.2.jar:na]

Maybe the issue is somewhere deeper in the relation between the reactive context and mongo... Please, find a sample below.

Sample

see mongo-misaligned-colors-example
It contains:

• Unit tests that isolate the problem: NextSpan && service calling service and ColorControllerTest.find
• 2 branches; the main branch is buggy & with_sleuth_mongodb_disable branch where all unit tests pass
• The issue can be check quickly by run the app via GET localhost:8080/... You will need mongo running locally docker run -it -v mongodata:/data/db -p 27017:27017 --name mongodb -d mong

On the buggy branch, once the Misalignment occurs some other tests fail!

Note

It seems to be similar to #1349 or #1510

Spec

springCloud : 2020.0.1
springBoot: 2.4.3 with project-reactor
kotlin: 1.4.30
java: 15
``

[marcingrzejszczak]

Is the problem still there if you use the new DECORATE_QUEUES reactor instrumentation mode (spring.sleuth.reactor.instrumentation-type=DECORATE_QUEUES) ?

[jiraguha]

sorry but springCloud 2020.0.1 or spring-cloud-sleuth-autoconfigure:3.0.1 do not have InstrumentationType.DECORATE_QUEUES

[marcingrzejszczak]

You need to use the snapshots

[jiraguha]

which snapshots exactly?

[marcingrzejszczak]

Cloud 2020.0.2-SNAPSHOT or Sleuth 3.0.2-SNAPSHOT

[jiraguha]

Waw thanks! 2020.0.2-SNAPSHOT seems to solve the issue even without InstrumentationType.DECORATE_QUEUES... see mongo-misaligned-colors-example! Any explanation about what is happening would be welcome :-)...

[marcingrzejszczak]

I have no idea why it would work without the decorate queues option 🤷 We've updated reactor so maybe there was a bug there?

[jiraguha]

Never mind, we will just wait for 2020.0.2 release... Thx for your time