Sleuth giving different traceIds for each RxJava's Single execution

[maradanasai]

Spring sleuth is giving different traceIds for each execution call with Single#fromSupplier(). Usually these are all should be under same traceId which is not happening here. I have tried creating a scoped span but didn't worked out. Here is the sample reproducible code

Single<Object> externalIOBoundApiCallOne = Single.fromSupplier(
						() -> {
							Span span = tracer.nextSpan();
							try (Tracer.SpanInScope ws = tracer.withSpanInScope(span.start())) {
								Future<Object> obj =
										appExecutorService.submit(() -> {
											return restTemplate.getForObject("https://random-data-api.com/api/address/random_address", Object.class);
										});
								return obj.get();
							}
						}
				)
				.timeout(5000, TimeUnit.MILLISECONDS)
				.subscribeOn(Schedulers.io())
				.onErrorReturn(t -> appExecutorService.submit(() -> "ERROR"));

		Single<Object> externalIOBoundApiCallTwo = Single.fromSupplier(
						() -> {
							Span span = tracer.nextSpan();
							try (Tracer.SpanInScope ws = tracer.withSpanInScope(span.start())) {
								Future<Object> obj =
										appExecutorService.submit(() -> {
											restTemplate.getForObject("https://random-data-api.com/api/app/random_app", Object.class);
											if (!Thread.currentThread().getName().isEmpty())
												throw new RuntimeException("Error while doing http call");
											return null;
										});
								return obj.get();
							}
						}
				)
				.timeout(5000, TimeUnit.MILLISECONDS)
				.subscribeOn(Schedulers.io())
				.onErrorReturn(t -> appExecutorService.submit(() -> "ERROR"));

Logs:

2023-12-04 17:14:50.135  INFO [rxjava-and-spring-sleuth,,] 21914 --- [           main] c.e.r.RxjavaAndSpringSleuthApplication   : Started RxjavaAndSpringSleuthApplication in 2.485 seconds (JVM running for 2.759)
2023-12-04 17:14:50.339  INFO [rxjava-and-spring-sleuth,5da4526dc52f3654,5da4526dc52f3654] 21914 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-12-04 17:14:50.339  INFO [rxjava-and-spring-sleuth,5da4526dc52f3654,5da4526dc52f3654] 21914 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-12-04 17:14:50.340  INFO [rxjava-and-spring-sleuth,5da4526dc52f3654,5da4526dc52f3654] 21914 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
2023-12-04 17:14:50.391  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Request Start
2023-12-04 17:14:50.391  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Request Start
2023-12-04 17:14:50.391  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : URI         : https://random-data-api.com/api/address/random_address
2023-12-04 17:14:50.391  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : URI         : https://random-data-api.com/api/app/random_app
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Method      : GET
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Method      : GET
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Headers     : [Accept:"application/json, application/*+json", Content-Length:"0", X-B3-TraceId:"af855cc1c07de96c", X-B3-SpanId:"c67bdecfa53bc526", X-B3-ParentSpanId:"47a65fa33eadbd5e", X-B3-Sampled:"1"]
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Headers     : [Accept:"application/json, application/*+json", Content-Length:"0", X-B3-TraceId:"8e9d1c9fead65b58", X-B3-SpanId:"2d86758d9754864a", X-B3-ParentSpanId:"c399e24a1765ef98", X-B3-Sampled:"1"]
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Request body: 
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Request body: 
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Request End
2023-12-04 17:14:50.392  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Request End
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Response Start
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Response Start
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Status code  : 200 OK
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Status code  : 200 OK
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Status text  : OK
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Status text  : OK
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Headers      : [Server:"nginx/1.18.0 (Ubuntu)", Date:"Mon, 04 Dec 2023 11:44:52 GMT", Content-Type:"application/json; charset=utf-8", Transfer-Encoding:"chunked", Connection:"keep-alive", X-Frame-Options:"SAMEORIGIN", X-XSS-Protection:"1; mode=block", X-Content-Type-Options:"nosniff", X-Download-Options:"noopen", X-Permitted-Cross-Domain-Policies:"none", Referrer-Policy:"strict-origin-when-cross-origin", ETag:"W/"4c899233d87069c8f80da79c13c4dcff"", Cache-Control:"max-age=0, private, must-revalidate", X-Request-Id:"af0367ee-5204-452b-aea9-b49ae40b9c07", X-Runtime:"0.009164", Strict-Transport-Security:"max-age=31536000; includeSubDomains", "max-age=15768000", Vary:"Origin"]
2023-12-04 17:14:52.091  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Headers      : [Server:"nginx/1.18.0 (Ubuntu)", Date:"Mon, 04 Dec 2023 11:44:52 GMT", Content-Type:"application/json; charset=utf-8", Transfer-Encoding:"chunked", Connection:"keep-alive", X-Frame-Options:"SAMEORIGIN", X-XSS-Protection:"1; mode=block", X-Content-Type-Options:"nosniff", X-Download-Options:"noopen", X-Permitted-Cross-Domain-Policies:"none", Referrer-Policy:"strict-origin-when-cross-origin", ETag:"W/"f5c223dbc7c6afb3507609a83c0adbac"", Cache-Control:"max-age=0, private, must-revalidate", X-Request-Id:"78e57367-e48b-4958-94e2-b6e186bcc0ab", X-Runtime:"0.012449", Strict-Transport-Security:"max-age=31536000; includeSubDomains", "max-age=15768000", Vary:"Origin"]
2023-12-04 17:14:52.092  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Response body: {"id":4876,"uid":"0c26addb-cb29-4279-b4dd-e1fb7cfc061a","city":"Kimberliefort","street_name":"Toy Valley","street_address":"46585 Valentine Shoal","secondary_address":"Suite 519","building_number":"9927","mail_box":"PO Box 9180","community":"Park Court","zip_code":"35324-9167","zip":"71882","postcode":"80285-5638","time_zone":"Europe/Copenhagen","street_suffix":"Squares","city_suffix":"port","city_prefix":"Port","state":"Minnesota","state_abbr":"VT","country":"Kazakhstan","country_code":"SO","latitude":-45.18655087019896,"longitude":85.33671235235875,"full_address":"802 Jon Run, West Marshaburgh, WY 37769-2197"}
2023-12-04 17:14:52.092  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Response body: {"id":3758,"uid":"1235740f-e330-4098-92de-f491e80b596f","app_name":"Konklux","app_version":"3.1.9","app_author":"Kam Dooley Ret.","app_semantic_version":"3.7.8","app_major_version":"3.1.2","app_minor_version":"9.109.5","app_patch_version":"7.1.6"}
2023-12-04 17:14:52.092  INFO [rxjava-and-spring-sleuth,af855cc1c07de96c,c67bdecfa53bc526] 21914 --- [pool-3-thread-1] c.e.r.RequestResponseLoggingInterceptor  : Response End
2023-12-04 17:14:52.092  INFO [rxjava-and-spring-sleuth,8e9d1c9fead65b58,2d86758d9754864a] 21914 --- [pool-3-thread-2] c.e.r.RequestResponseLoggingInterceptor  : Response End

Attaching here complete source code. Please assist on this.

Using
Java8
Sleuth version: 3.1.2
Spring cloud: 2021.0.2
Spring boot: 2.6.0

cc: @marcingrzejszczak

[marcingrzejszczak]

Spring Cloud Sleuth is out of oss support (https://spring.io/projects/spring-cloud-sleuth#support).

[maradanasai]

@marcingrzejszczak I see. But can you suggest a work around or fix? would appreciate if you can help here, thanks!

[marcingrzejszczak]

The only thing that comes to my mind is that you would need to wrap a Scheduler that it stores a previous Span and continues it when the signal is processed, but I'm not an expert in RxJava by no means