Intermittent Sleuth failures on GraphQL service

[erdemedeiros]

We can see the following stack trace when the GraphQL service is running:

Exception in thread "clientInboundChannel-1" org.springframework.messaging.MessageDeliveryException: Failed to handle GenericMessage [payload=GraphQLMessage [payload={query=subscription($appName: String!, $eventTypes: [EngineEventType!]) {   engineEvents(appName: [$appName], eventType: $eventTypes) {     processInstanceId     processDefinitionId     entity     eventType   } }, variables={appName=default-app, eventTypes=[MESSAGE_SENT, MESSAGE_WAITING, MESSAGE_RECEIVED]}}, id=1, type=start], headers={simpMessageType=MESSAGE, nativeHeaders={}, simpSessionAttributes={HTTP.SESSION.ID=C840AA5029CCCD42F5B0496CFF2D9293, SPRING_SECURITY_CONTEXT=SecurityContextImpl [Authentication=KeycloakAuthenticationToken [Principal=5f682999-d11d-4a42-bc42-86b7e675bb76, Credentials=[PROTECTED], Authenticated=true, Details=org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@7c857768, Granted Authorities=[ROLE_ACTIVITI_ADMIN, ROLE_offline_access, ROLE_ACTIVITI_USER, ROLE_uma_authorization]]]}, simpUser=KeycloakAuthenticationToken [Principal=5f682999-d11d-4a42-bc42-86b7e675bb76, Credentials=[PROTECTED], Authenticated=true, Details=org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@72429688, Granted Authorities=[ROLE_ACTIVITI_ADMIN, ROLE_offline_access, ROLE_ACTIVITI_USER, ROLE_uma_authorization]], graphQLMessageType=start, simpSessionId=d9de796c-2134-150d-1635-65694c6019b5, simpDestination=/ws/graphql}] to org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask@767d1c18 in org.activiti.cloud.services.notifications.graphql.ws.transport.GraphQLBrokerMessageHandler@1d215a52; nested exception is java.lang.IllegalStateException: Already immutable, failedMessage=GenericMessage [payload=GraphQLMessage [payload={query=subscription($appName: String!, $eventTypes: [EngineEventType!]) {   engineEvents(appName: [$appName], eventType: $eventTypes) {     processInstanceId     processDefinitionId     entity     eventType   } }, variables={appName=default-app, eventTypes=[MESSAGE_SENT, MESSAGE_WAITING, MESSAGE_RECEIVED]}}, id=1, type=start], headers={simpMessageType=MESSAGE, nativeHeaders={}, simpSessionAttributes={HTTP.SESSION.ID=C840AA5029CCCD42F5B0496CFF2D9293, SPRING_SECURITY_CONTEXT=SecurityContextImpl [Authentication=KeycloakAuthenticationToken [Principal=5f682999-d11d-4a42-bc42-86b7e675bb76, Credentials=[PROTECTED], Authenticated=true, Details=org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@7c857768, Granted Authorities=[ROLE_ACTIVITI_ADMIN, ROLE_offline_access, ROLE_ACTIVITI_USER, ROLE_uma_authorization]]]}, simpUser=KeycloakAuthenticationToken [Principal=5f682999-d11d-4a42-bc42-86b7e675bb76, Credentials=[PROTECTED], Authenticated=true, Details=org.keycloak.adapters.springsecurity.account.SimpleKeycloakAccount@72429688, Granted Authorities=[ROLE_ACTIVITI_ADMIN, ROLE_offline_access, ROLE_ACTIVITI_USER, ROLE_uma_authorization]], graphQLMessageType=start, simpSessionId=d9de796c-2134-150d-1635-65694c6019b5, simpDestination=/ws/graphql}]
	at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:153)
	at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:63)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: Already immutable
	at org.springframework.util.Assert.state(Assert.java:76)
	at org.springframework.messaging.support.NativeMessageHeaderAccessor.removeNativeHeader(NativeMessageHeaderAccessor.java:277)
	at org.springframework.cloud.sleuth.instrument.messaging.MessageHeaderPropagatorSetter.removeAnyTraceHeaders(MessageHeaderPropagatorSetter.java:62)
	at org.springframework.cloud.sleuth.instrument.messaging.TracingChannelInterceptor.beforeHandle(TracingChannelInterceptor.java:324)
	at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.applyBeforeHandle(ExecutorSubscribableChannel.java:167)
	at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:140)
	... 4 more

ActivitiGraphQLStarterIT also fails intermittently and when this happens we can see the stack trace above.