Problems with Method DestinationAccessor.tryGetDestination and DwC

[davidfoerderer]

Describe the Bug

We are actually migrate some jobs to a new job lib. And now we've encountered a problem that we're currently stuck on. It concerns our DataCollection jobs. These jobs need to retrieve the destination via the Cloud SDK during execution. This worked perfectly until recently.
However, since the job change, we keep running into the issue that the DwCHeaders aren't being passed correctly when calling DestinationAccessor.tryGetDestination. We've tried several things to get the DwCHeaders included in the call, but unfortunately without success.

We hope you can help us with this? Do yo know this problem? Or is the a default way for it?

Steps to Reproduce

• A DataCollectionJob is executed.
• During the execution we use the Method 'DestinationAccessor.tryGetDestination'
• A BAD Request Exception from this method (see Log)

Expected Behavior

The method is working and gives us back a destination like before.

Screenshots

No response

Used Versions

• SAP Cloud SDK version: com.sap.cloud.sdk:sdk-core:jar:5.27.0
• Spring Boot or CAP version: com.sap.dwc:util-cap:jar:2.13.6

Code Examples

We could not paste SAP internal coding in a public github issue. If you need we could send these via mail or have a short Teams session.

Stack Trace

No response

Log File

Affected Development Phase

Development

Impact

Blocked

Timeline

No response

[Jonas-Isr]

Hi @davidfoerderer,

thanks for the additional information but this is unfortunately still not enough for a good assessment. As far as I understand, the DwC headers should be provided from the DwC environment and then picked up by the SDK automatically at runtime. Is this also what you expect to happen or are you trying to manually add/change the headers?

Please provide the full stack trace of the error you are seeing in the logs. This is important as otherwise we have no chance of knowing what is happening in your code.

Best,
Jonas

[davidfoerderer]

Hi @Jonas-Isr,

yes the dwc header should provided from the environment. But it seems that will not happen with the new job library. So we tried to insert the dwcHeader manuall with e.g. DwCContextWrapper. This is working until we will call the trygetDestination method. There it seems that the call is jumping to another thread and losing the dwcContext.

I will reproduce it again and add the stacktrace from the logfile to this issue.

Best regards,
David

[davidfoerderer]

Here is the Call-Stack:

com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to get destination.
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.resilientCall(DestinationService.java:416)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.loadAndParseAllDestinations(DestinationService.java:339)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationRetrievalStrategyResolver.lambda$prepareSupplierAllDestinations$9(DestinationRetrievalStrategyResolver.java:272)
	at io.vavr.control.Try.mapTry(Try.java:692)
	at io.vavr.control.Try.map(Try.java:634)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.getAllDestinationsByRetrievalStrategy(DestinationService.java:329)
	at com.sap.cloud.sdk.cloudplatform.connectivity.GetOrComputeAllDestinationsCommand.lambda$prepareCommand$0(GetOrComputeAllDestinationsCommand.java:40)
	at com.sap.cloud.sdk.cloudplatform.connectivity.GetOrComputeAllDestinationsCommand.execute(GetOrComputeAllDestinationsCommand.java:71)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService$Cache.getOrComputeAllDestinations(DestinationService.java:946)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.getAllDestinationProperties(DestinationService.java:257)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.validateDestinationLookup(DestinationService.java:153)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.tryGetDestination(DestinationService.java:129)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationLoaderChain.tryGetDestination(DestinationLoaderChain.java:85)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationLoader.tryGetDestination(DestinationLoader.java:33)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationAccessor.tryGetDestination(DestinationAccessor.java:120)
	at com.sap.crun.perfmon.scheduler.advancedJobs.PerfmonJobExecutor.lambda$executeSchedule$4(PerfmonJobExecutor.java:121)
	at com.sap.dwc.util.headers.DwcContextTaskDecorator.lambda$decorate$0(DwcContextTaskDecorator.java:33)
	at com.sap.crun.perfmon.scheduler.advancedJobs.PerfmonJobExecutor.lambda$executeSchedule$5(PerfmonJobExecutor.java:135)
	at com.sap.dwc.util.headers.selfinitiated.DwcContextWrapper.lambda$withDwcContextForTenantById$2(DwcContextWrapper.java:81)
	at com.sap.dwc.util.headers.selfinitiated.DwcContextWrapper.withDwcContextForTenantById(DwcContextWrapper.java:102)
	at com.sap.dwc.util.headers.selfinitiated.DwcContextWrapper.withDwcContextForTenantById(DwcContextWrapper.java:80)
	at com.sap.crun.perfmon.scheduler.advancedJobs.PerfmonJobExecutor.executeSchedule(PerfmonJobExecutor.java:115)
	at com.sap.calm.advancedscheduler.ExecuteScheduleWorker.executeTaskAndUpdateStatus(ExecuteScheduleWorker.java:219)
	at com.sap.calm.advancedscheduler.ExecuteScheduleWorker.run(ExecuteScheduleWorker.java:80)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.sap.cloud.sdk.cloudplatform.resilience.ResilienceRuntimeException: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to get destinations: destination service responded with HTTP status 400 (BAD REQUEST) at 'destination-configuration/v1/instanceDestinations'.
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$3(Resilience4jDecorationStrategy.java:194)
	at io.vavr.control.Try.onFailure(Try.java:718)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$4(Resilience4jDecorationStrategy.java:190)
	at io.vavr.control.Try.of(Try.java:81)
	at io.vavr.control.Try.ofCallable(Try.java:117)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateSupplier$1(Resilience4jDecorationStrategy.java:156)
	at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorationStrategy.executeSupplier(ResilienceDecorationStrategy.java:99)
	at com.sap.cloud.sdk.cloudplatform.resilience.ResilienceDecorator.executeSupplier(ResilienceDecorator.java:197)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.resilientCall(DestinationService.java:410)
	... 28 more
Caused by: com.sap.cloud.sdk.cloudplatform.thread.exception.ThreadContextExecutionException: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to get destinations: destination service responded with HTTP status 400 (BAD REQUEST) at 'destination-configuration/v1/instanceDestinations'.
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:242)
	at com.sap.cloud.sdk.cloudplatform.thread.DefaultThreadContextExecutorService.lambda$decorate$0(DefaultThreadContextExecutorService.java:68)
	at com.sap.cds.integration.cloudsdk.decorator.CdsThreadContextDecorator.lambda$decorateCallable$0(CdsThreadContextDecorator.java:37)
	at com.sap.cds.services.impl.runtime.RequestContextRunnerImpl.run(RequestContextRunnerImpl.java:306)
	at com.sap.cds.integration.cloudsdk.decorator.CdsThreadContextDecorator.lambda$decorateCallable$1(CdsThreadContextDecorator.java:34)
	... 4 more
Caused by: com.sap.cloud.sdk.cloudplatform.connectivity.exception.DestinationAccessException: Failed to get destinations: destination service responded with HTTP status 400 (BAD REQUEST) at 'destination-configuration/v1/instanceDestinations'.
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServiceAdapter$DestinationHttpClientResponseHandler.handleResponse(DestinationServiceAdapter.java:181)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServiceAdapter$DestinationHttpClientResponseHandler.handleResponse(DestinationServiceAdapter.java:142)
	at io.opentelemetry.javaagent.instrumentation.apachehttpclient.v5_0.WrappingStatusSettingResponseHandler.handleResponse(WrappingStatusSettingResponseHandler.java:44)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:247)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:162)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationServiceAdapter.getConfigurationAsJson(DestinationServiceAdapter.java:135)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.getAndDeserializeDestinations(DestinationService.java:376)
	at com.sap.cloud.sdk.cloudplatform.connectivity.DestinationService.lambda$loadAndParseAllDestinations$7(DestinationService.java:339)
	at com.sap.cloud.sdk.cloudplatform.resilience4j.Resilience4jDecorationStrategy.lambda$decorateCallable$2(Resilience4jDecorationStrategy.java:175)
	at io.github.resilience4j.bulkhead.Bulkhead.lambda$decorateCallable$5(Bulkhead.java:173)
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.call(ThreadContextExecutor.java:293)
	at com.sap.cloud.sdk.cloudplatform.thread.ThreadContextExecutor.execute(ThreadContextExecutor.java:236)
	... 8 more

[Jonas-Isr]

Thanks for the additional information. Some thoughts on this:

yes the dwc header should provided from the environment. But it seems that will not happen with the new job library.

The cleanest solution would probably be to find out why this is not working anymore for you? Maybe the DwC colleagues can help you with that if you open a ticket on their repo.

There it seems that the call is jumping to another thread and losing the dwcContext.

From what I understand, this might be the case. You can try to use a ThreadContextExecutor to ensure context propagation when calling tryGetDestination. Something like the following might work.

ThreadContextExecutor.fromCurrentContext().execute(() -> {  
    // your code 
    DestinationAccessor.tryGetDestination(yourDestinationName); 
});

Best,
Jonas

[davidfoerderer]

Hi @Jonas-Isr,

the old job lib triggered a job executing via a HTTP Call to our service. Now the job executing is triggered not via a remote call. We could get the DwCContext but it seems that the request headers not there as you could see in the second log entry showed in the attached screenshot above in this ticket.
Is there any way to inject the headers manually before calling the cloudsdk methods?

And I also will try the hint from you with the ThreadContextExecutor.

Best regards,
David