Client leaking thread-pools after trying to connect with expired refresh token

[kdvolder]

Using cf-java-client version 2.11.0.RELEASE.

Might be the result of some incorrect error handling. We pull a refresh token from a config.json file on disk (saved by the cf cli). Try to connect to CF using that as the credentials. This fails with an error if the token is expired.

Normally we would cache and reuse a connection, but if there was an error establishing a connection we don't cache it (to ensure the user doesn't get stuck with a 'broken' state in the cache for extended period of time).

The result of this is that we will try to establish connection again soon thereafter (next time we need the info). And everytime we make an attempt to create a client connection, there are 8 new cloudfoundry-client-nio-* threads being created.

E.g. here's a dump of the list of threads after a few attempts:

Java HotSpot(TM) 64-Bit Server VM [localhost:8000]	
	Thread [main] (Running)	
	Thread [pool-2-thread-1] (Running)	
	Thread [Reconciler-1] (Running)	
	Daemon Thread [reactor-http-nio-1] (Running)	
	Daemon Thread [reactor-http-nio-2] (Running)	
	Daemon Thread [reactor-http-nio-3] (Running)	
	Daemon Thread [reactor-http-nio-4] (Running)	
	Daemon Thread [reactor-http-nio-6] (Running)	
	Daemon Thread [reactor-http-nio-5] (Running)	
	Daemon Thread [reactor-http-nio-7] (Running)	
	Daemon Thread [reactor-http-nio-8] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	
	Daemon Thread [elastic-evictor-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	
	Daemon Thread [cloudfoundry-client-nio-3] (Running)	
	Daemon Thread [cloudfoundry-client-nio-2] (Running)	
	Daemon Thread [cloudfoundry-client-nio-1] (Running)	
	Daemon Thread [cloudfoundry-client-nio-4] (Running)	
	Daemon Thread [cloudfoundry-client-nio-5] (Running)	
	Daemon Thread [cloudfoundry-client-nio-6] (Running)	
	Daemon Thread [cloudfoundry-client-nio-7] (Running)	
	Daemon Thread [cloudfoundry-client-nio-8] (Running)	

[kdvolder]

Not sure this is useful. But just in case. A bunch of errors being logged:

[cloudfoundry-client-nio-6] WARN io.netty.util.concurrent.AbstractEventExecutor - A task raised an exception. Task: reactor.ipc.netty.channel.ContextHandler$$Lambda$183/447361330@11749e70
reactor.core.Exceptions$BubblingException: org.cloudfoundry.uaa.UaaException: invalid_token: Invalid refresh token (expired): eyJhbGciOiJSUzI1NiIsImtpZCI6InNoYTItMjAxNy0wMS0yMC1rZXkiLCJ0eXAiOiJKV1QifQ.eyJqdGkiOiI0NjkwMzExNDBmNjc0ZjhiOWFhYWZkZjlhMzFiYjIxNy1yIiwic3ViIjoiNzY0NzQ4YzgtY2U3ZS00OTA1LWI1NzMtYjAyNDU3OGM4Y2U5Iiwic2NvcGUiOlsib3BlbmlkIiwidWFhLnVzZXIiLCJjbG91ZF9jb250cm9sbGVyLnJlYWQiLCJwYXNzd29yZC53cml0ZSIsImNsb3VkX2NvbnRyb2xsZXIud3JpdGUiXSwiaWF0IjoxNDkwODA4MTkyLCJleHAiOjE0OTE0MTI5OTIsImNpZCI6ImNmIiwiY2xpZW50X2lkIjoiY2YiLCJpc3MiOiJodHRwczovL3VhYS5ydW4ucGl2b3RhbC5pby9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImdyYW50X3R5cGUiOiJwYXNzd29yZCIsInVzZXJfbmFtZSI6ImtkZXZvbGRlckBnb3Bpdm90YWwuY29tIiwib3JpZ2luIjoidWFhIiwidXNlcl9pZCI6Ijc2NDc0OGM4LWNlN2UtNDkwNS1iNTczLWIwMjQ1NzhjOGNlOSIsInJldl9zaWciOiI0ZDYwZDI2MCIsImF1ZCI6WyJjbG91ZF9jb250cm9sbGVyIiwicGFzc3dvcmQiLCJjZiIsInVhYSIsIm9wZW5pZCJdfQ.iH8TXuQgdYsCP1Ls-RlzPR4JneyoXd46RBJJsfIkzZM6DV0pVRiF7tiaiH9nWEge2VsTkEOfFq4-yKa--V_7dTqbgUJMk5HjrG7_oS8KrPlNnlUjQYNXQrj187Q07qcxH3m-CY9Yks7nl8uiI74SwWPJGu09KYCUorg3-hYSICC9e-v2H1hRVPmf20kICmGmDLY_-Tnk9U8-Vob1DHcLEMlRiUb0KLmCK6NtG0AAfA49MvWIjlnm2RxqAoRAci3k4HdHh9a7HZ60jvM4hdaMh-n28BoFssgfngR_qBjQ7zrooGtpeN9MXgcvRM5Xm32HALoZN2F7eNQrEn1cAGeGyQ expired at Wed Apr 05 17:23:12 UTC 2017
	at reactor.core.Exceptions.bubble(Exceptions.java:98)
	at reactor.core.publisher.Operators.onErrorDropped(Operators.java:292)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:152)
	at reactor.ipc.netty.channel.ContextHandler.fireContextError(ContextHandler.java:265)
	at reactor.ipc.netty.http.client.HttpClientOperations.onOutboundError(HttpClientOperations.java:491)
	at reactor.ipc.netty.channel.ChannelOperations.onError(ChannelOperations.java:225)
	at reactor.core.publisher.MonoThenMap$ThenMapMain.onError(MonoThenMap.java:164)
	at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:120)
	at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:120)
	at reactor.core.publisher.MonoWhen$WhenCoordinator.signalError(MonoWhen.java:189)
	at reactor.core.publisher.MonoWhen$WhenInner.onError(MonoWhen.java:315)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:379)
	at reactor.core.publisher.Operators.error(Operators.java:227)
	at reactor.core.publisher.MonoProcessor.subscribe(MonoProcessor.java:402)
	at reactor.core.publisher.FluxOnAssembly.subscribe(FluxOnAssembly.java:195)
	at reactor.core.publisher.MonoOnAssembly.subscribe(MonoOnAssembly.java:59)
	at reactor.core.publisher.MonoWhen$WhenCoordinator.subscribe(MonoWhen.java:178)
	at reactor.core.publisher.MonoWhen.subscribe(MonoWhen.java:124)
	at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55)
	at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55)
	at reactor.core.publisher.MonoThenMap.subscribe(MonoThenMap.java:59)
	at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:382)
	at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:468)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.cloudfoundry.uaa.UaaException: invalid_token: Invalid refresh token (expired): eyJhbGciOiJSUzI1NiIsImtpZCI6InNoYTItMjAxNy0wMS0yMC1rZXkiLCJ0eXAiOiJKV1QifQ.eyJqdGkiOiI0NjkwMzExNDBmNjc0ZjhiOWFhYWZkZjlhMzFiYjIxNy1yIiwic3ViIjoiNzY0NzQ4YzgtY2U3ZS00OTA1LWI1NzMtYjAyNDU3OGM4Y2U5Iiwic2NvcGUiOlsib3BlbmlkIiwidWFhLnVzZXIiLCJjbG91ZF9jb250cm9sbGVyLnJlYWQiLCJwYXNzd29yZC53cml0ZSIsImNsb3VkX2NvbnRyb2xsZXIud3JpdGUiXSwiaWF0IjoxNDkwODA4MTkyLCJleHAiOjE0OTE0MTI5OTIsImNpZCI6ImNmIiwiY2xpZW50X2lkIjoiY2YiLCJpc3MiOiJodHRwczovL3VhYS5ydW4ucGl2b3RhbC5pby9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImdyYW50X3R5cGUiOiJwYXNzd29yZCIsInVzZXJfbmFtZSI6ImtkZXZvbGRlckBnb3Bpdm90YWwuY29tIiwib3JpZ2luIjoidWFhIiwidXNlcl9pZCI6Ijc2NDc0OGM4LWNlN2UtNDkwNS1iNTczLWIwMjQ1NzhjOGNlOSIsInJldl9zaWciOiI0ZDYwZDI2MCIsImF1ZCI6WyJjbG91ZF9jb250cm9sbGVyIiwicGFzc3dvcmQiLCJjZiIsInVhYSIsIm9wZW5pZCJdfQ.iH8TXuQgdYsCP1Ls-RlzPR4JneyoXd46RBJJsfIkzZM6DV0pVRiF7tiaiH9nWEge2VsTkEOfFq4-yKa--V_7dTqbgUJMk5HjrG7_oS8KrPlNnlUjQYNXQrj187Q07qcxH3m-CY9Yks7nl8uiI74SwWPJGu09KYCUorg3-hYSICC9e-v2H1hRVPmf20kICmGmDLY_-Tnk9U8-Vob1DHcLEMlRiUb0KLmCK6NtG0AAfA49MvWIjlnm2RxqAoRAci3k4HdHh9a7HZ60jvM4hdaMh-n28BoFssgfngR_qBjQ7zrooGtpeN9MXgcvRM5Xm32HALoZN2F7eNQrEn1cAGeGyQ expired at Wed Apr 05 17:23:12 UTC 2017
	at org.cloudfoundry.reactor.util.ErrorPayloadMapper.lambda$null$8(ErrorPayloadMapper.java:89)
	at org.cloudfoundry.reactor.util.ErrorPayloadMapper.lambda$null$10(ErrorPayloadMapper.java:108)
	at reactor.core.publisher.MonoThenMap$ThenMapMain.onNext(MonoThenMap.java:120)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
	at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:115)
	at reactor.core.publisher.FluxUsing$UsingFuseableSubscriber.onNext(FluxUsing.java:351)
	at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:103)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:428)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:962)
	at reactor.core.publisher.MonoReduceSeed$ReduceSeedSubscriber.onComplete(MonoReduceSeed.java:150)
	at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:130)
	at reactor.ipc.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:375)
	at reactor.ipc.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:198)
	at reactor.ipc.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:338)
	at reactor.ipc.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:343)
	at reactor.ipc.netty.channel.ChannelOperations.onHandlerTerminate(ChannelOperations.java:419)
	at reactor.ipc.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:563)
	at reactor.ipc.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:125)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1240)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1041)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:559)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:476)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438)
	... 2 more
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Assembly trace from producer [reactor.core.publisher.MonoProcessor] :
	reactor.core.publisher.Mono.checkpoint(Mono.java:1400)
	org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.token(AbstractUaaTokenProvider.java:257)
	java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
	org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.getToken(AbstractUaaTokenProvider.java:108)
	org.cloudfoundry.reactor.util.AbstractReactorOperations.addAuthorization(AbstractReactorOperations.java:227)
	reactor.core.publisher.Mono.transform(Mono.java:3516)
	org.cloudfoundry.reactor.util.AbstractReactorOperations.lambda$null$3(AbstractReactorOperations.java:91)
	reactor.ipc.netty.http.client.MonoHttpClientResponse$HttpClientHandler.apply(MonoHttpClientResponse.java:117)
	reactor.ipc.netty.http.client.MonoHttpClientResponse$HttpClientHandler.apply(MonoHttpClientResponse.java:82)
	reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:381)
	reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:468)
	io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
Error has been observed by the following operator(s):
	|_	Mono.checkpoint(AbstractUaaTokenProvider.java:257)
	|_	Mono.checkpoint(ReactorOrganizations.java:199)
	|_	Flux.checkpoint(DefaultServices.java:298)
		|_	Mono.checkpoint(AbstractUaaTokenProvider.java:257)
	|_	Mono.checkpoint(ReactorPrivateDomains.java:70)
	|_	Flux.checkpoint(DefaultDomains.java:102)
	|_	Flux.checkpoint(DefaultServices.java:298)
			|_	Mono.checkpoint(AbstractUaaTokenProvider.java:257)
	|_	Mono.checkpoint(ReactorPrivateDomains.java:70)
	|_	Flux.checkpoint(DefaultDomains.java:102)
				|_	Mono.checkpoint(AbstractUaaTokenProvider.java:257)

	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.MonoOnAssembly.<init>(MonoOnAssembly.java:54)
		at reactor.core.publisher.Mono.checkpoint(Mono.java:1400)
		at org.cloudfoundry.reactor.client.v2.organizations.ReactorOrganizations.list(ReactorOrganizations.java:199)
		at org.cloudfoundry.operations._DefaultCloudFoundryOperations.lambda$null$3(_DefaultCloudFoundryOperations.java:270)
		at org.cloudfoundry.util.PaginationUtils.requestClientV2Resources(PaginationUtils.java:45)
		at org.cloudfoundry.operations._DefaultCloudFoundryOperations.lambda$requestOrganizations$4(_DefaultCloudFoundryOperations.java:269)
		at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:140)
		at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:46)
		at reactor.core.publisher.MonoSingle.subscribe(MonoSingle.java:56)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoMap.subscribe(MonoMap.java:55)
		at reactor.core.publisher.MonoProcessor.connect(MonoProcessor.java:438)
		at reactor.core.publisher.MonoProcessor.getOrStart(MonoProcessor.java:513)
		at reactor.core.publisher.MonoProcessor.subscribe(MonoProcessor.java:409)
		at reactor.core.publisher.MonoThenMap.subscribe(MonoThenMap.java:59)
		at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
		at reactor.core.publisher.MonoProcessor.connect(MonoProcessor.java:438)
		at reactor.core.publisher.MonoProcessor.getOrStart(MonoProcessor.java:513)
		at reactor.core.publisher.MonoProcessor.subscribe(MonoProcessor.java:409)
		at reactor.core.publisher.MonoWhen$WhenCoordinator.subscribe(MonoWhen.java:178)
		at reactor.core.publisher.MonoWhen.subscribe(MonoWhen.java:124)
		at reactor.core.publisher.MonoThenMap.subscribe(MonoThenMap.java:59)
		at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:49)
		at reactor.core.publisher.FluxOnAssembly.subscribe(FluxOnAssembly.java:195)
		at reactor.core.publisher.FluxOnAssembly.subscribe(FluxOnAssembly.java:227)
		at reactor.core.publisher.FluxMapFuseable.subscribe(FluxMapFuseable.java:63)
		at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:57)
		at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
		at reactor.core.publisher.Mono.block(Mono.java:1322)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:86)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getServices(DefaultClientRequestsV2.java:76)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.lambda$load$0(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:71)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getServices(CFTarget.java:145)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFServicesProvider.getHints(ManifestYamlCFServicesProvider.java:42)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.FluxOnAssembly.<init>(FluxOnAssembly.java:74)
		at reactor.core.publisher.Flux.checkpoint(Flux.java:2914)
		at org.cloudfoundry.operations.services.DefaultServices.listInstances(DefaultServices.java:298)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getServices(DefaultClientRequestsV2.java:80)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.lambda$load$0(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:71)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getServices(CFTarget.java:145)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFServicesProvider.getHints(ManifestYamlCFServicesProvider.java:42)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:120)
		at reactor.core.publisher.Mono.block(Mono.java:1323)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:86)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getServices(DefaultClientRequestsV2.java:76)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.lambda$load$0(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:71)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getServices(CFTarget.java:145)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFServicesProvider.getHints(ManifestYamlCFServicesProvider.java:42)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.MonoOnAssembly.<init>(MonoOnAssembly.java:54)
		at reactor.core.publisher.Mono.checkpoint(Mono.java:1400)
		at org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.token(AbstractUaaTokenProvider.java:257)
		at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
		at org.cloudfoundry.reactor.tokenprovider.AbstractUaaTokenProvider.getToken(AbstractUaaTokenProvider.java:108)
		at org.cloudfoundry.reactor.util.AbstractReactorOperations.addAuthorization(AbstractReactorOperations.java:227)
		at reactor.core.publisher.Mono.transform(Mono.java:3516)
		at org.cloudfoundry.reactor.util.AbstractReactorOperations.lambda$null$3(AbstractReactorOperations.java:91)
		at reactor.ipc.netty.http.client.MonoHttpClientResponse$HttpClientHandler.apply(MonoHttpClientResponse.java:117)
		at reactor.ipc.netty.http.client.MonoHttpClientResponse$HttpClientHandler.apply(MonoHttpClientResponse.java:82)
		at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:381)
		at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:468)
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
		... 2 more
	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.MonoOnAssembly.<init>(MonoOnAssembly.java:54)
		at reactor.core.publisher.Mono.checkpoint(Mono.java:1400)
		at org.cloudfoundry.reactor.client.v2.privatedomains.ReactorPrivateDomains.list(ReactorPrivateDomains.java:70)
		at org.cloudfoundry.operations.domains.DefaultDomains.lambda$requestListPrivateDomains$12(DefaultDomains.java:208)
		at org.cloudfoundry.util.PaginationUtils.requestClientV2Resources(PaginationUtils.java:45)
		at org.cloudfoundry.operations.domains.DefaultDomains.requestListPrivateDomains(DefaultDomains.java:207)
		at org.cloudfoundry.operations.domains.DefaultDomains.lambda$list$5(DefaultDomains.java:97)
		at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:140)
		at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:46)
		at reactor.core.publisher.FluxOnAssembly.subscribe(FluxOnAssembly.java:195)
		at reactor.core.publisher.FluxOnAssembly.subscribe(FluxOnAssembly.java:227)
		at reactor.core.publisher.FluxMapFuseable.subscribe(FluxMapFuseable.java:63)
		at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:57)
		at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
		at reactor.core.publisher.Mono.block(Mono.java:1322)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:86)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getDomains(DefaultClientRequestsV2.java:91)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.lambda$load$0(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:97)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getDomains(CFTarget.java:150)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFDomainsProvider.getHints(ManifestYamlCFDomainsProvider.java:37)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.FluxOnAssembly.<init>(FluxOnAssembly.java:74)
		at reactor.core.publisher.Flux.checkpoint(Flux.java:2914)
		at org.cloudfoundry.operations.domains.DefaultDomains.list(DefaultDomains.java:102)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getDomains(DefaultClientRequestsV2.java:95)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.lambda$load$0(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:97)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getDomains(CFTarget.java:150)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFDomainsProvider.getHints(ManifestYamlCFDomainsProvider.java:37)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:120)
		at reactor.core.publisher.Mono.block(Mono.java:1323)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:86)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getDomains(DefaultClientRequestsV2.java:91)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.lambda$load$0(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:97)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getDomains(CFTarget.java:150)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFDomainsProvider.getHints(ManifestYamlCFDomainsProvider.java:37)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.FluxOnAssembly.<init>(FluxOnAssembly.java:74)
		at reactor.core.publisher.Flux.checkpoint(Flux.java:2914)
		at org.cloudfoundry.operations.services.DefaultServices.listInstances(DefaultServices.java:298)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getServices(DefaultClientRequestsV2.java:80)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.lambda$load$0(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:71)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getServices(CFTarget.java:145)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFServicesProvider.getHints(ManifestYamlCFServicesProvider.java:42)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:120)
		at reactor.core.publisher.Mono.block(Mono.java:1323)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:86)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getServices(DefaultClientRequestsV2.java:76)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.lambda$load$0(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:78)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$2.load(CFTarget.java:71)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getServices(CFTarget.java:145)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFServicesProvider.getHints(ManifestYamlCFServicesProvider.java:42)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	[CIRCULAR REFERENCE:reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException]
	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.MonoOnAssembly.<init>(MonoOnAssembly.java:54)
		at reactor.core.publisher.Mono.checkpoint(Mono.java:1400)
		at org.cloudfoundry.reactor.client.v2.privatedomains.ReactorPrivateDomains.list(ReactorPrivateDomains.java:70)
		at org.cloudfoundry.operations.domains.DefaultDomains.lambda$requestListPrivateDomains$12(DefaultDomains.java:208)
		at org.cloudfoundry.util.PaginationUtils.requestClientV2Resources(PaginationUtils.java:45)
		at org.cloudfoundry.operations.domains.DefaultDomains.requestListPrivateDomains(DefaultDomains.java:207)
		at org.cloudfoundry.operations.domains.DefaultDomains.lambda$list$5(DefaultDomains.java:97)
		at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:140)
		at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:46)
		at reactor.core.publisher.FluxOnAssembly.subscribe(FluxOnAssembly.java:195)
		at reactor.core.publisher.FluxOnAssembly.subscribe(FluxOnAssembly.java:227)
		at reactor.core.publisher.FluxMapFuseable.subscribe(FluxMapFuseable.java:63)
		at reactor.core.publisher.MonoCollectList.subscribe(MonoCollectList.java:57)
		at reactor.core.publisher.MonoMapFuseable.subscribe(MonoMapFuseable.java:59)
		at reactor.core.publisher.Mono.block(Mono.java:1322)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:86)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getDomains(DefaultClientRequestsV2.java:91)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.lambda$load$0(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:97)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getDomains(CFTarget.java:150)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFDomainsProvider.getHints(ManifestYamlCFDomainsProvider.java:37)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException
		at reactor.core.publisher.FluxOnAssembly.<init>(FluxOnAssembly.java:74)
		at reactor.core.publisher.Flux.checkpoint(Flux.java:2914)
		at org.cloudfoundry.operations.domains.DefaultDomains.list(DefaultDomains.java:102)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getDomains(DefaultClientRequestsV2.java:95)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.lambda$load$0(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:97)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getDomains(CFTarget.java:150)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFDomainsProvider.getHints(ManifestYamlCFDomainsProvider.java:37)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:120)
		at reactor.core.publisher.Mono.block(Mono.java:1323)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.ReactorUtils.get(ReactorUtils.java:86)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.v2.DefaultClientRequestsV2.getDomains(DefaultClientRequestsV2.java:91)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.lambda$load$0(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFCallableContext.checkConnection(CFCallableContext.java:39)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.runAndCheckForFailure(CFTarget.java:110)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:101)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget$4.load(CFTarget.java:97)
		at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
		at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
		at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
		at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
		at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
		at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
		at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
		at org.springframework.ide.vscode.commons.cloudfoundry.client.cftarget.CFTarget.getDomains(CFTarget.java:150)
		at org.springframework.ide.vscode.manifest.yaml.ManifestYamlCFDomainsProvider.getHints(ManifestYamlCFDomainsProvider.java:37)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:57)
		at org.springframework.ide.vscode.manifest.yaml.AbstractCFHintsProvider.call(AbstractCFHintsProvider.java:31)
		at org.springframework.ide.vscode.commons.yaml.schema.YTypeFactory.lambda$valuesFromHintProvider$7(YTypeFactory.java:857)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.lambda$provider$1(EnumValueParser.java:90)
		at org.springframework.ide.vscode.commons.util.EnumValueParser.parse(EnumValueParser.java:57)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:193)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:180)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:167)
		at org.springframework.ide.vscode.commons.yaml.reconcile.SchemaBasedYamlASTReconciler.reconcile(SchemaBasedYamlASTReconciler.java:105)
		at org.springframework.ide.vscode.commons.yaml.reconcile.YamlReconcileEngine.reconcile(YamlReconcileEngine.java:46)
		at org.springframework.ide.vscode.commons.languageserver.util.SimpleLanguageServer.lambda$validateWith$5(SimpleLanguageServer.java:358)
		at reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:40)
		at reactor.core.publisher.MonoOnErrorResume.subscribe(MonoOnErrorResume.java:44)
		at reactor.core.publisher.MonoDoFinally.subscribe(MonoDoFinally.java:47)
		at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:101)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		... 1 more
	[CIRCULAR REFERENCE:reactor.core.publisher.FluxOnAssembly$AssemblySnapshotException]

[nebhale]

That looks like you're recreating the ConnectionContext repeatedly, which isn't necessary and discouraged. The TokenProvider and ConnectionContext are kept separate because one is scoped to a host/port and the other to a user. If a token is invalid, it's appropriate to dispose of the TokenProvider and continue using the ConnectionContext instead of recreating it.

[nebhale]

The implication of this design is that multiple users can safely use a single ConnectionContext to a given host, concurrently. You can also use a single TokenProvider to connect to multiple hosts, concurrently (given that they share the same credentials, obviously).

You can imagine a scenario where you've got a Map of host/port to ConnectionContext and a Map of user to TokenProvider. You can then pick the appropriate pair for any given request. The only state stored in the ConnectionContext is open sockets (no user information associated with them) and the state stored in the TokenProvider is scoped to the ConnectionContext passed it.

[kdvolder]

:-( This will make for rather complex caching logic to determine which parts can be reused and which parts can't, depending on the type of error coming back from the code trying to create a configured client.

I'm also not so sure we can really determine generally what class of errors should map to reusing which parts of the client.

Anyhow.... so (re-)creating connection context is expected to create a new thread pool every time? If that is really 'by design', then there should be a 'dispose' or 'close' method to clean it up if you no longer need/want to use it. Is there? I couldn't find one.

[nebhale]

There certainly is an argument for a cleanup method, and this issue will stay open to add that, but I don't think there's any complex logic about what errors result in the need for disposal.

Effectively, you should never dispose of a ConnectionContext; it should live for the duration of the application. There's nothing that can happen during a request that would require you to dispose of it before you were completely finished with connections to a given server.

The TokenProvider (really subclasses of AbstractUaaTokenProvider) pass through the UaaException generated by the UAA when authentication fails because of credentials. So really, the test is whether the UaaException has a status code matching unauthorized. If so, you need to request new credentials from the user and create a new TokenProvider with those credentials. Note that if we experience an unauthorized with a token, we attempt a refresh. If that refresh fails as unauthorized, we go back to the "primary" token flow and attempt again. Only at that point, if the "primary" token flow fails, do we expose the unauthorized to the user.

So, to summarize:

1. Never dispose of ConnectionContext.
2. Dispose of TokenProvider whenever a UAA unauthorized exception makes it out of the API.

[kdvolder]

There's nothing that can happen during a request that would require you to dispose of it before you were completely finished with connections to a given server.

What if we were finished? What if we wanted to switch to connecting to a different server, for example, because the user ran a cf login and switched to a different server?

There are also scenarios where the connection breaks. e.g network interruptions. This would break the connection context because the socket closed. We do not want users to be forced to reboot their IDE/editor, to flush the broken connection context from the cache.

[kdvolder]

We also don't want to keep the connection context (and the rest of the connection) if it hasn't been used in a while. I.e. we only cache these things for a 'reasonable' time. If the user stops editing CF manifest files, we don't need that connection anymore, and its really not so nice to keep this stuff around indefinitely.

[kdvolder]

So really, the test is whether the UaaException has a status code matching unauthorized.

I think, at least sometimes, we don't get that. Instead we get 'peer closed the connection' or something like that.

[nebhale]

That's a fair point, and we'll add a close() method for that purpose, but to be clear the most expensive part of the Java Client is the setup of a new ConnectionContext. So being conservative about its disposal is a good idea. Sticking to that policy of sharing between users as much as possible (one per host, not one per host per user) and never disposing one (except as you identify when it's idle for a significant period of time) is a good idea.

[nebhale]

I think, at least sometimes, we don't get that. Instead we get 'peer closed the connection' or something like that.

Since around Java Client 2.8.0 or so (I can't remember exactly when @smaldini fixed it), reactor-netty automatically retries requests on connections that have been closed by load balancers/gateways. Even in the case where you do somehow experience that, there's nothing wrong with the ConnectionContext itself; the next request through it will create a new connection in the connection pool and the request will go through (if there aren't significant networking issues).

[kdvolder]

The 90% case for us is ... there are no errors and nothing changes (same host, same target, same user, same credentials, same... everything). In that case we reuse the connection. We put in a special case for 'connection looks busted'. Going more fine-grained than that, based on various error conditions and what changed or didn't change just seems 'complicated' to me to handle. I guess opinions on this can be debated. But to me it seems too complex to handle what are essentialy just edge cases. So what if in those few cases we recreated the connection context. That's fine I think. Especially if the simpler logic makes it much less likely that their are bugs in our code that mean 'broken states' get stuck in the cache and force users to reboot their IDE to fix that.

[kdvolder]

Since around Java Client 2.8.0 ...

Very interesting. Thanks for that info. If we really can count on connection context to 'self repair' after experiencing NW interruptions that's great. And then it might make good sense to cache the connection context separately. As long as we can really count on this 'self repair'. A big reason of why we go and 'flush' the cache on errors was because we experienced cases with 'permanently broken' connections after some network issues.