Channel in pool in an invalid state after SSLException

[madgnome]

Issue

After encountering the SSLException:

javax.net.ssl.SSLException: SSLEngine closed already
	at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)

the channel will be unusable and will throw this exception every time:

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
	at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106)
	at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	at reactor.ipc.netty.channel.ChannelOperationsHandler.doWrite(ChannelOperationsHandler.java:290)
	at reactor.ipc.netty.channel.ChannelOperationsHandler.drain(ChannelOperationsHandler.java:464)
	at reactor.ipc.netty.channel.ChannelOperationsHandler.flush(ChannelOperationsHandler.java:192)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:831)
	at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1071)
	at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:300)
	at reactor.ipc.netty.http.HttpOperations.lambda$then$0(HttpOperations.java:135)
	at reactor.ipc.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:197)
	at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:167)
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
	at reactor.core.publisher.Mono.subscribe(Mono.java:3080)
	at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:141)
	at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60)
	at reactor.core.publisher.MonoSourceFlux.subscribe(MonoSourceFlux.java:47)
	at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:46)
	at reactor.ipc.netty.channel.ChannelOperations.applyHandler(ChannelOperations.java:380)
	at reactor.ipc.netty.http.client.HttpClientOperations.onHandlerStart(HttpClientOperations.java:502)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest
	at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86)
	at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167)
	at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88)
	... 32 common frames omitted

Reactor Netty version

0.7.7.RELEASE

Workaround

Disabling connection pool fixes the issue.

[madgnome]

I've managed to reproduce with HttpClient logging handler logs enabled:

Last successful log sequences:
⚠️It's in reverse order (older log are at the end)

5/17/18 1:28:15.650 Releasing channel: [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443]
5/17/18 1:28:15.650 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Disposing context reactor.ipc.netty.channel.PooledClientContextHandler@7dbb06e5
5/17/18 1:28:15.650 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] USER_EVENT: [Handler Terminated]
5/17/18 1:28:15.649 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Received last HTTP packet
5/17/18 1:28:15.649 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Subscribing inbound receiver [pending: 0, cancelled:false, inboundDone: false]
5/17/18 1:28:15.649 <= [200] POST "https://xyz.net/v1/events" 87ms
5/17/18 1:28:15.649 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] Received response (auto-read:false) : [Access-Control-Allow-Credentials=true, Access-Control-Allow-Headers=Origin, X-Requested-With, Content-Type, Accept, Access-Control-Allow-Methods=POST, OPTIONS, Access-Control-Allow-Origin=*, Access-Control-Max-Age=3600, Content-Type=application/json, Date=Thu, 17 May 2018 03:28:15 GMT, Server=nginx/1.12.2, Content-Length=65, Connection=keep-alive]
5/17/18 1:28:15.648 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] READ: 444B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d |HTTP/1.1 200 OK.| |00000010| 0a 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d |.Access-Control-| |00000020| 41 6c 6c 6f 77 2d 43 72 65 64 65 6e 74 69 61 6c |Allow-Credential| |00000030| 73 3a 20 74 72 75 65 0d 0a 41 63 63 65 73 73 2d |s: true..Access-| |00000040| 43 6f 6e 74 72 6f 6c 2d 41 6c 6c 6f 77 2d 48 65 |Control-Allow-He| |00000050| 61 64 65 72 73 3a 20 4f 72 69 67 69 6e 2c 20 58 |aders: Origin, X| |00000060| 2d 52 65 71 75 65 73 74 65 64 2d 57 69 74 68 2c |-Requested-With,| |00000070| 20 43 6f 6e 74 65 6e 74 2d 54 79 70 65 2c 20 41 | Content-Type, A| |00000080| 63 63 65 70 74 0d 0a 41 63 63 65 73 73 2d 43 6f |ccept..Access-Co| |00000090| 6e 74 72 6f 6c 2d 41 6c 6c 6f 77 2d 4d 65 74 68 |ntrol-Allow-Meth| |000000a0| 6f 64 73 3a 20 50 4f 53 54 2c 20 4f 50 54 49 4f |ods: POST, OPTIO| |000000b0| 4e 53 0d 0a 41 63 63 65 73 73 2d 43 6f 6e 74 72 |NS..Access-Contr| |000000c0| 6f 6c 2d 41 6c 6c 6f 77 2d 4f 72 69 67 69 6e 3a |ol-Allow-Origin:| |000000d0| 20 2a 0d 0a 41 63 63 65 73 73 2d 43 6f 6e 74 72 | *..Access-Contr| |000000e0| 6f 6c 2d 4d 61 78 2d 41 67 65 3a 20 33 36 30 30 |ol-Max-Age: 3600| |000000f0| 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 |..Content-Type: | |00000100| 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e |application/json| |00000110| 0d 0a 44 61 74 65 3a 20 54 68 75 2c 20 31 37 20 |..Date: Thu, 17 | |00000120| 4d 61 79 20 32 30 31 38 20 30 33 3a 32 38 3a 31 |May 2018 03:28:1| |00000130| 35 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 6e |5 GMT..Server: n| |00000140| 67 69 6e 78 2f 31 2e 31 32 2e 32 0d 0a 43 6f 6e |ginx/1.12.2..Con| |00000150| 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 36 35 0d |tent-Length: 65.| |00000160| 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 6b 65 65 |.Connection: kee| |00000170| 70 2d 61 6c 69 76 65 0d 0a 0d 0a 7b 0a 20 20 22 |p-alive....{. "| |00000180| 69 6e 76 61 6c 69 64 5f 70 72 6f 64 75 63 74 22 |invalid_product"| |00000190| 3a 20 30 2c 20 0a 20 20 22 70 72 6f 63 65 73 73 |: 0, . "process| |000001a0| 65 64 20 22 3a 20 31 33 2c 20 0a 20 20 22 73 6b |ed ": 13, . "sk| |000001b0| 69 70 70 65 64 22 3a 20 30 0a 7d 0a |ipped": 0.}. | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.565 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] FLUSH
5/17/18 1:28:15.565 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 5B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 30 0d 0a 0d 0a |0.... | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.565 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] FLUSH
5/17/18 1:28:15.564 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 2B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 0d 0a |.. | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.564 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 7391B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 7b 22 65 76 65 6e 74 73 22 3a 5b 7b 22 6e 61 6d |{"events":[{"nam| |00000010| 65 22 3a 22 61 74 6c 61 73 73 69 61 6e 2e 66 61 |e":"atlassian.fa| |00000020| 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 70 75 62 |bric.frontendpub| |00000030| 73 75 62 2e 63 68 61 6e 6e 65 6c 73 2e 73 75 62 |sub.channels.sub| |00000040| 73 63 72 69 62 65 64 22 2c 22 73 65 72 76 65 72 |scribed","server| |00000050| 22 3a 22 70 72 6f 64 75 63 74 69 6f 6e 22 2c 22 |":"production","| |00000060| 70 72 6f 64 75 63 74 22 3a 22 68 69 70 63 68 61 |product":"hipcha| |00000070| 74 22 2c 22 70 72 6f 70 65 72 74 69 65 73 22 3a |t","properties":| |00000080| 7b 22 63 6c 69 65 6e 74 2e 70 6c 61 74 66 6f 72 |{"client.platfor| |00000090| 6d 22 3a 22 57 45 42 22 2c 22 63 6c 69 65 6e 74 |m":"WEB","client| |000000a0| 2e 76 65 72 73 69 6f 6e 22 3a 22 31 2e 30 22 2c |.version":"1.0",| |000000b0| 22 63 68 61 6e 6e 65 6c 43 6f 75 6e 74 22 3a 31 |"channelCount":1| |000000c0| 32 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |2},"serverTime":| |000000d0| 31 35 32 36 35 32 37 36 39 35 32 33 35 2c 22 75 |1526527695235,"u| |000000e0| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |000000f0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000100| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000110| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00000120| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000130| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000140| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00000150| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00000160| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000170| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000180| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000190| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |000001a0| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |000001b0| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |000001c0| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |000001d0| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |000001e0| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |000001f0| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000200| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00000210| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000220| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000230| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00000240| 2f 35 35 32 33 35 35 33 37 2d 34 38 37 34 2d 35 |/55235537-4874-5| |00000250| 34 30 34 2d 61 38 62 34 2d 33 36 32 61 65 39 37 |404-a8b4-362ae97| |00000260| 34 62 37 39 64 22 2c 22 63 68 61 6e 6e 65 6c 54 |4b79d","channelT| |00000270| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00000280| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00000290| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |000002a0| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |000002b0| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |000002c0| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |000002d0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |000002e0| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |000002f0| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00000300| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |00000310| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |00000320| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00000330| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000340| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000350| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00000360| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00000370| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000380| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000390| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |000003a0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000003b0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |000003c0| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |000003d0| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |000003e0| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |000003f0| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00000400| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00000410| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00000420| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00000430| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00000440| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00000450| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00000460| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000470| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00000480| 65 72 73 61 74 69 6f 6e 2f 32 63 30 39 39 30 66 |ersation/2c0990f| |00000490| 36 2d 63 30 61 39 2d 35 64 31 62 2d 39 33 30 36 |6-c0a9-5d1b-9306| |000004a0| 2d 36 63 37 64 63 64 66 62 33 36 66 39 22 2c 22 |-6c7dcdfb36f9","| |000004b0| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |000004c0| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |000004d0| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |000004e0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |000004f0| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00000500| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00000510| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00000520| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00000530| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00000540| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00000550| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00000560| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00000570| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00000580| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000590| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |000005a0| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |000005b0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |000005c0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |000005d0| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |000005e0| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |000005f0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000600| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000610| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000620| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00000630| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00000640| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00000650| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00000660| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00000670| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00000680| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000690| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |000006a0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000006b0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |000006c0| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |000006d0| 2f 32 64 33 66 39 30 30 31 2d 31 65 61 39 2d 34 |/2d3f9001-1ea9-4| |000006e0| 38 34 31 2d 38 32 34 37 2d 35 38 37 30 64 30 32 |841-8247-5870d02| |000006f0| 63 38 66 36 31 22 2c 22 63 68 61 6e 6e 65 6c 54 |c8f61","channelT| |00000700| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00000710| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00000720| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00000730| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00000740| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00000750| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00000760| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00000770| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |00000780| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00000790| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |000007a0| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |000007b0| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |000007c0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |000007d0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |000007e0| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |000007f0| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00000800| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000810| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000820| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00000830| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000840| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000850| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00000860| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00000870| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00000880| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00000890| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |000008a0| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |000008b0| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |000008c0| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |000008d0| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |000008e0| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |000008f0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000900| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00000910| 65 72 73 61 74 69 6f 6e 2f 37 36 30 32 66 64 33 |ersation/7602fd3| |00000920| 31 2d 65 32 62 32 2d 35 36 61 33 2d 39 35 66 31 |1-e2b2-56a3-95f1| |00000930| 2d 36 36 35 36 31 32 39 66 37 61 31 61 22 2c 22 |-6656129f7a1a","| |00000940| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00000950| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00000960| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00000970| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00000980| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00000990| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |000009a0| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |000009b0| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |000009c0| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |000009d0| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |000009e0| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |000009f0| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00000a00| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00000a10| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000a20| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000a30| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00000a40| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000a50| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000a60| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00000a70| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00000a80| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000a90| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000aa0| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000ab0| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00000ac0| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00000ad0| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00000ae0| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00000af0| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00000b00| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00000b10| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000b20| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00000b30| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000b40| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000b50| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00000b60| 2f 34 34 66 36 34 33 35 62 2d 36 36 32 35 2d 35 |/44f6435b-6625-5| |00000b70| 36 34 31 2d 62 31 34 64 2d 65 38 37 32 39 38 65 |641-b14d-e87298e| |00000b80| 65 61 39 31 30 22 2c 22 63 68 61 6e 6e 65 6c 54 |ea910","channelT| |00000b90| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00000ba0| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00000bb0| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00000bc0| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00000bd0| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00000be0| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00000bf0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00000c00| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |00000c10| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00000c20| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |00000c30| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |00000c40| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00000c50| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000c60| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000c70| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00000c80| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00000c90| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000ca0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000cb0| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00000cc0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000cd0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000ce0| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00000cf0| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00000d00| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00000d10| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00000d20| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00000d30| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00000d40| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00000d50| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00000d60| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00000d70| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00000d80| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000d90| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00000da0| 65 72 73 61 74 69 6f 6e 2f 35 33 64 34 34 31 39 |ersation/53d4419| |00000db0| 62 2d 34 61 34 32 2d 34 61 62 62 2d 39 65 61 35 |b-4a42-4abb-9ea5| |00000dc0| 2d 32 37 66 65 35 37 62 36 34 32 30 33 22 2c 22 |-27fe57b64203","| |00000dd0| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00000de0| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00000df0| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00000e00| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00000e10| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00000e20| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00000e30| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00000e40| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00000e50| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00000e60| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00000e70| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00000e80| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00000e90| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00000ea0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00000eb0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00000ec0| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00000ed0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00000ee0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00000ef0| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00000f00| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00000f10| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00000f20| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00000f30| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00000f40| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00000f50| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00000f60| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00000f70| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00000f80| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00000f90| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00000fa0| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00000fb0| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00000fc0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00000fd0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00000fe0| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00000ff0| 2f 63 35 39 32 36 35 64 61 2d 33 33 61 30 2d 35 |/c59265da-33a0-5| |00001000| 35 63 35 2d 62 37 31 36 2d 64 66 36 39 30 37 65 |5c5-b716-df6907e| |00001010| 37 34 39 31 35 22 2c 22 63 68 61 6e 6e 65 6c 54 |74915","channelT| |00001020| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00001030| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00001040| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00001050| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00001060| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00001070| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00001080| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00001090| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |000010a0| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |000010b0| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |000010c0| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |000010d0| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |000010e0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |000010f0| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001100| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00001110| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00001120| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001130| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001140| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00001150| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00001160| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001170| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00001180| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00001190| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |000011a0| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |000011b0| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |000011c0| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |000011d0| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |000011e0| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |000011f0| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00001200| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00001210| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001220| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00001230| 65 72 73 61 74 69 6f 6e 2f 66 64 62 38 38 31 38 |ersation/fdb8818| |00001240| 31 2d 61 39 63 30 2d 35 37 65 37 2d 38 38 30 34 |1-a9c0-57e7-8804| |00001250| 2d 35 37 63 33 36 64 36 64 64 33 39 30 22 2c 22 |-57c36d6dd390","| |00001260| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00001270| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00001280| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00001290| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |000012a0| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |000012b0| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |000012c0| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |000012d0| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |000012e0| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |000012f0| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00001300| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00001310| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00001320| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00001330| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001340| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001350| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00001360| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001370| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001380| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00001390| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |000013a0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |000013b0| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |000013c0| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |000013d0| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |000013e0| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |000013f0| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00001400| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |00001410| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |00001420| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |00001430| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |00001440| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |00001450| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00001460| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001470| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00001480| 2f 32 35 30 34 31 61 34 33 2d 63 65 62 36 2d 35 |/25041a43-ceb6-5| |00001490| 65 62 35 2d 61 34 38 38 2d 64 66 39 63 63 38 63 |eb5-a488-df9cc8c| |000014a0| 39 31 38 38 64 22 2c 22 63 68 61 6e 6e 65 6c 54 |9188d","channelT| |000014b0| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |000014c0| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |000014d0| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |000014e0| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |000014f0| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00001500| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |00001510| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |00001520| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |00001530| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |00001540| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |00001550| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |00001560| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00001570| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001580| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001590| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |000015a0| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |000015b0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |000015c0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |000015d0| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |000015e0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000015f0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001600| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00001610| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00001620| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00001630| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00001640| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00001650| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00001660| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00001670| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00001680| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00001690| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |000016a0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |000016b0| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |000016c0| 65 72 73 61 74 69 6f 6e 2f 33 31 39 62 39 32 36 |ersation/319b926| |000016d0| 63 2d 62 66 30 61 2d 35 39 33 63 2d 62 36 38 32 |c-bf0a-593c-b682| |000016e0| 2d 34 37 64 34 62 61 65 64 37 62 31 62 22 2c 22 |-47d4baed7b1b","| |000016f0| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00001700| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00001710| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00001720| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00001730| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00001740| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00001750| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00001760| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00001770| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00001780| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00001790| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |000017a0| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |000017b0| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |000017c0| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |000017d0| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |000017e0| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |000017f0| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001800| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001810| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00001820| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00001830| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001840| 35 30 64 36 64 33 38 36 62 65 33 22 7d 2c 7b 22 |50d6d386be3"},{"| |00001850| 6e 61 6d 65 22 3a 22 61 74 6c 61 73 73 69 61 6e |name":"atlassian| |00001860| 2e 66 61 62 72 69 63 2e 66 72 6f 6e 74 65 6e 64 |.fabric.frontend| |00001870| 70 75 62 73 75 62 2e 63 68 61 6e 6e 65 6c 2e 72 |pubsub.channel.r| |00001880| 65 73 75 62 73 63 72 69 62 65 64 22 2c 22 73 65 |esubscribed","se| |00001890| 72 76 65 72 22 3a 22 70 72 6f 64 75 63 74 69 6f |rver":"productio| |000018a0| 6e 22 2c 22 70 72 6f 64 75 63 74 22 3a 22 68 69 |n","product":"hi| |000018b0| 70 63 68 61 74 22 2c 22 70 72 6f 70 65 72 74 69 |pchat","properti| |000018c0| 65 73 22 3a 7b 22 63 68 61 6e 6e 65 6c 22 3a 22 |es":{"channel":"| |000018d0| 61 72 69 3a 63 6c 6f 75 64 3a 63 68 61 74 3a 35 |ari:cloud:chat:5| |000018e0| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |000018f0| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001900| 62 65 33 3a 63 6f 6e 76 65 72 73 61 74 69 6f 6e |be3:conversation| |00001910| 2f 31 34 36 63 66 64 35 61 2d 37 63 32 37 2d 35 |/146cfd5a-7c27-5| |00001920| 62 65 66 2d 62 39 31 31 2d 39 39 38 38 38 64 34 |bef-b911-99888d4| |00001930| 36 65 38 63 65 22 2c 22 63 68 61 6e 6e 65 6c 54 |6e8ce","channelT| |00001940| 79 70 65 22 3a 22 63 6f 6e 76 65 72 73 61 74 69 |ype":"conversati| |00001950| 6f 6e 22 2c 22 63 68 61 6e 6e 65 6c 4f 77 6e 65 |on","channelOwne| |00001960| 72 22 3a 22 63 68 61 74 22 2c 22 73 75 62 73 63 |r":"chat","subsc| |00001970| 72 69 70 74 69 6f 6e 49 64 22 3a 22 64 35 63 61 |riptionId":"d5ca| |00001980| 32 30 35 63 2d 38 66 63 62 2d 34 39 36 31 2d 62 |205c-8fcb-4961-b| |00001990| 33 37 34 2d 61 35 61 65 62 61 39 32 66 66 35 63 |374-a5aeba92ff5c| |000019a0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 43 |","subscriptionC| |000019b0| 6f 75 6e 74 22 3a 31 36 2c 22 73 75 62 73 63 72 |ount":16,"subscr| |000019c0| 69 70 74 69 6f 6e 44 61 74 65 22 3a 31 35 32 36 |iptionDate":1526| |000019d0| 34 39 30 33 33 36 30 33 30 7d 2c 22 73 65 72 76 |490336030},"serv| |000019e0| 65 72 54 69 6d 65 22 3a 31 35 32 36 35 32 37 36 |erTime":15265276| |000019f0| 39 35 32 34 31 2c 22 75 73 65 72 22 3a 22 35 35 |95241,"user":"55| |00001a00| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001a10| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001a20| 35 34 34 63 64 37 38 30 33 22 2c 22 75 73 65 72 |544cd7803","user| |00001a30| 5f 69 64 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |_id":"557058:bb8| |00001a40| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001a50| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001a60| 33 22 2c 22 63 6c 6f 75 64 5f 69 64 22 3a 22 35 |3","cloud_id":"5| |00001a70| 36 35 65 30 66 61 36 2d 36 38 32 34 2d 34 33 31 |65e0fa6-6824-431| |00001a80| 61 2d 62 36 38 63 2d 33 35 30 64 36 64 33 38 36 |a-b68c-350d6d386| |00001a90| 62 65 33 22 7d 2c 7b 22 6e 61 6d 65 22 3a 22 61 |be3"},{"name":"a| |00001aa0| 74 6c 61 73 73 69 61 6e 2e 66 61 62 72 69 63 2e |tlassian.fabric.| |00001ab0| 66 72 6f 6e 74 65 6e 64 70 75 62 73 75 62 2e 63 |frontendpubsub.c| |00001ac0| 68 61 6e 6e 65 6c 2e 72 65 73 75 62 73 63 72 69 |hannel.resubscri| |00001ad0| 62 65 64 22 2c 22 73 65 72 76 65 72 22 3a 22 70 |bed","server":"p| |00001ae0| 72 6f 64 75 63 74 69 6f 6e 22 2c 22 70 72 6f 64 |roduction","prod| |00001af0| 75 63 74 22 3a 22 68 69 70 63 68 61 74 22 2c 22 |uct":"hipchat","| |00001b00| 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 63 68 |properties":{"ch| |00001b10| 61 6e 6e 65 6c 22 3a 22 61 72 69 3a 63 6c 6f 75 |annel":"ari:clou| |00001b20| 64 3a 63 68 61 74 3a 35 36 35 65 30 66 61 36 2d |d:chat:565e0fa6-| |00001b30| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001b40| 35 30 64 36 64 33 38 36 62 65 33 3a 63 6f 6e 76 |50d6d386be3:conv| |00001b50| 65 72 73 61 74 69 6f 6e 2f 39 32 63 31 65 35 35 |ersation/92c1e55| |00001b60| 66 2d 64 39 35 63 2d 35 36 32 34 2d 39 31 63 64 |f-d95c-5624-91cd| |00001b70| 2d 37 64 36 32 62 32 34 61 34 37 33 38 22 2c 22 |-7d62b24a4738","| |00001b80| 63 68 61 6e 6e 65 6c 54 79 70 65 22 3a 22 63 6f |channelType":"co| |00001b90| 6e 76 65 72 73 61 74 69 6f 6e 22 2c 22 63 68 61 |nversation","cha| |00001ba0| 6e 6e 65 6c 4f 77 6e 65 72 22 3a 22 63 68 61 74 |nnelOwner":"chat| |00001bb0| 22 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 49 |","subscriptionI| |00001bc0| 64 22 3a 22 64 35 63 61 32 30 35 63 2d 38 66 63 |d":"d5ca205c-8fc| |00001bd0| 62 2d 34 39 36 31 2d 62 33 37 34 2d 61 35 61 65 |b-4961-b374-a5ae| |00001be0| 62 61 39 32 66 66 35 63 22 2c 22 73 75 62 73 63 |ba92ff5c","subsc| |00001bf0| 72 69 70 74 69 6f 6e 43 6f 75 6e 74 22 3a 31 36 |riptionCount":16| |00001c00| 2c 22 73 75 62 73 63 72 69 70 74 69 6f 6e 44 61 |,"subscriptionDa| |00001c10| 74 65 22 3a 31 35 32 36 34 39 30 33 33 36 30 33 |te":152649033603| |00001c20| 30 7d 2c 22 73 65 72 76 65 72 54 69 6d 65 22 3a |0},"serverTime":| |00001c30| 31 35 32 36 35 32 37 36 39 35 32 34 31 2c 22 75 |1526527695241,"u| |00001c40| 73 65 72 22 3a 22 35 35 37 30 35 38 3a 62 62 38 |ser":"557058:bb8| |00001c50| 36 37 61 34 35 2d 66 37 37 65 2d 34 36 33 32 2d |67a45-f77e-4632-| |00001c60| 62 61 61 33 2d 35 36 64 35 34 34 63 64 37 38 30 |baa3-56d544cd780| |00001c70| 33 22 2c 22 75 73 65 72 5f 69 64 22 3a 22 35 35 |3","user_id":"55| |00001c80| 37 30 35 38 3a 62 62 38 36 37 61 34 35 2d 66 37 |7058:bb867a45-f7| |00001c90| 37 65 2d 34 36 33 32 2d 62 61 61 33 2d 35 36 64 |7e-4632-baa3-56d| |00001ca0| 35 34 34 63 64 37 38 30 33 22 2c 22 63 6c 6f 75 |544cd7803","clou| |00001cb0| 64 5f 69 64 22 3a 22 35 36 35 65 30 66 61 36 2d |d_id":"565e0fa6-| |00001cc0| 36 38 32 34 2d 34 33 31 61 2d 62 36 38 63 2d 33 |6824-431a-b68c-3| |00001cd0| 35 30 64 36 64 33 38 36 62 65 33 22 7d 5d 7d |50d6d386be3"}]} | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.564 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 6B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 31 63 64 66 0d 0a |1cdf.. | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.563 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] FLUSH
5/17/18 1:28:15.563 [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] WRITE: 299B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 50 4f 53 54 20 2f 76 31 2f 65 76 65 6e 74 73 20 |POST /v1/events | |00000010| 48 54 54 50 2f 31 2e 31 0d 0a 75 73 65 72 2d 61 |HTTP/1.1..user-a| |00000020| 67 65 6e 74 3a 20 52 65 61 63 74 6f 72 4e 65 74 |gent: ReactorNet| |00000030| 74 79 2f 30 2e 37 2e 37 2e 52 45 4c 45 41 53 45 |ty/0.7.7.RELEASE| |00000040| 0d 0a 74 72 61 6e 73 66 65 72 2d 65 6e 63 6f 64 |..transfer-encod| |00000050| 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a 68 6f |ing: chunked..ho| |00000060| 73 74 3a 20 6d 67 61 73 2e 70 72 6f 64 2e 70 75 |st: mgas.prod.pu| |00000070| 62 6c 69 63 2e 61 74 6c 2d 70 61 61 73 2e 6e 65 |blic.atl-paas.ne| |00000080| 74 0d 0a 41 63 63 65 70 74 3a 20 61 70 70 6c 69 |t..Accept: appli| |00000090| 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 |cation/json;char| |000000a0| 73 65 74 3d 55 54 46 2d 38 0d 0a 58 2d 42 33 2d |set=UTF-8..X-B3-| |000000b0| 54 72 61 63 65 49 64 3a 20 33 61 39 66 33 35 36 |TraceId: 3a9f356| |000000c0| 37 38 33 62 39 31 62 34 62 0d 0a 58 2d 42 33 2d |783b91b4b..X-B3-| |000000d0| 53 70 61 6e 49 64 3a 20 33 61 39 66 33 35 36 37 |SpanId: 3a9f3567| |000000e0| 38 33 62 39 31 62 34 62 0d 0a 58 2d 42 33 2d 53 |83b91b4b..X-B3-S| |000000f0| 61 6d 70 6c 65 64 3a 20 30 0d 0a 43 6f 6e 74 65 |ampled: 0..Conte| |00000100| 6e 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 |nt-Type: applica| |00000110| 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 73 65 |tion/json;charse| |00000120| 74 3d 55 54 46 2d 38 0d 0a 0d 0a |t=UTF-8.... | +--------+-------------------------------------------------+----------------+
5/17/18 1:28:15.563 [HttpClient] [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1321/1871970505@2bc6c88c}
5/17/18 1:28:15.563 Acquired active channel: [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443]
5/17/18 1:28:15.562 Acquired [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443], now 1 active connections
5/17/18 1:28:15.562 Acquiring existing channel from pool: DefaultPromise@15d7d7a5(success: [id: 0x164a6972, L:/10.255.0.3:49512 - R:xyz.net/52.72.168.184:443]) SimpleChannelPool{activeConnections=0}
5/17/18 1:28:15.562 => POST "https://xyz.net/v1/events"

Shit starts to hit the fan from here:

Log when the SSL exception happens:
⚠️It's in reverse order (older log are at the end)

5/17/18 5:33:11.548 Released [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443], now 0 active connections
5/17/18 5:33:11.548 Releasing channel: [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443]
5/17/18 5:33:11.548 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] Disposing context reactor.ipc.netty.channel.PooledClientContextHandler@7dbb06f4
5/17/18 5:33:11.548 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] USER_EVENT: [Handler Terminated]
5/17/18 5:33:11.548 Error publishing analytic events
javax.net.ssl.SSLException: SSLEngine closed already
	at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source)
5/17/18 5:33:11.547 <= [500] POST "https://xyz.net/v1/events" 2ms
5/17/18 5:33:11.546 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] FLUSH
5/17/18 5:33:11.546 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] WRITE: 299B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ |00000000| 50 4f 53 54 20 2f 76 31 2f 65 76 65 6e 74 73 20 |POST /v1/events | |00000010| 48 54 54 50 2f 31 2e 31 0d 0a 75 73 65 72 2d 61 |HTTP/1.1..user-a| |00000020| 67 65 6e 74 3a 20 52 65 61 63 74 6f 72 4e 65 74 |gent: ReactorNet| |00000030| 74 79 2f 30 2e 37 2e 37 2e 52 45 4c 45 41 53 45 |ty/0.7.7.RELEASE| |00000040| 0d 0a 74 72 61 6e 73 66 65 72 2d 65 6e 63 6f 64 |..transfer-encod| |00000050| 69 6e 67 3a 20 63 68 75 6e 6b 65 64 0d 0a 68 6f |ing: chunked..ho| |00000060| 73 74 3a 20 6d 67 61 73 2e 70 72 6f 64 2e 70 75 |st: mgas.prod.pu| |00000070| 62 6c 69 63 2e 61 74 6c 2d 70 61 61 73 2e 6e 65 |blic.atl-paas.ne| |00000080| 74 0d 0a 41 63 63 65 70 74 3a 20 61 70 70 6c 69 |t..Accept: appli| |00000090| 63 61 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 |cation/json;char| |000000a0| 73 65 74 3d 55 54 46 2d 38 0d 0a 58 2d 42 33 2d |set=UTF-8..X-B3-| |000000b0| 54 72 61 63 65 49 64 3a 20 62 32 34 34 33 32 30 |TraceId: b244320| |000000c0| 35 39 63 33 30 66 30 64 30 0d 0a 58 2d 42 33 2d |59c30f0d0..X-B3-| |000000d0| 53 70 61 6e 49 64 3a 20 62 32 34 34 33 32 30 35 |SpanId: b2443205| |000000e0| 39 63 33 30 66 30 64 30 0d 0a 58 2d 42 33 2d 53 |9c30f0d0..X-B3-S| |000000f0| 61 6d 70 6c 65 64 3a 20 30 0d 0a 43 6f 6e 74 65 |ampled: 0..Conte| |00000100| 6e 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 |nt-Type: applica| |00000110| 74 69 6f 6e 2f 6a 73 6f 6e 3b 63 68 61 72 73 65 |tion/json;charse| |00000120| 74 3d 55 54 46 2d 38 0d 0a 0d 0a |t=UTF-8.... | +--------+-------------------------------------------------+----------------+
5/17/18 5:33:11.546 [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] Writing object DefaultHttpRequest(decodeResult: success, version: HTTP/1.1) POST /v1/events HTTP/1.1 user-agent: ReactorNetty/0.7.7.RELEASE transfer-encoding: chunked host: xyz.net Accept: application/json;charset=UTF-8 X-B3-TraceId: b24432059c30f0d0 X-B3-SpanId: b24432059c30f0d0 X-B3-Sampled: 0 Content-Type: application/json;charset=UTF-8
5/17/18 5:33:11.546 [HttpClient] [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] handler is being applied: HttpClientHandler{startURI=https://xyz.net/v1/events, method=POST, handler=org.springframework.http.client.reactive.ReactorClientHttpConnector$$Lambda$1325/830898785@60bee0e3}
5/17/18 5:33:11.545 Acquired active channel: [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443]
5/17/18 5:33:11.545 Acquired [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443], now 1 active connections
5/17/18 5:33:11.545 Acquiring existing channel from pool: DefaultPromise@45009327(success: [id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443]) SimpleChannelPool{activeConnections=0}
5/17/18 5:33:11.545 => POST "https://xyz.net/v1/events"
5/17/18 5:33:11.540 PM	[id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] READ COMPLETE
5/17/18 5:33:11.540 PM	[id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] USER_EVENT: SslCloseCompletionEvent(SUCCESS)

It looks like the ssl connection gets closed but the channel doesn't.

5/17/18 5:33:11.540 PM	[id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] READ COMPLETE
5/17/18 5:33:11.540 PM	[id: 0x6563e819, L:/10.255.0.3:59388 - R:xyz.net/52.3.191.109:443] USER_EVENT: SslCloseCompletionEvent(SUCCESS)

Looking at previous logs following a SslCloseCompletionEvent we can see the channel being unregistered

5/17/18 5:30:47.852 [id: 0x8f4fc18c, L:/10.255.0.3:44856 ! R:xyz.net/34.230.139.16:443] UNREGISTERED
5/17/18 5:30:47.852 [id: 0x8f4fc18c, L:/10.255.0.3:44856 ! R:xyz.net/34.230.139.16:443] INACTIVE
5/17/18 5:30:47.852 [id: 0x8f4fc18c, L:/10.255.0.3:44856 ! R:xyz.net/34.230.139.16:443] CLOSE
5/17/18 5:30:47.851 [id: 0x8f4fc18c, L:/10.255.0.3:44856 - R:xyz.net/34.230.139.16:443] READ COMPLETE
5/17/18 5:30:47.770 [id: 0x8f4fc18c, L:/10.255.0.3:44856 - R:xyz.net/34.230.139.16:443] READ COMPLETE
5/17/18 5:30:47.770 [id: 0x8f4fc18c, L:/10.255.0.3:44856 - R:xyz.net/34.230.139.16:443] USER_EVENT: SslCloseCompletionEvent(SUCCESS)

[madgnome]

@violetagg After much trouble, I was able to create a convoluted test case reproducing the issue:

madgnome@a5acdb3

[madgnome]

A potential workaround (without having to disable pooling entirely) is to explicitly close the channel on SSL close:

HttpClient.create(
        opt -> applyHostAndPortFromContext(opt, context)
            .afterChannelInit(channel -> {
              final SslHandler sslHandler = channel.pipeline().get(SslHandler.class);
              sslHandler.sslCloseFuture().addListener(
                  (GenericFutureListener<Future<Channel>>) future -> {
                    if (future.isSuccess()) {
                      final Channel c = future.getNow();
                      if (c.isActive()) {
                        c.close();
                      }
                    }
                  });
            })
            .sslContext(sslClient))

[bmorganatlas]

We are also seeing the same symptoms, small numbers of requests fail with the message

java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest

[violetagg]

Please try the current snapshot, it is available on the http://repo.spring.io/snapshot

[madgnome]

@violetagg I've tried the current snapshot, let it soak for 1 day and couldn't reproduce the initial issue!
Awesome job!