pulsar-admin connection to Pulsar Proxy is unstable when TLS is enabled, function jar uploads fail

[lhotari]

Describe the bug

pulsar-admin connection to Pulsar Proxy is unstable when TLS is enabled. Large function jar uploads fail frequently.

pulsar-admin output:

Creating func1623409183
Warning: Nashorn engine is planned to be removed from a future JDK release
null

Reason: java.util.concurrent.CompletionException: io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown

and logs in broker logs

javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: org.eclipse.jetty.io.EofException: Early EOF
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162) ~[org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) ~[org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279) [org.eclipse.jetty-jetty-server-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.eclipse.jetty-jetty-io-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540) [org.eclipse.jetty-jetty-io-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395) [org.eclipse.jetty-jetty-io-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) [org.eclipse.jetty-jetty-io-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.eclipse.jetty-jetty-io-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [org.eclipse.jetty-jetty-io-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [org.eclipse.jetty-jetty-util-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [org.eclipse.jetty-jetty-util-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [org.eclipse.jetty-jetty-util-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [org.eclipse.jetty-jetty-util-9.4.39.v20210325.jar:9.4.39.v20210325]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) [org.eclipse.jetty-jetty-util-9.4.39.v20210325.jar:9.4.39.v20210325]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

To Reproduce

See https://github.com/lhotari/pulsar-playground/tree/master/proxy-tls-issue-repro-2.7.2

Expected behavior

pulsar-admin should be stable

Additional context

Caused by Jetty bug is SSL buffering, jetty/jetty.project#6152 (comment)