ktor-client performance

[brendanw]

Subsystem
ktor-client-cio:1.2.6

https://github.com/brendanw/profilehttp

Code uses both ktor client and server. I am specifically testing the cio engine.

Is your feature request related to a problem? Please describe.
I am expecting the ktor client to have similar performance to okhttp. I am seeing worse performance from the ktor client in each benchmark so far.

Describe the solution you'd like
I'd like feedback on if I am configuring the client in an unfair way or if there is a way I should restructure the benchmarks that would make for a more fair comparison.

Motivation to include to ktor
I'd like to switch to use the CIO engine with the ktor client on android as the one thread per request model seems wasteful.

[cy6erGn0m]

How did you test it? According to ours test cio worked faster

[brendanw]

@cy6erGn0m Test code with results and how to run is at https://github.com/brendanw/profilehttp

Ktor server is setup here https://github.com/brendanw/profilehttp/blob/master/server/src/main/kotlin/main.kt

Ktor client with cio engine is setup here
https://github.com/brendanw/profilehttp/blob/master/client/src/main/kotlin/ktor.kt

Okhttp client is setup here
https://github.com/brendanw/profilehttp/blob/master/client/src/main/kotlin/okhttp.kt

Here's the ktor client configuration

@UseExperimental(KtorExperimentalAPI::class)
private fun getClient(numThreads: Int = 3): HttpClient {
  return HttpClient(CIO) {
    engine {
      /**
       * Maximum number of socket connections.
       */
      maxConnectionsCount = 1000

      threadsCount = numThreads

      endpoint {
        /**
         * Maximum number of requests for a specific endpoint route.
         */
        maxConnectionsPerRoute = 100

        /**
         * Max size of scheduled requests per connection(pipeline queue size).
         */
        pipelineMaxSize = 20

        /**
         * Max number of milliseconds to keep iddle connection alive.
         */
        keepAliveTime = 20000

        /**
         * Number of milliseconds to wait trying to connect to the server.
         */
        connectTimeout = 5000

        /**
         * Maximum number of attempts for retrying a connection.
         */
        connectRetryAttempts = 5
      }
    }
  }
}

[brendanw]

@cy6erGn0m Would it be possible to publish your guys' tests? That might be a quicker way to allow me to figure out what we are doing differently and see if it is an error on my end.

[e5l]

We have some benchmarks in our repo: https://github.com/ktorio/ktor/tree/master/ktor-client/ktor-client-benchmarks. Could you make the PR with the sample?

[oleg-larshin]

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.

[marcdejonge]

We see similar performance issues with ktor client using CIO. In our system tests we were using the ktor client, but with CIO the whole test took 30 - 40 seconds. But after switching it to OkHttp it ran in about 6 - 7 seconds.

When I'm running the official benchmarks I also see that CIO is much slower right now:

io.ktor.client.benchmarks.ClientBenchmark.download | size=0, zengineName=OkHttp
Success: 2885.414 ±(99.9%) 882.758 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=0, zengineName=CIO
Success: 454.837 ±(99.9%) 113.129 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=1, zengineName=OkHttp
Success: 3556.404 ±(99.9%) 388.089 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=1, zengineName=CIO
Success: 482.755 ±(99.9%) 123.173 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=16, zengineName=OkHttp
Success: 3141.806 ±(99.9%) 187.889 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=16, zengineName=CIO
Success: 509.466 ±(99.9%) 57.166 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=32, zengineName=OkHttp
Success: 2369.638 ±(99.9%) 194.745 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=32, zengineName=CIO
Success: 469.913 ±(99.9%) 115.855 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=64, zengineName=OkHttp
Success: 1671.196 ±(99.9%) 167.898 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=64, zengineName=CIO
Success: 389.027 ±(99.9%) 80.279 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=256, zengineName=OkHttp
Success: 582.972 ±(99.9%) 51.639 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=256, zengineName=CIO
Success: 279.887 ±(99.9%) 25.967 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=1024, zengineName=OkHttp
Success: 77.138 ±(99.9%) 89.474 ops/s [Average]

io.ktor.client.benchmarks.ClientBenchmark.download | size=1024, zengineName=CIO
Success: 89.104 ±(99.9%) 23.004 ops/s [Average]

ps. We recently upgraded to Kotlin 1.4. And I'm running on macOS right now.