High variance of latency when doing a benchmark of query (GET) by id

[lilac]

How to reproduce this bug?

1. Start the server: ./tools/dev/run_dev_server.sh local-development.
2. Ingest a sample record.
3. Verify that the data is ready: curl http://localhost:8080/v1/objects/Article/f561e132-d7f0-48af-9dee-950dba0963c3

   {
     "class": "Article",
     "creationTimeUnix": 1714703782111,
     "id": "f561e132-d7f0-48af-9dee-950dba0963c3",
     "lastUpdateTimeUnix": 1714703782111,
     "properties": {
       "title": "A title"
     },
     "vectorWeights": null
   }

4. Benchmark the query by id: ab -n 20000 "http://127.0.0.1:8080/v1/objects/Article/f561e132-d7f0-48af-9dee-950dba0963c3"

What is the expected behavior?

The latency should be stable and no timeout, given the query is quite simple.

What is the actual behavior?

There is a long pause after 16000 requests are completed. Sometimes the benchmark timeouts with logs like below.

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
Completed 12000 requests
Completed 14000 requests
Completed 16000 requests
apr_socket_recv: Operation timed out (60)

Sometimes it finishes with high variance of latency.

This is ApacheBench, Version 2.3 <$Revision: 1903618 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
Completed 12000 requests
Completed 14000 requests
Completed 16000 requests
Completed 18000 requests
Completed 20000 requests
Finished 20000 requests


Server Software:
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /v1/objects/Article/f561e132-d7f0-48af-9dee-950dba0963c3
Document Length:        186 bytes

Concurrency Level:      1
Time taken for tests:   33.097 seconds
Complete requests:      20000
Failed requests:        0
Total transferred:      15980000 bytes
HTML transferred:       3720000 bytes
Requests per second:    604.28 [#/sec] (mean)
Time per request:       1.655 [ms] (mean)
Time per request:       1.655 [ms] (mean, across all concurrent requests)
Transfer rate:          471.50 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2 146.7      0   19505
Processing:     0    0   0.3      0      24
Waiting:        0    0   0.3      0      24
Total:          0    2 146.7      0   19506

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      1
 100%  19506 (longest request)

Supporting information

The Get query via graphql is normal.

query {
  Get {
    Article(where: {
      operator: Equal
      path: ["id"]
      valueText: "f561e132-d7f0-48af-9dee-950dba0963c3"
    }) {
      title
    }
  }
}

Response

{
  "data": {
    "Get": {
      "Article": [
        {
          "title": "A title"
        }
      ]
    }
  }
}

Server Version

212f77f

Code of Conduct

• I have read and agree to the Weaviate's Contributor Guide and Code of Conduct