Some spike elapsed time of ANN benchmark

[zzong2006]

I try some (personal) benchmark on distributed HNSW index.
Deploy 5 pods in k8s and each pod has 2gb RAM / 10 CPU cores.

During benchmark, I noticed some spike on elapsed time on query but cannot figure out.

Here is some configuration / scripts in bench

DATA_SIZE = 2000
QUERY_SIZE = 100
DIM = 256
TOP_K = 100
client = weaviate.Client("http://localhost:8080") 
samples = np.random.randn(DATA_SIZE, DIM).astype("float32")
....

for k in range(QUERY_SIZE):
    start_time = time.time()
    target_query = {"vector": samples[k]}

    result = (client.query.get(TARGET_CLASS, ["index_id"]).with_near_vector(target_query).with_limit(TOP_K).do())
    end_time = time.time()
    elapsed = 1000. * (end_time - start_time)
    print(f"{k}: Elapsed", elapsed)

trial (1)

....
91: Elapsed 8.621692657470703
92: Elapsed 8.2550048828125
93: Elapsed 7.802248001098633
94: Elapsed 86.1060619354248
95: Elapsed 7.91478157043457
96: Elapsed 81.93159103393555
97: Elapsed 78.98426055908203  # why this happened?
....

trial (2)

91: Elapsed 9.263992309570312
92: Elapsed 87.82005310058594
93: Elapsed 8.857011795043945
94: Elapsed 88.12785148620605
95: Elapsed 9.41157341003418
96: Elapsed 87.73326873779297
97: Elapsed 8.994817733764648  # same vector as previous trial but doesn't show spike this time.

schema

scahem.get {
    "class": "RandomVectors",
    "invertedIndexConfig": {
        "bm25": {
            "b": 0.75,
            "k1": 1.2
        },
        "cleanupIntervalSeconds": 60,
        "stopwords": {
            "additions": null,
            "preset": "en",
            "removals": null
        }
    },
    "properties": [
        {
            "dataType": [
                "int"
            ],
            "indexInverted": false,
            "name": "index_id"
        }
    ],
    "replicationConfig": {
        "factor": 1
    },
    "shardingConfig": {
        "virtualPerPhysical": 128,
        "desiredCount": 1,
        "actualCount": 1,
        "desiredVirtualCount": 128,
        "actualVirtualCount": 128,
        "key": "_id",
        "strategy": "hash",
        "function": "murmur3"
    },
    "vectorIndexConfig": {
        "skip": false,
        "cleanupIntervalSeconds": 300,
        "maxConnections": 48,
        "efConstruction": 128,
        "ef": -1,
        "dynamicEfMin": 100,
        "dynamicEfMax": 500,
        "dynamicEfFactor": 8,
        "vectorCacheMaxObjects": 1000000000000,
        "flatSearchCutoff": 0,
        "distance": "dot"
    },
    "vectorIndexType": "hnsw",
    "vectorizer": "none"
}

I changed shardingConfig from 1 to 5 but spike always happened.

[zzong2006]

Plus, I found that increasing number of replication makes elapsed time also.

[etiennedi]

Hey, thanks for your intereting in trying out - and benchmarking - Weaviate.

For benchmarks to be meaningful, you need to use an actual dataset with a more realistic distribution than random vectors. See @nreimers' excellent Twitter thread Why ANN benchmarking on random embeddings is a bad idea.

I try some (personal) benchmark on distributed HNSW index.
Deploy 5 pods in k8s and each pod has 2gb RAM / 10 CPU cores.

With those relatively small resources, there is no need for sharding. A single pod with the combined resources will perform much better. The ratio of CPUs to RAM also seems quite uncommon. Typically we would see something like 4GB of memory per 1 CPU.

That said, I'm not sure if any of the above would explain those spikes. Could it be that the script itself introduces latencies? I vaguely remember for loops being problematic in python (cc @StefanBogdan)? Or could it be that the python GC starts running in some iterations, but not in others?

If you look at our benchmarks, you can see that mean and p99 tend to be quite close to one another. And that is done with massive concurrency:

[zzong2006]

Thank you for responding.

I noticed your benchmarks run with Go script (especially in speed? test).

So I made Go script do with same query as previous python one (my comment above).

cfg := weaviate.Config{
    Host:   "localhost:8080",
    Scheme: "http",
}
client := weaviate.New(cfg)

for index, query_vec := range q {
    start := time.Now()
    nearVector := client.GraphQL().NearVectorArgBuilder().WithVector(query_vec)

    ctx := context.Background()
    _, err := client.GraphQL().Get().WithClassName("RandomVectors").WithFields(name, _additional).WithNearVector(nearVector).Do(ctx)

    if err != nil {
        panic(err)
    }
    elapsed := time.Since(start)
    log.Printf("%d Single Query took %v", index, elapsed)
}

And it works like a charm without spike.

2023/02/15 13:52:11 91 Single Query took 5.481889ms
2023/02/15 13:52:11 92 Single Query took 5.732956ms
2023/02/15 13:52:11 93 Single Query took 5.568925ms
2023/02/15 13:52:11 94 Single Query took 5.675747ms
2023/02/15 13:52:11 95 Single Query took 5.75065ms
2023/02/15 13:52:11 96 Single Query took 6.083298ms
2023/02/15 13:52:11 97 Single Query took 9.029891ms

As you said, the loop of Python script may problematic.

---

When the number of sharding is increased (and +data size 2,000 -> 20,000), python performance goes much worse. I think python client has some weak points in distributed environment (?).

python trial

91: Elapsed 80.45363426208496
92: Elapsed 167.67144203186035
93: Elapsed 14.807462692260742
94: Elapsed 74.05209541320801
95: Elapsed 17.160415649414062
96: Elapsed 12.421846389770508
97: Elapsed 79.99849319458008

go trial

2023/02/15 14:29:50 91 Single Query took 8.901125ms
2023/02/15 14:29:50 92 Single Query took 9.26966ms
2023/02/15 14:29:50 93 Single Query took 8.958153ms
2023/02/15 14:29:50 94 Single Query took 8.935988ms
2023/02/15 14:29:50 95 Single Query took 8.942517ms
2023/02/15 14:29:50 96 Single Query took 9.135497ms
2023/02/15 14:29:50 97 Single Query took 10.233201ms

[etiennedi]

Thanks for coming batch with your results. Great to hear that using the Go script you no longer see any latency spikes.

When the number of sharding is increased (and +data size 2,000 -> 20,000), python performance goes much worse. I think python client has some weak points in distributed environment (?).

Let me also loop @dirkkul here. I think the python client should not know whether a setup is distributed or not. But I'm not sure. So maybe the larger size is the culprit here. Either way this is an interesting finding and maybe there is something we can improve about the python client. Thank you for sharing your results.

[dirkkul]

Looks like a python client issue, I'll transfer it over to our repo

[dirkkul]

@zzong2006
Thanks for bringing this up. Does this only happen on distributed setups?

[zzong2006]

@dirkkul Well.. I guess not.

Whether I deployed single or multiple(>5) weaviate pods (and adjust num of sharding accordingly), I experienced same issue .
But I didn't try on local docker setup.

[dirkkul]

Finally had time to look into this.

I noticed two things:

1. The times do not seem to depend on the vector that is send. Eg if you send exactly the same queries several times, the times are completely different
2. Almost all the time is spend in the request library that is outside of our control (Profiling results). I assume that that library soetimes just takes longer :(

Sadly, I don't think that theres anything we can do here. Parallelizing requests is on the "we should do that at some point"-list, but I' not sure when we have time for that.

Closing for now.