Set default HNSW "ef_construction" to 64

[jkatz]

The "ef_construction" parameter for HNSW controls how many nearest-neighbors are considered when inserting a vector into the index. A higher value of ef_construction generally improves recall when querying the index, but at the tradeoff of increasing index build times.

This patch sets the default value of ef_construction to 64. This takes into account several different testing methodologies to determine what makes sense for this HNSW implementation, including:

• Performance/recall of queries relative to different ef_construction values
• Impact on index build times

to optimize what will give users a good performance/recall ratio while accounting for the amount of time it takes to build an index.

For example, using the dbpedia-openai-1000k-angular dataset from ANN Benchmarks[1], we can see the impact of recall and index build times with doubling values of ef_construction (values in parentheses are percentages over the preceding value):

Build Times

• ef_construction=32: 56min
• ef_construction=64: 71min (+26%)
• ef_construction=128: 95min (+35%)

Recall @ ef_search=10

• ef_construction=32: 0.7745
• ef_construction=64: 0.8185 (5.68%)
• ef_construction=128: 0.8442 (3.14%)

Recall @ ef_search=20

• ef_construction=32: 0.8596
• ef_construction=64: 0.8960 (4.24%)
• ef_construction=128: 0.9192 (2.58%)

The results show that increasing ef_consturction does increase build times, but we begin to see diminishing returns in the amount of recall of the builds at higher levels in comparison to the effort it takes to build the index. However, at ef_construciton=64, though we pay a bit more effort up front, we do see a measurable impact on recall that can substantiate defaulting ef_construction to 64.

For comparison, here is the gist-960-euclidean dataset, which has presented many other algorithms challanges with achieving good recall at lower search levels[1]. We can draw similar conclusions to the previous data set:

Build Time

• ef_construction=32: 57min
• ef_construction=64: 68min (+20%)
• ef_construction=128: 86min (+26%)
• ef_construction=256: 121min (+39%)
• ef_construction=512: 181min (+50%)

Recall @ ef_search=10

• ef_construction=32: 0.4346
• ef_construction=64: 0.4734 (8.93%)
• ef_construction=128: 0.4974 (5.07%)
• ef_construction=256: 0.5024 (1.01%)
• ef_construction=512: 0.5069 (0.90%)

Recall @ ef_search=20

• ef_construction=32: 0.5562
• ef_construction=64: 0.6140 (10.39%)
• ef_construction=128: 0.6401 (4.25%)
• ef_construction=256: 0.6443 (0.66%)
• ef_construction=512: 0.6481 (0.59%)

Recall @ ef_search=40

• ef_construction=32: 0.6898
• ef_construction=64: 0.7392 (7.16%)
• ef_construction=128: 0.7681 (3.91%)
• ef_construction=256: 0.7795 (1.48%)
• ef_construction=512: 0.7875 (1.03%)

To recap, the slight increase in build time under ef_construction=64 does yield much higher recall, whereas beyond this value the build time diminishes.

The last thing to consider is the behavior when inserting into an empty index, or inserts as we go. While doing this kind of build will generally be slower, we want to ensure that the chosen value of ef_construction will have a slowdown proportional to building the index in bulk. Again, here we use the dbpedia-openai-1000k-angular data set to look at these differences:

Empty Build

• ef_construction=32: 141min
• ef_construction=64: 169min (+20%)

which is inline with the timings from the full build.

These tests were run repeatedly with a variety of the ANN Benchmarks, including[1]:

• mnist-784-euclidean
• sift-128-euclidean
• glove-100-angular
• glove-25-angular
• gist-960-euclidean
• dbpedia-openai-1000k-angular

and additional data can be furnished upon request.

[1] https://github.com/erikbern/ann-benchmarks

[ankane]

Thanks @jkatz