“Metric not found or supported” on load tests for GPU indexes

[bobtoalice]

Hello,

I’m attempting to run load tests for a GPU-enabled Milvus cluster and I’ve been using IP as a metric. However, the tests always result in an error that tells me L2/IP are either nonexistent or not supported. The error message indicates that Cosine is being used. Would it be possible to specify the metric type from the UI?

This is the error in full:

RPC error: [create_index], <MilvusException: (code=65535, message=metric type not found or not supported, supported: [L2 IP])>, <Time:{'RPC start': '2024-05-22 00:10:32.288218', 'RPC error': '2024-05-22 00:10:32.300662'}>
2024-05-22 00:10:32,300 | WARNING: pre run case error: <MilvusException: (code=65535, message=metric type not found or not supported, supported: [L2 IP])> (task_runner.py:92) (3138906)
2024-05-22 00:10:32,300 | WARNING: [1/1] case {'label': <CaseLabel.Performance: 2>, 'dataset': {'data': {'name': 'Cohere', 'size': 1000000, 'dim': 768, 'metric_type': <MetricType.COSINE: 'COSINE'>}}, 'db': 'Milvus-default'} failed to run, reason=<MilvusException: (code=65535, message=metric type not found or not supported, supported: [L2 IP])>

Thanks!

[alwayslove2013]

@bobtoalice Thank you for bringing this issue to our attention. We appreciate you taking the time to identify this limitation in the GPU Index project.

You're right that the current GPU-Index only supports IP and L2 metrics, but the Cohere / OpenAI datasets (provided by VectorDBBench) rely on Cosine similarity. Performing an additional normalization step to handle this discrepancy is a sensible approach.

I will try to fix this issue as soon as possible, and let you know once it's done.

[alwayslove2013]

@bobtoalice The latest code has resolved it. We will publish the updated version to PyPI soon. In the meantime, you can test the changes using the following steps.

git clone https://github.com/zilliztech/VectorDBBench.git
cd VectorDBBench
pip install -e ".[test]"
init_bench