[BUG] Server throws 500 on ValueError when vector dimensions are mismatched

[shabani1]

Describe the bug

If you create an embedding field with the wrong dims, it causes a 500 in the query endpoint.

To Reproduce
Steps to reproduce the behavior:

Create an index with an embedding field that has the wrong value for dims, and then query against it.

index_fields = {
    "embedding": {"type": "embedding", "extras": {"dims": 1536, "model": "text.embeddings.minilm"}},
}
idx = lx.create_index(
    index_id='bug_example',
    description='bug',
    index_fields=index_fields
)

idx.query(query_text="hi")

Additional context
Stacktrace from Vishal below.

2024-04-10 10:35:50 The above exception was the direct cause of the following exception:
2024-04-10 10:35:50 
2024-04-10 10:35:50 Traceback (most recent call last):
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
2024-04-10 10:35:50     result = await app(  # type: ignore[func-returns-value]
2024-04-10 10:35:50              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
2024-04-10 10:35:50     return await self.app(scope, receive, send)
2024-04-10 10:35:50            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2024-04-10 10:35:50     await super().__call__(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
2024-04-10 10:35:50     await self.middleware_stack(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
2024-04-10 10:35:50     raise exc
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
2024-04-10 10:35:50     await self.app(scope, receive, _send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
2024-04-10 10:35:50     await self.app(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2024-04-10 10:35:50     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-04-10 10:35:50     raise exc
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-04-10 10:35:50     await app(scope, receive, sender)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
2024-04-10 10:35:50     await self.middleware_stack(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
2024-04-10 10:35:50     await route.handle(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
2024-04-10 10:35:50     await self.app(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
2024-04-10 10:35:50     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-04-10 10:35:50     raise exc
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-04-10 10:35:50     await app(scope, receive, sender)
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
2024-04-10 10:35:50     response = await func(request)
2024-04-10 10:35:50                ^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
2024-04-10 10:35:50     raw_response = await run_endpoint_function(
2024-04-10 10:35:50                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
2024-04-10 10:35:50     return await dependant.call(**values)
2024-04-10 10:35:50            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/home/app/lexy/api/endpoints/index_records.py", line 161, in query_records
2024-04-10 10:35:50     search_result = await session.exec(
2024-04-10 10:35:50                     ^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlmodel/ext/asyncio/session.py", line 83, in exec
2024-04-10 10:35:50     result = await greenlet_spawn(
2024-04-10 10:35:50              ^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 190, in greenlet_spawn
2024-04-10 10:35:50     result = context.switch(*args, **kwargs)
2024-04-10 10:35:50              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlmodel/orm/session.py", line 68, in exec
2024-04-10 10:35:50     results = super().execute(
2024-04-10 10:35:50               ^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2306, in execute
2024-04-10 10:35:50     return self._execute_internal(
2024-04-10 10:35:50            ^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2191, in _execute_internal
2024-04-10 10:35:50     result: Result[Any] = compile_state_cls.orm_execute_statement(
2024-04-10 10:35:50                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
2024-04-10 10:35:50     result = conn.execute(
2024-04-10 10:35:50              ^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1422, in execute
2024-04-10 10:35:50     return meth(
2024-04-10 10:35:50            ^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 514, in _execute_on_connection
2024-04-10 10:35:50     return connection._execute_clauseelement(
2024-04-10 10:35:50            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1644, in _execute_clauseelement
2024-04-10 10:35:50     ret = self._execute_context(
2024-04-10 10:35:50           ^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1825, in _execute_context
2024-04-10 10:35:50     self._handle_dbapi_exception(
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2357, in _handle_dbapi_exception
2024-04-10 10:35:50     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
2024-04-10 10:35:50     context = constructor(
2024-04-10 10:35:50               ^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 1452, in _init_compiled
2024-04-10 10:35:50     l_param: List[Any] = [
2024-04-10 10:35:50                          ^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 1454, in <listcomp>
2024-04-10 10:35:50     flattened_processors[key](compiled_params[key])
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/pgvector/sqlalchemy/__init__.py", line 23, in process
2024-04-10 10:35:50     return to_db(value, self.dim)
2024-04-10 10:35:50            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/pgvector/utils/__init__.py", line 35, in to_db
2024-04-10 10:35:50     raise ValueError('expected %d dimensions, not %d' % (dim, len(value)))
2024-04-10 10:35:50 sqlalchemy.exc.StatementError: (builtins.ValueError) expected 1536 dimensions, not 384
2024-04-10 10:35:50 [SQL: SELECT zzidx__splunkdemo_index.document_id, zzidx__splunkdemo_index.custom_id, zzidx__splunkdemo_index.meta, zzidx__splunkdemo_index.index_record_id, abs(zzidx__splunkdemo_index.embedding <-> $1) AS abs_distance, pow(zzidx__splunkdemo_index.embedding <-> $2, $3::INTEGER) AS distance, document_tbl.content AS "document.content" 
2024-04-10 10:35:50 FROM zzidx__splunkdemo_index JOIN documents AS document_tbl ON zzidx__splunkdemo_index.document_id = document_tbl.document_id ORDER BY distance ASC 
2024-04-10 10:35:50  LIMIT $4::INTEGER]
2024-04-10 10:35:50 [parameters: [{}]]
2024-04-10 10:35:50 INFO:     192.168.65.1:17674 - "POST /api/bindings HTTP/1.1" 201 Created
2024-04-10 10:35:50 INFO:     192.168.65.1:17674 - "POST /api/indexes/splunkdemo_index/records/query?query_field=embedding&k=5&return_document=false HTTP/1.1" 500 Internal Server Error
2024-04-10 10:35:50 ERROR:    Exception in ASGI application
2024-04-10 10:35:50 Traceback (most recent call last):
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context
2024-04-10 10:35:50     context = constructor(
2024-04-10 10:35:50               ^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 1452, in _init_compiled
2024-04-10 10:35:50     l_param: List[Any] = [
2024-04-10 10:35:50                          ^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 1454, in <listcomp>
2024-04-10 10:35:50     flattened_processors[key](compiled_params[key])
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/pgvector/sqlalchemy/__init__.py", line 23, in process
2024-04-10 10:35:50     return to_db(value, self.dim)
2024-04-10 10:35:50            ^^^^^^^^^^^^^^^^^^^^^^
2024-04-10 10:35:50   File "/usr/local/lib/python3.11/site-packages/pgvector/utils/__init__.py", line 35, in to_db
2024-04-10 10:35:50     raise ValueError('expected %d dimensions, not %d' % (dim, len(value)))
2024-04-10 10:35:50 ValueError: expected 1536 dimensions, not 384