Add VectorStore integration for Vespa

llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/.gitignore

@@ -0,0 +1,153 @@
+llama_index/_static
+.DS_Store
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+bin/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+etc/
+include/
+lib/
+lib64/
+parts/
+sdist/
+share/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+.ruff_cache
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+notebooks/
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+pyvenv.cfg
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# Jetbrains
+.idea
+modules/
+*.swp
+
+# VsCode
+.vscode
+
+# pipenv
+Pipfile
+Pipfile.lock
+
+# pyright
+pyrightconfig.json

llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/BUILD

@@ -0,0 +1,4 @@
+poetry_requirements(
+    name="poetry",
+    module_mapping={"pyvespa": ["vespa"]},
+)

llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/CHANGELOG.md

@@ -0,0 +1,9 @@
+# CHANGELOG — llama-index-vector-stores-opensearch
+
+## [0.1.2]
+
+- Adds OpensearchVectorClient as top-level import
+
+## [0.1.1]
+
+- Fixes strict equality in dependency of llama-index-core

llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/Makefile

@@ -0,0 +1,17 @@
+GIT_ROOT ?= $(shell git rev-parse --show-toplevel)
+
+help:	## Show all Makefile targets.
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[33m%-30s\033[0m %s\n", $$1, $$2}'
+
+format:	## Run code autoformatters (black).
+	pre-commit install
+	git ls-files | xargs pre-commit run black --files
+
+lint:	## Run linters: pre-commit (black, ruff, codespell) and mypy
+	pre-commit install && git ls-files | xargs pre-commit run --show-diff-on-failure --files
+
+test:	## Run tests via pytest.
+	pytest tests
+
+watch-docs:	## Build and watch documentation.
+	sphinx-autobuild docs/ docs/_build/html --open-browser --watch $(GIT_ROOT)/llama_index/

llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/README.md

@@ -0,0 +1,143 @@
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://vespa.ai/assets/vespa-ai-logo-heather.svg">
+  <source media="(prefers-color-scheme: light)" srcset="https://vespa.ai/assets/vespa-ai-logo-rock.svg">
+  <img alt="#Vespa" width="200" src="https://vespa.ai/assets/vespa-ai-logo-rock.svg" style="margin-bottom: 25px;">
+</picture>
+
+# LlamaIndex Vector_Stores Integration: Vespa
+
+[Vespa.ai](https://vespa.ai/) is an open-source big data serving engine. It is designed for low-latency and high-throughput serving of data and models. Vespa.ai is used by many companies to serve search results, recommendations, and rankings for billions of documents and users, expecting response times in the milliseconds.
+
+This integration allows you to use Vespa.ai as a vector store for LlamaIndex. Vespa has integrated support for [embedding inference](https://docs.vespa.ai/en/embedding.html), so you don't need to run a separate service for these tasks.
+
+Huggingface 🤗 embedders are supported, as well as SPLADE and ColBERT.
+
+## Abstraction level of this integration
+
+To make it really simple to get started, we provide a template Vespa application that will be deployed upon initializing the vector store. This removes some of the complexity of setting up Vespa for the first time, but for serious use cases, we strongly recommend that you read the [Vespa documentation](docs.vespa.ai) and tailor the application to your needs.
+
+## The template
+
+The provided template Vespa application can be seen below:
+
+```python
+from vespa.package import (
+    ApplicationPackage,
+    Field,
+    Schema,
+    Document,
+    HNSW,
+    RankProfile,
+    Component,
+    Parameter,
+    FieldSet,
+    GlobalPhaseRanking,
+    Function,
+)
+
+hybrid_template = ApplicationPackage(
+    name="hybridsearch",
+    schema=[
+        Schema(
+            name="doc",
+            document=Document(
+                fields=[
+                    Field(name="id", type="string", indexing=["summary"]),
+                    Field(
+                        name="metadata", type="string", indexing=["summary"]
+                    ),
+                    Field(
+                        name="text",
+                        type="string",
+                        indexing=["index", "summary"],
+                        index="enable-bm25",
+                        bolding=True,
+                    ),
+                    Field(
+                        name="embedding",
+                        type="tensor<float>(x[384])",
+                        indexing=[
+                            "input text",
+                            "embed",
+                            "index",
+                            "attribute",
+                        ],
+                        ann=HNSW(distance_metric="angular"),
+                        is_document_field=False,
+                    ),
+                ]
+            ),
+            fieldsets=[FieldSet(name="default", fields=["text", "metadata"])],
+            rank_profiles=[
+                RankProfile(
+                    name="bm25",
+                    inputs=[("query(q)", "tensor<float>(x[384])")],
+                    functions=[
+                        Function(name="bm25sum", expression="bm25(text)")
+                    ],
+                    first_phase="bm25sum",
+                ),
+                RankProfile(
+                    name="semantic",
+                    inputs=[("query(q)", "tensor<float>(x[384])")],
+                    first_phase="closeness(field, embedding)",
+                ),
+                RankProfile(
+                    name="fusion",
+                    inherits="bm25",
+                    inputs=[("query(q)", "tensor<float>(x[384])")],
+                    first_phase="closeness(field, embedding)",
+                    global_phase=GlobalPhaseRanking(
+                        expression="reciprocal_rank_fusion(bm25sum, closeness(field, embedding))",
+                        rerank_count=1000,
+                    ),
+                ),
+            ],
+        )
+    ],
+    components=[
+        Component(
+            id="e5",
+            type="hugging-face-embedder",
+            parameters=[
+                Parameter(
+                    "transformer-model",
+                    {
+                        "url": "https://github.com/vespa-engine/sample-apps/raw/master/simple-semantic-search/model/e5-small-v2-int8.onnx"
+                    },
+                ),
+                Parameter(
+                    "tokenizer-model",
+                    {
+                        "url": "https://raw.githubusercontent.com/vespa-engine/sample-apps/master/simple-semantic-search/model/tokenizer.json"
+                    },
+                ),
+            ],
+        )
+    ],
+)
+```
+
+Note that the fields `id`, `metadata`, `text`, and `embedding` are required for the integration to work.
+The schema name must also be `doc`, and the rank profiles must be named `bm25`, `semantic`, and `fusion`.
+
+Other than that you are free to modify as you see fit by switching out embedding models, adding more fields, or changing the ranking expressions.
+
+For more details, check out this Pyvespa example notebook on [hybrid search](https://pyvespa.readthedocs.io/en/latest/getting-started-pyvespa.html).
+
+## Going to production
+
+If you are ready to graduate to a production setup, we highly recommend to check out the [Vespa Cloud](https://cloud.vespa.ai/) service, where we manage all infrastructure and operations for you. Free trials are available.
+
+## Next steps
+
+There are many awesome features in Vespa, that are not exposed directly in this integration, check out [Pyvespa examples](https://pyvespa.readthedocs.io/en/latest/examples/pyvespa-examples.html) for some inspiration on what you can do with Vespa.
+
+Teasers:
+
+- Binary + Matryoshka embeddings.
+- ColBERT.
+- ONNX models.
+- XGBoost and lightGBM models for ranking.
+- Multivector indexing.
+- and much more.

llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/llama_index/vector_stores/vespa/BUILD

@@ -0,0 +1 @@
+python_sources()

llama-index-integrations/vector_stores/llama-index-vector-stores-vespa/llama_index/vector_stores/vespa/__init__.py

@@ -0,0 +1,6 @@
+from llama_index.vector_stores.vespa.base import (
+    VespaVectorStore,
+)
+from llama_index.vector_stores.vespa.templates import hybrid_template
+
+__all__ = ["VespaVectorStore", "hybrid_template"]