Memgraph Custom Query Modules

This repository provides three lightweight Python query modules for Memgraph that integrate external storage and vector search systems with Memgraph's query module system.

Included modules:

• query_modules/elastic.py — helpers to retrieve document text and embeddings from Elasticsearch.
• query_modules/mindb.py — a read-only procedure that queries a small/low-memory vector DB (MindB) and returns external IDs with similarity scores.
• query_modules/qdrant.py — a read-only procedure that queries a Qdrant collection and (optionally) filters by a payload label.

These modules are implemented using Memgraph's mgp Python API and are intended to be deployed inside Memgraph's query modules directory or mounted into a Memgraph container.

Why these modules

• Keep Memgraph memory usage low by storing large text and embeddings externally (Elasticsearch) while keeping only essential metadata inside the graph.
• Use a lightweight vector DB (MindB) for very low-memory nearest-neighbor search, or Qdrant for more feature-rich vector search with payload filtering.
• Provide simple, reusable procedures/functions that return consistent records suitable for joining with graph data.

Repository layout

query_modules/
    elastic.py
    mindb.py
    qdrant.py
README.md

Requirements

• Python environment used by Memgraph query modules (these modules use mgp provided by Memgraph).
• Additional Python packages required by the modules (install in the same environment Memgraph loads query modules from):

pip install requests elasticsearch

Note: mgp is not a pip package — it is provided by Memgraph's runtime. Make sure Memgraph is configured to load Python query modules.

Environment variables and hosts

• elastic.py uses ES_HOST constant set to http://elasticsearch:9200 by default. Adjust the host in the source or ensure a host named elasticsearch is resolvable (e.g., Docker Compose service name).
• qdrant.py reads QDRANT_URL from the environment (default http://qdrant:6333). Set QDRANT_URL if your Qdrant instance runs elsewhere.
• mindb.py calls http://mindb:8019 by default. Adjust the URL in mindb.py if you run MindB on a different host or port.

Common deployment pattern is to run Memgraph, Elasticsearch, Qdrant and MindB as services in the same Docker network and use service names (e.g., elasticsearch, qdrant, mindb) so the default host strings work without changes.

Module details and usage

1. query_modules/elastic.py

• Exposes two functions via mgp.function:

  • get_text(doc_id: str, index_name: str) -> str — returns the text field (or None).
  • get_embedding(doc_id: str, index_name: str) -> Any — returns the embedding field (or None).

• How it works: it uses the Elasticsearch Python client and retrieves a document by ID using es.get(index=index_name, id=doc_id) and returns the relevant field.

• Example usage in Memgraph Cypher:

RETURN elastic.get_text('document-id-123', 'my_index') AS text

If you want to fetch embeddings and use them in-memory for a nearest-neighbor routine or to pass to other procedures, call get_embedding the same way.

2. query_modules/mindb.py

• Exposes one read-only procedure via mgp.read_proc:

  • mindb_search(db_name: str, query_vector: list, final_top_k: int = 100) -> record(external_id=str, score=float)

• Behavior: sends a POST request to MindB at /db/{db_name}/query with a payload containing query_vector, preliminary_top_k and final_top_k. Expects a JSON response with metadata (list) and cosine_similarity (list). Returns a list of mgp.Record(external_id, score).

• Reference: MindB project and source code available at https://github.com/D-Star-AI/minDB

• Example Memgraph usage:

CALL mindb.mindb_search('my_vector_db', [0.12, 0.04, ...], 10)
YIELD external_id, score
RETURN external_id, score
LIMIT 10

• Notes:
  • external_id is taken from metadata[i]['id'] as returned by MindB.
  • The procedure expects the MindB API to return consistent lists for metadata and cosine_similarity.

3. query_modules/qdrant.py

• Exposes one read-only procedure via mgp.read_proc:

  • qdrant_search(collection_name: str, query_vector: list, label_filter: list = None, limit: int = 20) -> record(external_id=str, score=float)

• Behavior: performs an HTTP POST to Qdrant's /collections/{collection_name}/points/search endpoint. The request payload includes vector, limit, with_payload, and a filter with a must match on the label key when label_filter is provided.

• The returned results are parsed. Each hit's payload should contain original_id which is returned as external_id and the hit score is returned as score.

• Example Memgraph usage:

CALL qdrant.qdrant_search('articles', [0.03, 0.01, ...], ['news'], 25)
YIELD external_id, score
RETURN external_id, score
LIMIT 25

• Notes and expectations:
  • Qdrant's payload entries should include original_id if your application needs a stable external identifier.
  • The procedure reads QDRANT_URL from the environment; set it when running Memgraph if Qdrant is remote.

Examples: joining vector hits with graph nodes

If you store a property on nodes that holds an external ID that matches external_id returned by the vector DBs, you can join results with nodes in Memgraph.

Example:

CALL qdrant.qdrant_search('articles', [0.1,0.2,0.3], NULL, 10)
YIELD external_id, score
MATCH (d:Document {id: external_id})
RETURN d, score
ORDER BY score DESC
LIMIT 10

Troubleshooting

• Timeouts / connection errors: requests to external services use short timeouts (5–10s). Verify network reachability and correct hostnames/ports.
• Invalid response formats: MindB and Qdrant procedures expect specific JSON structures. If you get parsing errors, inspect raw responses by running the service endpoints with curl or Postman.
• Elasticsearch mismatches: get_text and get_embedding expect _source to contain text and embedding keys. Adjust retrieve_text_by_id or map your index mapping accordingly.

Recommended requirements.txt

requests
elasticsearch

Add this file to the environment used by Memgraph's Python module loader if you maintain a custom Python environment. If Memgraph is running in a container, add these packages to the container image.

Deployment tips

• Docker Compose: run Memgraph, Elasticsearch, Qdrant, and MindB in the same Docker network and use the service names shown in defaults so the code works without changes.
• Mount query_modules/ into Memgraph's module directory (/usr/lib/memgraph/query_modules/) or use Memgraph startup option --query-modules-directory to point to a custom location.