RAG deploymen

[gadieid]

Several Issues:
Enable a method to use without ollama docker since may interfere with running ollama instance (Changing port didn;t work, maybe need to change 11343 on docker as well, not only on host)
Add RAG models download options - each hugging face model used should be saved locally to enable easy solution for offline mode
Add offline mode - will not try to download from docker or huggingface but use local models

In ONLINE mode:
from transformers import AutoModel, AutoTokenizer

model_id = "bert-base-uncased" # Replace with your model's name

This pulls instantly from your local cache

model = AutoModel.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

This saves a clean, standalone copy without symlinks

model.save_pretrained("./my_offline_model")
tokenizer.save_pretrained("./my_offline_model")

In OFFLINE mode:
from transformers import AutoModel, AutoTokenizer

Point to the local folder path

model = AutoModel.from_pretrained("./my_offline_model", local_files_only=True)
tokenizer = AutoTokenizer.from_pretrained("./my_offline_model", local_files_only=True)

[yehudarav]

● All changes validate. Summary of what's in place:

Non-Docker Ollama — make rag-services-up-native starts Qdrant + Cassandra only, no Ollama container. Set RAG_OLLAMA_BASE_URL to your host Ollama (default http://localhost:11434
already matches). The Docker conflict is sidestepped by not running the container at all.

Local HF model cache — hera-rag-search download-models (or make rag-download-models) fetches RAG_EMBED_MODEL and save()s it under RAG_MODELS_CACHE_DIR (default ~/.pyhera/rag_models).
_embed_model() loads from that local path whenever it exists — same behavior you described with save_pretrained / load-from-folder, using SentenceTransformer.save().

Offline mode — RAG_OFFLINE_MODE=true makes _embed_model() refuse to reach the network and raise a message telling the user to run download-models first; download-models itself
refuses to run in offline mode.

Files touched: hera/utils/rag/config.py, indexer.py, init.py, hera/bin/hera-rag-search, .env.example, Makefile.

Lets see if it works.

[gadieid]

Works!
Another improvements is to give option in which LLM to use (llama3 is quite old) in ENV var or as paramters.

Another fix is required in search.py

This error occurs because the qdrant-client Python package recently removed the .search() method. In recent versions, Qdrant unified its search variants (dense, sparse, hybrid) under a single, improved .query_points() method.

To resolve this issue, you need to update the search.py file to use the new API signature.

What Changed
Method Name: .search() is now .query_points().

Parameter Name: The query_vector argument has been renamed to query.

Return Type: While .search() returned a list of points directly, .query_points() returns a QueryResponse object. You must append .points to extract the list of hits.

The Fix
Open /home/ubuntu/Development/hera/hera/utils/rag/search.py and modify the code starting at line 59.

Change the old code:

Python
hits = _qdrant().search(
settings.qdrant_collection,
query_vector=q_vec,
limit=k * 3 if source_prefix else k,
# query_filter=f (assuming 'f' is passed here based on line 57)
)
To the updated code:

Python
hits = _qdrant().query_points(
collection_name=settings.qdrant_collection,
query=q_vec,
limit=k * 3 if source_prefix else k,
query_filter=f # Keep any other original arguments you had here
).points
Once you make this adjustment and save the file, your vector search execution will run properly again.

[gadieid]

make rag-docs-server doesn't show option to ASK AI in the site
(Is there need to build the docs with RAG_ENABLED TRUE?)
note: the entries in setup.py are meanningless as we don't install hera as pacakge (bur use i with PYTHONPATH)
Attempt to add lines to mkdocs.yml
Option 2 — Manual registration in mkdocs.yml:

plugins:

• search # keep the default search plugin
• hera_rag_search # add the RAG widget

Did not made any change.

Other RAG services are operational

dockers active

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
787c5c262639 cassandra:4 "docker-entrypoint.s…" 25 hours ago Up 25 hours 7000-7001/tcp, 7199/tcp, 9160/tcp, 127.0.0.1:9042->9042/tcp hera-cassandra
85ec4a9cd832 qdrant/qdrant "./entrypoint.sh" 25 hours ago Up 25 hours 127.0.0.1:6333->6333/tcp, 6334/tcp hera-qdrant

rag-serve is up

curl http://localhost:8765/health
INFO: 127.0.0.1:55042 - "GET /health HTTP/1.1" 200 OK
{"status":"ok","model":"llama3"}

hera-rag-search gives answers