Multimodal search

[DimasfromLavoisier]

This is a PR for a new version f a multimodal search module. Few things left until its final version:

1. Tests
   2. A method for multi-query search
   3. Small fixes according copilot and other AI reviews
   4. Update demo notebook

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sonarqubecloud]

Quality Gate passed

Issues
10 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[iulusoy]

I'm having some issues with memory on the GPU. Is it possible that there is a memory leak in the image encoding when building the FAISS?
Maybe you can try allocating your local GPU memory to something else largely, to reduce the amount of memory ammico can use, to reproduce the issue; or use much more data.

[iulusoy]

here my GPU runs out of memory and it fails with

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 464.00 MiB. GPU 0 has a total capacity of 5.55 GiB of which 337.88 MiB is free. Including non-PyTorch memory, this process has 5.20 GiB memory in use. Of the allocated memory 4.57 GiB is allocated by PyTorch, and 553.08 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

maybe here we can add a try - except block to catch this error and fallback to CPU?
The problem starts in

ammico/multimodal_search.py:237: in index_images
    embeddings = self.model.encode_image(
../../../miniforge3/envs/ammico/lib/python3.13/site-packages/torch/utils/_contextlib.py:120: in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
ammico/model.py:285: in encode_image
    embeddings = self.model.encode(
../../../miniforge3/envs/ammico/lib/python3.13/site-packages/torch/utils/_contextlib.py:120: in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
../../../miniforge3/envs/ammico/lib/python3.13/site-packages/sentence_transformers/SentenceTransformer.py:1094: in encode
    out_features = self.forward(features, **kwargs)
    ...

[iulusoy]

If I set

def test_multimodal_search_combined_query(get_path):
    model = MultimodalEmbeddingsModel(device="cpu")
    mms = MultimodalSearch(model=model)

the test runs fine.

[iulusoy]

Here also the kernel crashes, I assume it is the same memory problem. I suspect that it is due to a memory leak, torch maybe not releasing memory when it should..? Otherwise, why does the memory use accumulate so much during the run? (The first few encodings are usually fine.)

[iulusoy]

Also, somehow the code coverage is not showing, I assume this is because the PR is opened from a branch.
Other than the memory issue, the implementation looks very good.

[iulusoy]

With the other faiss library version, I also could not get it to run locally. But since it runs fine on the CPU, I would postpone this to the testing stage and merge the PR now.