Mounting Local Volume for Langchain Embeddings in Docker Container

[pseudo-usama]

I have mounted my Docker container with a local volume, and embeddings are being saved locally. I need to create embeddings directly from the Langchain from_document method and save them to my local machine without using Docker.

Is there a way to mount this local folder with the Docker container so that both the container and my local environment can access and use the same embeddings?

This capability is crucial for my workflow to ensure seamless integration and accessibility of embeddings between local and containerized environments. Any guidance on how to achieve this would be greatly appreciated.

Steps to Reproduce:

Mount Docker container with a local volume.
Create embeddings using Langchain from_document method and save locally without Docker.
Attempt to access the same local folder within the Docker container.

Expected Behavior:

The Docker container should be able to access and utilize embeddings created and saved locally.

Actual Behavior:

Currently, there is no clear method to achieve this setup.
Environment:

Additional Context:

Any detailed steps or configuration examples to set up the volume mounting correctly would be very helpful.

[pseudo-usama]

@timvisee @generall Can you please look at this issue?

[joein]

@pseudo-usama it's not completely clear what you're trying to achieve and how the setup looks like

Could you describe it more detailed please?

What does your containerized environment look like, is it the same machine or a separate server?
What do you mean by your local environment? How do you create QdrantClient in your local environment?

[pseudo-usama]

Actually, i have created embeddings and saved at my local system in a folder. Now i want to mount this embedding folder with docker container next. But when i did this embedding are not being showed at the qdrant ui dashboard at 6333. Is there any method available of mounting already created embeddings folder with docker?

[joein]

How did you create them?

Unfortunately, it is still not clear what exactly you have, because there are multiple options.

Did you use langchain with qdrant? How did you create your QdrantClient? Did you use path=<path> or :memory:?
Or did you use qdrant in docker and provided a url?

[pseudo-usama]

I have created embeddings using the LangChain library from documents and saved them locally by specifying a path (path=<path>). Now, I want to mount these locally saved embeddings to a Docker container and run the container. However, when I attempt to mount the locally saved embeddings with Docker, they do not appear to be accessible within the container. Is there a way to properly mount these embeddings with Docker?

To save embeddings locally

qdrant_ind = Qdrant.from_documents(
    texts,
    # embeddings,
    hf,
    path="/Users/mac/Desktop/cdr_experimenttaion/local_embeddings_bge",
    collection_name="test_cdr",
    force_recreate=False
    # client=client
)

My yml file

version: '3.8'

services:
  qdrant:
    image: qdrant/qdrant
    container_name: test_qdrant_crd
    ports:
      - "5000:6333"
    volumes:
      - /Users/mac/Desktop/cdr_experimenttaion/local_embeddings_bge:/qdrant/storage
    restart: always # Add restart directive
    mem_limit: 6.5g # Add memory limit directive
    memswap_limit: 7.5g

[joein]

You can't do it this way, since they are not compatible

You would need to migrate your data from local qdrant to qdrant in docker

E.g., you can do that with qdrant client and .migrate method

[Faiee50]

Hi @joein ... can we migrate data from the local to Docker, is yes then how can we do this?

[joein]

E.g. you can use migrate method of QdrantClient