-
Notifications
You must be signed in to change notification settings - Fork 183
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ENHANCEMENT] disk persistent storage for indexes and documents #296
Comments
Disk storage is supported by default the form of a Docker Volume. The quick start guide commands are to run a Marqo instance for the first time. If you want to access the same Marqo instance after restarting your computer, follow the starting and stopping guide in Marqo: https://docs.marqo.ai/0.0.12/starting_and_stopping/ If there are persistence issues because of your cloud computing environment (for example, if you are using SageMaker), you can change the Docker storage location: https://docs.marqo.ai/0.0.12/Advanced-Usage/change_storage_location/ |
Also, if you want to transfer Marqo's state to a new Marqo container (for example, a version update), follow this guide: https://docs.marqo.ai/0.0.12/Advanced-Usage/transferring_state/ |
can i config the disk storage path? I mean the disk storage path of document, not docker itself @pandu-k |
in other words, docker contaner store in alternative places(e.g. /var/lib/docker), the text or image codes(so-called document) store in fixed disk storage(e.g. /mnt/disk1). |
hi @achao2013 , would you be able to provide some more details? To change the docker storage location you can use this https://docs.marqo.ai/0.0.12/Advanced-Usage/change_storage_location/ . The images can live in another location and only the corresponding embeddings will be stored in marqo-os. For text, the original will be stored within marqo-os along with the embeddings. To summarise, pointers to images can be used but at the moment the original text will be also stored and pointer only for text is not supported. Does that help answer? |
If i want to store the image or text embeddings in the disk, not the marqo-os in docker, does the design of marqo support it or how can i edit the marqo code to implement this function? @jn2clark |
you can run the backend (opensearch) outside of the marqo docker. this means the opensearch volume can persist without the marqo docker. see the developer guide here https://github.com/marqo-ai/marqo/tree/mainline/src/marqo. option C is what you want. just make sure that opensearch is started first |
thanks, It's getting close to what I want. Further down the line, is there a specific location in the code where you can set the disk storage path of the opensearch volume. @jn2clark |
|
if it's a configurable disk path, i think it works. |
Is your feature request related to a problem? Please describe.
The document is cached in docker.If docker is down, you need to run the database again. Can we support disk storage?
Describe the solution you'd like
provide a config which can set database storage path in the disk which has maped into docker.
Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered: