Explore Long Term Memory

[abrichr]

This task involves undestanding whether long term memory is necessary, and if so, building a minimum viable implementation, e.g:

pg_vector:

https://github.com/pgvector/pgvector
https://www.youtube.com/watch?v=aywZrzNaKjs&t=539s
https://news.ycombinator.com/item?id=35311113

Supabase? https://supabase.com/

Pinecone:

https://python.langchain.com/en/latest/ecosystem/pinecone.html

Weaviate

[nhooey]

pgvector does look promising, based on this comment:

Vector database built for scalable similarity search:
bayan1234

If the vectors are in the same database as the tabular/structured data then text to sql applications of llm's are so much more powerful. The generative models will then be able to form complex queries to find similarity as well as perform aggregation, filtering and joining across datasets. To do this today with a separate dedicated vector db is quite painful. | If the vectors are in the same database as the tabular/structured data then text to sql applications of llm's are so much more powerful. The generative models will then be able to form complex queries to find similarity as well as perform aggregation, filtering and joining across datasets. To do this today with a separate dedicated vector db is quite painful.

[abrichr]

Thank you for your recent interest in requesting support for pgvector. Thanks to your interest, Amazon RDS for PostgreSQL is pleased to announce support for pgvector.

To learn more, please see the following:

Announcement: https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-rds-postgresql-pgvector-ml-model-integration/
Blog: https://aws.amazon.com/blogs/database/building-ai-powered-search-in-postgresql-using-amazon-sagemaker-and-pgvector/d
GitHub: https://github.com/pgvector/pgvector
Amazon RDS User Guide: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.Extensions.html
Thanks,

The Amazon RDS for PostgreSQL team

See also pgvector/pgvector#54 (comment)

[abrichr]

Let's try https://github.com/asg017/sqlite-vss

[abrichr]

Please check https://paperswithcode.com/ and https://scholar.google.com/ for review papers on embeddings functions, in particular for structured data.

[AvidEslami]

Currently, it seems that in order for us to make use of Long Term Memory we would need a distance metric more suited for our specific tasks as opposed to the defaults provided by most VectorDB.