Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: use pgvecto.rs #3605

Merged
merged 42 commits into from
Dec 8, 2023
Merged

feat: use pgvecto.rs #3605

merged 42 commits into from
Dec 8, 2023

Conversation

jrasm91
Copy link
Contributor

@jrasm91 jrasm91 commented Aug 8, 2023 •
edited by mertalev

Migrate to pgvecto.rs

TODO:

  • Add postgres 15 migration guide
  • Upgrade to postgres 15
  • Database migration for vector embeddings to pgvecto.rs
  • Validate machine learning jobs with new database types
  • Upgrade FR to use pgvecto.rs
  • Upgrade CLIP to use pgvecto.rs
  • Upgrade explore page to use postgres (can happen in a separate PR)
  • Upgrade search page to use postgres for non-vector search (can happen in a separate PR)
  • Remove typesense code
  • Remove search jobs
  • Remove typesense documentation / environment variables
  • Handle changing embedding size
  • Add HNSW index
  • Add unit and e2e tests
  • Fix TypeORM migration action
  • Fix links at the bottom of the explore page (or temporarily remove)

Fixes #4714
Fixes #4780
Fixes #4929
Fixes #5033
Fixes #5079
Fixes #5084
Fixes #5261
Fixes #5464
Fixes #5550

@vercel
Copy link

vercel bot commented Aug 8, 2023 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
immich ⬜️ Ignored (Inspect) Visit Preview Nov 12, 2023 3:14am

@danieldietzler
Copy link
Member

danieldietzler commented Sep 25, 2023 •
edited

Wouldn't it be helpful to directly upgrade to postgres 16? I mean if we need to upgrade the postgres version anyway, so having to advice people on how to do this without loosing data, why not upgrade to the latest version directly?

(I know that the PR has started when postgres 16 wasn't available, so this should be just a heads up to consider before merging :))

Also: I like the diff of this PR haha! Feels good to throw away so much junk.

@samip5
Copy link
Contributor

samip5 commented Oct 4, 2023 •
edited

This is gonna be a pain for us on Kubernetes. Maybe not as their images do have pgvector already, never mind. :)

pgvector/pgvector#228

@mertalev mertalev force-pushed the dev/pgvector branch 2 times, most recently from 5907b93 to bda3093 Compare October 10, 2023 01:14
@jrasm91 jrasm91 force-pushed the dev/pgvector branch 2 times, most recently from 0df3319 to 8f18700 Compare October 13, 2023 05:37
@martabal martabal mentioned this pull request Oct 14, 2023
Closed
@mertalev mertalev force-pushed the dev/pgvector branch 2 times, most recently from 732c8d5 to a7a230c Compare November 9, 2023 00:53
@jrasm91 jrasm91 changed the title feat: use pgvector feat: use pgvecto.rs Dec 7, 2023
danieldietzler
danieldietzler approved these changes Dec 7, 2023
View reviewed changes
Copy link
Member

@danieldietzler danieldietzler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing job! :)

martabal
martabal approved these changes Dec 7, 2023
View reviewed changes
Copy link
Member

@martabal martabal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing 🎉

@vekexasia
Copy link

Does this allow us to see more than x videos? I have 2k videos in immigh but i can only see a chunk of them via the search?type=VIDEO tab.

@mertalev mertalev merged commit 1e99ba8 into main Dec 8, 2023
18 checks passed
@mertalev mertalev deleted the dev/pgvector branch December 8, 2023 16:15
@moll-re
Copy link

moll-re commented Dec 21, 2023 •
edited

This is gonna be a pain for us on Kubernetes. Maybe not as their images do have pgvector already, never mind. :)

pgvector/pgvector#228

This is false, I believe. CNPG images ship pgvector (no s) by default. A vecto-rs CNPG image is only provided by @bo0tzz under ghcr.io/bo0tzz/cnpgvecto.rs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bo0tzz bo0tzz bo0tzz approved these changes

@alextran1502 alextran1502 alextran1502 approved these changes

@danieldietzler danieldietzler danieldietzler approved these changes

@martabal martabal martabal approved these changes

@mertalev mertalev mertalev approved these changes

@zackpollard zackpollard Awaiting requested review from zackpollard

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
10 participants
@jrasm91 @danieldietzler @samip5 @andrewgdunn @mertalev @vekexasia @moll-re @bo0tzz @alextran1502 @martabal