Skip to content

Add more pgvector types.#522

Merged
vpetrovykh merged 3 commits intomasterfrom
pgvector
Nov 6, 2024
Merged

Add more pgvector types.#522
vpetrovykh merged 3 commits intomasterfrom
pgvector

Conversation

@vpetrovykh
Copy link
Contributor

Add support for sparsevec and halfvec.
halfvec is handled as a float array (converted from float32 to float16 or vice versa as needed).
sparsevec is handled as a dict with non-zero values and "dimensions" specified in it.

@vpetrovykh
Copy link
Contributor Author

vpetrovykh commented Sep 27, 2024

The halfvec codecs need more robust conversion for float16. This seems to outline a good algo that I can implement here: http://www.fox-toolkit.org/ftp/fasthalffloatconversion.pdf. As a bonus, I don't even need to handle NaN or Inf values because they are not valid in vectors (I checked with the actual pgvector types). So it's a little bit fewer corner cases to cover.

Sometimes _Float16 might be available via #include <float.h>, which should take care of casting between float/double and float16, but it would still need a fall-back.

@vpetrovykh vpetrovykh force-pushed the pgvector branch 2 times, most recently from 7c8b440 to bdebf88 Compare October 7, 2024 17:00
@vpetrovykh vpetrovykh marked this pull request as ready for review October 11, 2024 16:34
@vpetrovykh vpetrovykh force-pushed the pgvector branch 4 times, most recently from 8e9fa06 to 98f326c Compare November 6, 2024 00:25
Add support for `sparsevec` and `halfvec`.
`halfvec` is handled as a float array (converted from float32 to float16
or vice versa as needed).
`sparsevec` is handled as a dict with non-zero values and "dimensions"
specified in it.
@vpetrovykh vpetrovykh merged commit d5d4f5c into master Nov 6, 2024
@vpetrovykh vpetrovykh deleted the pgvector branch November 6, 2024 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants