Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RI-5545] [RI-5543] Add support for non-native big integers and allow non-native big integers for Vector formatting #3185

Merged
merged 1 commit into from
Mar 28, 2024

Conversation

GnaneshKunal
Copy link
Collaborator

@GnaneshKunal GnaneshKunal commented Mar 26, 2024

Allow non-native big integers in JSONBigInt to convert large decimals. This can be the case for some language models that use large decimals in data. JSONBigInt will fall back to non-native big integers if they can't be packed in the native ones, so we still have support for the old ones. The support is explicitly added to Vector 32-bit and Vector 64-bit data.

When viewing some vector data that uses large numbers, which are often decimals, we try to pack them into javascript's native big int. Javascript's native big int doesn't support Big decimals yet. - sidorares/json-bigint#88

We have two options:

  1. Add support for non-native big integers when native big integers are not supported
  2. Raise an exception and show the data in UTF8

This PR chooses 1. Please check ticket comment.

Vector visualization

Data that can be packed into native big int

image

Data that are too large to be packed into native big Int

image

decimals. This can be the case for some language models using large decimals.
@vlad-dargel vlad-dargel changed the base branch from main to release/2.46.0 March 28, 2024 09:37
@vlad-dargel vlad-dargel dismissed stale reviews from AmirAllayarovSofteq and romansergeenkosofteq March 28, 2024 09:37

The base branch was changed.

@vlad-dargel vlad-dargel merged commit 93dc00c into release/2.46.0 Mar 28, 2024
25 of 29 checks passed
@vlad-dargel vlad-dargel deleted the bugfix/RI-5545_RI-5543 branch March 28, 2024 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants