-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support nested datatypes in from_repr
#15842
Comments
from_repr
does not support data type 'List'from_repr
Thanks for the issue. This would definitely be good to support. |
Hey, can I take this up? I assume I would need to support just polars.datatypes.FLOAT_DTYPES and polars.datatypes.INTEGER_DTYPES inside the List right? I have made a draft pull request, would appreciate any comments :) If you think I am in the right direction, I can work on test cases and other functionalities associated with this feature. |
Sure! Lists can contain anything though (also strings, decimals, ...). So it's not just constrained to floats/integers. |
Got it, I'm working on it. I have doubt regarding wrap around for string representation of polars dataframe, the column data is wrapping around as follows:
Due to this, the data is truncated, any suggestion on how I can handle this? |
The reasonable thing to do is load only the whole/valid data; truncated columns (when a frame has more cols than can be displayed) are similarly dropped. There is, after all, no way (at all) to reconstruct the truncated values, so... |
Got it, thanks! I have raised a pull request, could you please review and let me know if there are any suggestions? |
Description
In many ML/NLP use cases it's useful to have the reading from_repr feature supporting list type so that reading from a feature store where numerical representation are stored, e.g. embeddings vectors for unit testing.
Today if we run
we have
Thanks
The text was updated successfully, but these errors were encountered: