Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to choose data type for Categorical storage, U8, U16, instead of only U32 #16087

Closed
Houski opened this issue May 6, 2024 · 3 comments
Labels
A-dtype-categorical Area: categorical data type enhancement New feature or an improvement of an existing feature

Comments

@Houski
Copy link

Houski commented May 6, 2024

Description

We can significantly decrease the memory usage of our dataset by using U8s instead of U32s for categorical encoding.

In a discussion on the Rust Polars Discord, it was mentioned that while the underlying Arrow dictionary type supports specifying the index type, although Polars currently has it hardcoded to U32.

@Houski Houski added the enhancement New feature or an improvement of an existing feature label May 6, 2024
@stinodego stinodego added A-dtype-categorical Area: categorical data type accepted Ready for implementation labels May 6, 2024
@stinodego
Copy link
Member

This is something we want to support in the future, but it currently does not have priority for us.

@mcrumiller
Copy link
Contributor

Also duplicate of #13109.

@stinodego
Copy link
Member

Thanks for the link, I knew there had to be an issue but I couldn't find it. I'll close this one in favor of the existing issue.

@stinodego stinodego closed this as not planned Won't fix, can't repro, duplicate, stale May 6, 2024
@stinodego stinodego removed the accepted Ready for implementation label May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-dtype-categorical Area: categorical data type enhancement New feature or an improvement of an existing feature
Projects
None yet
Development

No branches or pull requests

3 participants