Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: serialize/deserialize LazyFrame #1073

Merged
merged 4 commits into from
May 5, 2024
Merged

feat: serialize/deserialize LazyFrame #1073

merged 4 commits into from
May 5, 2024

Conversation

eitsupi
Copy link
Collaborator

@eitsupi eitsupi commented May 4, 2024

The Python class method polars.LazyFrame.deserialize() cannot be expressed well here, so I assigned a function pl$deserialize_lf().
The function name could be change.

In addition, in Python, these functions can also have files as input/output destinations, but here I am simplifying to interacting with character vectors. Reading and writing to files must be done by other functions.

@eitsupi eitsupi requested a review from etiennebacher May 4, 2024 15:39
@eitsupi
Copy link
Collaborator Author

eitsupi commented May 4, 2024

It might be worthwhile to enable the format to be specified as chracter or raw, as in nodejs.
https://pola-rs.github.io/nodejs-polars/interfaces/pl.LazyDataFrame.html#serialize

Copy link
Collaborator

@etiennebacher etiennebacher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I don't know much about serializing/deserializing but LGTM. Should we add a comment about security concerns when deserializing an unknown input as they do in the py-polars docs?

@eitsupi
Copy link
Collaborator Author

eitsupi commented May 5, 2024

Should we add a comment about security concerns when deserializing an unknown input as they do in the py-polars docs?

I think there is a special feature for Python that saves Python objects as pickle for Python.
I don't think the concern exists because there is no R-only functionality here and Polars does not restore anything like rds files.

@eitsupi eitsupi merged commit 930bd38 into main May 5, 2024
33 of 35 checks passed
@eitsupi eitsupi deleted the serialize-lazyframe branch May 5, 2024 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants