Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make UnivariateDriftCalculator (and other objects) JSON serializable #394

Closed
KGoldsmith11 opened this issue Jun 4, 2024 · 2 comments
Closed
Assignees
Labels
enhancement New feature or request stale

Comments

@KGoldsmith11
Copy link

KGoldsmith11 commented Jun 4, 2024

Currently, the UnivariateDriftCalculator object supports serialization via pickle. However, this format is not compatible with Apache Spark, which I intend to use for processing. on the inference side.

For governance reasons, I need to fit the drift calculator object in a different machine to the one where I will perform inference and have access to the analysis chunks, and the machine performing inference uses spark. Therefore I need to fit the object, serialise it, move it to the inference machine, load it in pyspark and then calculate the data drift on the inference data. This is not working using pickle but spark does have json load methods (spark doesnt have pickle loading methods).

JSON would be a good alternative to pickle as there are json load methods in spark.

@KGoldsmith11 KGoldsmith11 added the enhancement New feature or request label Jun 4, 2024
Copy link

stale bot commented Aug 4, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Aug 4, 2024
@stale stale bot closed this as completed Aug 11, 2024
@nnansters nnansters removed the stale label Aug 11, 2024
@nnansters nnansters reopened this Aug 11, 2024
Copy link

stale bot commented Oct 19, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Oct 19, 2024
@stale stale bot closed this as completed Oct 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request stale
Projects
None yet
Development

No branches or pull requests

2 participants