Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support dumping datasets to pkl file #896

Open
jbwang1997 opened this issue Jan 19, 2023 · 2 comments · May be fixed by #1037
Open

[Feature] Support dumping datasets to pkl file #896

jbwang1997 opened this issue Jan 19, 2023 · 2 comments · May be fixed by #1037

Comments

@jbwang1997
Copy link
Contributor

jbwang1997 commented Jan 19, 2023

What's the feature?

Suggest adding a dump function to BaseDataset. This function will make the dataset more flexible. Users can first dump the dataset to a pkl file, and then reformat its structure to anther dataset type.

Users can dump the dataset like this:

dataset.dump('data_list.pkl')

Then, data_list can also be directly loaded

dataset.load('data_list.pkl')

Any other context?

No response

@RangiLyu RangiLyu transferred this issue from open-mmlab/mmdetection Jan 31, 2023
@RangiLyu
Copy link
Member

I think maybe the BaseDataset in mmengine can support a dumping function.

@jbwang1997
Copy link
Contributor Author

I can support a dump function to save all data in self.data_list into a .pkl file.

And I also suggest supporting dataset wise dump functions to format self.data_list into the certain dataset format. It will benefit many dataset operations like transformation, separation.

@HAOCHENYE HAOCHENYE changed the title [Feature] Support dumping function in datasets [Feature] Support dumping datasets to pkl file Mar 8, 2023
@zhouzaida zhouzaida linked a pull request Apr 11, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants