Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch Dataset to pandas Dataframe + torch Dataset #84

Closed
NickleDave opened this issue Nov 29, 2019 · 1 comment
Closed

switch Dataset to pandas Dataframe + torch Dataset #84

NickleDave opened this issue Nov 29, 2019 · 1 comment

Comments

@NickleDave
Copy link
Collaborator

  • lets pandas and keras do the heavy lifting
  • makes it possible to work on bigger data sets, avoid loading everything into memory all at once
  • don't have to maintain a fancy data structure
  • lets tests actually work again -- had non-deterministic failures because of Dask dark magic
@NickleDave NickleDave changed the title switch Dataset to pandas Dataframe + keras.utils.Sequence-based generator switch Dataset to pandas Dataframe + torch Dataset Mar 7, 2020
@NickleDave
Copy link
Collaborator Author

this ended up becoming a very large "feature branch", #88
Closing because that branch was merged in.
Switched to Pandas DataFrame for datasets, and switched to torch to use their Dataset / Dataloader classes (instead of keras, because of bugs with keras.utils.Sequence)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant