Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration with common dataset format #25

Closed
seanmor5 opened this issue Mar 30, 2021 · 3 comments
Closed

Integration with common dataset format #25

seanmor5 opened this issue Mar 30, 2021 · 3 comments
Labels
kind:feature New feature or request note:discussion Details or approval are up for discussion note:upstream The issue must be tackled upstream

Comments

@seanmor5
Copy link
Contributor

Requires developments in elixir-nx/nx#301 and whatever external efforts emerge.

@seanmor5 seanmor5 added kind:feature New feature or request note:discussion Details or approval are up for discussion note:upstream The issue must be tackled upstream labels Apr 1, 2021
@t-rutten
Copy link
Contributor

Would it be worth shifting or adding function clauses to the current Axon.Training API to be more friendly to inputs and targets that are streams? This would seem to entail swapping Enum functions (like zip/2 and with_index/1) for their Stream versions.

Additionally, we could avoid calling Enum.count/1 in Training.train_epoch/5 by requiring inputs and labels to each be a tuple of an enumerable and metadata about the enumerable--like {data, %{total_batches: _}} = _inputs.

@seanmor5
Copy link
Contributor Author

@t-rutten I think we should go ahead and make the switch to use streams in the training API instead of Enumerables. I think any dataset format will logically have to be implemented as a stream anyway to avoid having to load large datasets entirely into memory.

As far as avoiding usage of Enum.count, I think we can instead just change the average calculation to use the current training step, that way we can support cases where the size of the dataset is unknown ahead of time and where users are specifying steps per epoch

@seanmor5
Copy link
Contributor Author

seanmor5 commented Feb 5, 2022

With the new training API, I think we're pretty much set on allowing Axon to work with anything that implements Enumerable, which I think is flexible enough as is.

@seanmor5 seanmor5 closed this as completed Feb 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:feature New feature or request note:discussion Details or approval are up for discussion note:upstream The issue must be tackled upstream
Projects
None yet
Development

No branches or pull requests

2 participants