Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return validation losses in TaskTrainerHistory #26

Closed
mlprt opened this issue Mar 2, 2024 · 1 comment
Closed

Return validation losses in TaskTrainerHistory #26

mlprt opened this issue Mar 2, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@mlprt
Copy link
Owner

mlprt commented Mar 2, 2024

At the end of a training run, TaskTrainer returns auxiliary information such as losses and intermediate model states in a TaskTrainerHistory object.

The history of the training loss is returned in the loss field, but the validation loss is not returned.

Validation is only performed once every log_step trials, so a little care will need to be taken that the returned arrays are comparable with those for the training loss; it might be appropriate to return arrays that are all NaN except on the log steps. Memory should not be an issue. Usually there won't be more than 5 loss terms or so, and (conservatively) less than 100,000 iterations. That's about 2 MB of float32.

@mlprt mlprt added the enhancement New feature or request label Mar 2, 2024
@mlprt
Copy link
Owner Author

mlprt commented Apr 25, 2024

Added in 9d1056a.

Currently, the returned arrays of validation losses are zero everywhere, except on batches where the model is validated. I avoided using jnp.nan so that I wouldn't need to alter the program's behaviour when troubleshooting NaNs arising elsewhere. Since zero is a valid loss, it might be better to switch to -1 or something.

Also added a switch to feedbax.plotly.loss_history to plot these validation losses. Unlike training losses, the validation losses are typically sparse over time, so in their case I've switched plotting from "markers+lines" to just "markers".

@mlprt mlprt closed this as completed Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant