add regression plots and error histograms for training/validation/testing

[wiederm]

Pull Request Summary

Training progression plots

This PR adds regression plots and error histograms for

• each training/validation set evaluation
• final validation/test set evaluation
  and logs these plots automatically to Weights&Biases (or tensorboard).

The error is plotted as a histogram with frequency on a log scale to identify outliers better:

The regression plot is a simple xy scatter plot:

During training we plot:

• energy/force error histogram
• energy regression

During validation we plot:

• energy error histogram
• energy regression

Monitor outliers in the training and validation set

To better identify outliers, we log to W&B a counter for the k data points with the highest RMSE error. This allows us to match outliers to the histograms if needed.

Pull Request Checklist

• Appropriate docstring(s) added/updated
• Appropriate .rst doc file(s) added/updated
• PR is ready for review

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 88.48168% with 22 lines in your changes missing coverage. Please review.

Project coverage is 85.49%. Comparing base (f20c713) to head (8ca7fe3).

Additional details and impacted files

[wiederm]

@chrisiacovella , can you plaese review?