ENH: Add ability to run eval with and without post-processing transforms

[NickleDave]

This does two things:

• convert all the transforms for labeled timebins to actual transforms with functional forms and corresponding classes, as in ENH: Convert post-processing to Transform classes that can be specified in config  #537. For details see commit and discussion on that issue.
  • Essentially we factor a postprocess transform out of the function that was in vak.labeled_timebins that is applied to labeled timebins, and then returns labeled timebins (instead of segments or labels only).
  • This allows us to compute metrics like accuracy on the transformed labeled timebins, and also use it to get labels with a to_labels transform, or labels + onset and offset times with to_segments.
• add the ability to run eval with the same post-processing transforms applied that we use for prediction, as in ENH: eval should have option compute metrics with and without clean-ups #472. We achieve this by adding a post_tfm attribute to engine.Model which defaults to None, and a corresponding parameter post_tfm_kwargs to core.eval and core.learncurve that if specified is used to make an instance of PostProcess that is then passed into the Model when instantiating. If passed in, it is used during eval on the output of the model, e.g. the predicted labeled timebins, as described above.

Before merging we need to do the following (besides making sure CI passes):

• test that the new version replicates what we got with the TweetyNet paper when we did this "manually" with a script
• It doesn't have to be perfect but it shouldn't be wildly off base

We will then make a release of vak, and finally we'll need to release a new TweetyNet model that has the post_tfm parameter added to its from_config method. That new release will need to have the new version of vak as its minimum required version.

[codecov-commenter]

Codecov Report

Base: 93.22% // Head: 93.34% // Increases project coverage by +0.11% 🎉

Coverage data is based on head (adb248e) compared to base (1ae478d).
Patch coverage: 98.25% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #621      +/-   ##
==========================================
+ Coverage   93.22%   93.34%   +0.11%     
==========================================
  Files          66       68       +2     
  Lines        2494     2613     +119     
==========================================
+ Hits         2325     2439     +114     
- Misses        169      174       +5     

Impacted Files	Coverage Δ
tests/fixtures/annot.py	93.68% <90.74%> (-5.14%)	⬇️
tests/fixtures/config.py	94.16% <100.00%> (+0.09%)	⬆️
tests/test_core/test_eval.py	100.00% <100.00%> (ø)
tests/test_labeled_timebins.py	100.00% <100.00%> (ø)
tests/test_labels.py	100.00% <100.00%> (ø)
...ransforms/test_labeled_timebins/test_functional.py	100.00% <100.00%> (ø)
...ransforms/test_labeled_timebins/test_transforms.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[NickleDave]

@yardencsGitHub I was able to get a "good enough" replication of the paper result for one bird by running learncurve, and so I am going to consider this good enough to merge.

Here's the original paper result for bl26lb16:

And here's what I got from a re-run where we run eval with and without post-processing transforms at every step:

I used previous_run_path so differences here are really about what windows the network saw and floating point error. Qualitatively, trends are the same.