Unify model comparison api

[galexopoulos]

Description

Checklist

• Follows official PR format
• New features are properly documented (with an example if appropriate)?
• Includes new or updated tests to cover the new feature
• Code style correct (follows pylint and black guidelines)
• Changes are listed in changelog

[galexopoulos]

I haven't created any tests yet because I have a little trouble understanding all the parameters in the compare tests

[galexopoulos]

@OriolAbril could you please explain to me what I have to test in the compare function?

[OriolAbril]

Something along the lines of use_elpddata in https://github.com/arviz-devs/arviz/blob/main/arviz/tests/base_tests/test_plots_matplotlib.py#L1053 (no need for add_groups though. It may also be a good idea to create a helper function with all the common functionality between compare and plot_elpd to get from idata or elpddata to elpddata. i.e. I think right now error handling is better in compare whereas plot_elpd is more flexible and already allows both ehlddata and idata, so in creating a helper we'd get the best of both functions while homogenizing our API

[galexopoulos]

Thank you for your help! Of course and I can do that, great idea! Although where should I store this helper function? Should I store it somewhere separately and import it to both compare and elpd_plot or store it in every python file it is used?

[galexopoulos]

Created the Helper function Calculate_ICS, although I haven't run any tests to see if it works yet. Also, I am not sure if it is wrong that I don't return the names variable in the elpd_plot function.

[OriolAbril]

The function should be in a separate file, maybe in arviz/utils.py so that both functions import it and we therefore reduce the code duplication, after all, both should take the same input and this initial conversion is common between the two functions.

[galexopoulos]

Hello @oriol!
I can not understand what are the input values I can give the compare_dict or the dataset_dict in order to test my helper function.
I have difficulty running the tests I created for my function Calculate_ICS stored in the arivz.utils directory.

[OriolAbril]

I have difficulty running the tests I created for my function Calculate_ICS stored in the arivz.utils directory.

The problems might be due to the some of the comments like the wrong number of return arguments to assign or the circular import. The function itself looks great though, it simplifies the code while extending functionality, thanks!

I can not understand what are the input values I can give the compare_dict or the dataset_dict in order to test my helper function.

I think it will already be tested via the tests for compare and plot_elpd, so before working on tests, I would first get the current tests to run to see the coverage, and then see if we can for example simplify two tests on compare and plot_elpd to a single test on calculate_ics or if we need to add some specific calculate_ics tests in addition to the existing ones.

[derekpowell]

@galexopoulos I posted in issue #1643 some code but it looks like you are further along here than I was. Are you still working on this? Looks like mostly missing tests at this point?

[OriolAbril]

@derekpowell can you test the PR? It should be working for both compare and plot_elpd taking dicts of inferencedata, of elpddata or mixture of both. You can install arviz from this PR directly using https://stackoverflow.com/a/50095199/2504700

[codecov]

Codecov Report

Merging #1690 (3c61b5c) into main (6cc9f5e) will decrease coverage by 0.02%.
The diff coverage is 91.54%.

@@            Coverage Diff             @@
##             main    #1690      +/-   ##
==========================================
- Coverage   90.84%   90.82%   -0.03%     
==========================================
  Files         114      114              
  Lines       12327    12355      +28     
==========================================
+ Hits        11199    11221      +22     
- Misses       1128     1134       +6     

Impacted Files	Coverage Δ
arviz/rcparams.py	93.33% <ø> (ø)
arviz/utils.py	88.23% <ø> (ø)
arviz/stats/stats.py	95.58% <90.76%> (-0.71%)	⬇️
arviz/plots/elpdplot.py	100.00% <100.00%> (ø)
arviz/stats/__init__.py	100.00% <100.00%> (ø)
arviz/plots/distcomparisonplot.py	93.75% <0.00%> (+0.72%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6cc9f5e...3c61b5c. Read the comment docs.

[ahartikainen]

Thanks for the PR