Ensemble viz #565

paulstapor · 2021-02-04T10:22:42Z

This PR should complete very most of the ensemble functionality, including nice plotting and some more convenience things, such as storing ensemble_predictions directly to h5 files

…have an easier was to save ensemble predictions later one

…le_viz

codecov-io · 2021-02-04T12:48:57Z

Codecov Report

Merging #565 (976b817) into develop (ddcfd6b) will decrease coverage by 55.53%.
The diff coverage is 19.31%.

@@             Coverage Diff              @@
##           develop     #565       +/-   ##
============================================
- Coverage    89.81%   34.27%   -55.54%     
============================================
  Files           75       79        +4     
  Lines         4878     5117      +239     
============================================
- Hits          4381     1754     -2627     
- Misses         497     3363     +2866

Impacted Files	Coverage Δ
pypesto/petab/importer.py	`0.00% <0.00%> (-85.81%)`	⬇️
pypesto/prediction/amici_predictor.py	`13.88% <ø> (-83.34%)`	⬇️
pypesto/visualize/ensemble.py	`10.16% <0.00%> (-89.84%)`	⬇️
pypesto/ensemble/ensemble.py	`10.89% <7.14%> (-70.28%)`	⬇️
pypesto/prediction/prediction.py	`21.59% <7.14%> (-72.32%)`	⬇️
pypesto/ensemble/covariance_analysis.py	`18.75% <18.75%> (ø)`
pypesto/visualize/dimension_reduction.py	`18.91% <18.91%> (ø)`
pypesto/ensemble/utils.py	`21.05% <21.05%> (ø)`
pypesto/ensemble/dimension_reduction.py	`38.88% <38.88%> (ø)`
pypesto/ensemble/__init__.py	`100.00% <100.00%> (ø)`
... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ddcfd6b...976b817. Read the comment docs.

paulstapor · 2021-02-05T09:27:17Z

Shall I just also add an engine and create a task to be executed for predictions? I just found out that multi-processing within the amici objective doesn't work without an engine...

yannikschaelte · 2021-02-05T09:33:16Z

What is "ICB objective"? If you want to multi-process the execution, a first thing would be to check whether you can formalize this as an "EnsembleTask" derived from https://github.com/ICB-DCM/pyPESTO/blob/master/pypesto/engine/task.py, similar to for optimization and profiling. If that does not work (e.g. because you need more dynamic scheduling or interprocess communication), one would need to write the full Engine anew.

paulstapor · 2021-02-05T09:34:35Z

What is "ICB objective"? If you want to multi-process the execution, a first thing would be to check whether you can formalize this as an "EnsembleTask" derived from https://github.com/ICB-DCM/pyPESTO/blob/master/pypesto/engine/task.py, similar to for optimization and profiling. If that does not work (e.g. because you need more dynamic scheduling or interprocess communication), one would need to write the full Engine anew.

amici objecive... Too much distraction due to icb (com health) orga retreat... 😂
I looked at the optimization task... I think that should be redoable for predictions or ensembles

yannikschaelte · 2021-02-05T10:09:15Z

What is "ICB objective"? If you want to multi-process the execution, a first thing would be to check whether you can formalize this as an "EnsembleTask" derived from https://github.com/ICB-DCM/pyPESTO/blob/master/pypesto/engine/task.py, similar to for optimization and profiling. If that does not work (e.g. because you need more dynamic scheduling or interprocess communication), one would need to write the full Engine anew.

amici objecive... Too much distraction due to icb (com health) orga retreat...
I looked at the optimization task... I think that should be redoable for predictions or ensembles

Nice! ICB objective is a classical example of failed multitasking 😆

paulstapor · 2021-02-05T11:01:28Z

So, what's the consense? Adding an engine to predictions + ensembles? Or leave that t @dilpath in #527 `:D

yannikschaelte · 2021-02-05T11:25:00Z

No objections to anything from my side ;)

dilpath · 2021-02-05T12:52:29Z

So, what's the consense? Adding an engine to predictions + ensembles? Or leave that t @dilpath in #527 `:D

👍 Can work on this now

paulstapor · 2021-02-05T13:24:55Z

So, what's the consense? Adding an engine to predictions + ensembles? Or leave that t @dilpath in #527 `:D

+1 Can work on this now

No super big rush there... Just found out that mutlithreading within AmiciObjective also works well for the AmiciPredictor...
I'm more interested in getting this here into develop :)

dilpath

Will review the rest in a little bit, sorry for delays

pypesto/ensemble/covariance_analysis.py

dilpath

👍

There are no tests that assert that the new methods work as intended, only tests that ensure that errors are not produced during a standard workflow.

pypesto/ensemble/ensemble.py

dilpath · 2021-02-05T14:52:17Z

pypesto/ensemble/utils.py

+        f.create_group(base + f'{LOWER_BOUND}s')
+        for i_cond, lower_bounds in enumerate(ensemble_prediction.lower_bound):
+            condition_id = \
+                ensemble_prediction.prediction_results[0].condition_ids[i_cond]


If ensemble_prediction.prediction_results[i].condition_ids is equal for all i then maybe the condition_ids could be stored independently of ensemble_prediction.prediction_results to avoid duplication, e.g. ensemble_prediction.condition_ids?

In principle yes, but we store a multitude of PredictionResults here. As PredictionResult also must work as a class of its own, it makes sense to also store the conditionIds in there. One could now either reduce what's stored, therefore introduce a new class... or storing something redundant... What do you think?

For the purposes of e.g. plotting by condition or observable, it would be nice to automatically ensure that all PredictionResults share the same condition IDs, and the prediction for the same condition between different PredictionResults contains the same observable IDs. And also to provide methods to easily get all data by condition or observable, from an EnsemblePrediction. Maybe a wrapper for SampleEnsemblePrediction? Anyway, I can do this, let me know what you think

I would actually suggest to open 3 issues:

wishes and stuff we can use the ensemble functionality for (e.g., what you just wrote)

wishes and stuff we can use the prediction functionality for (e.g., prediction profiles)

wishes and stuff we can use the dimesnion reduction functionality for (e.g., analyzing optimization runs from a multi-start visually)

But yes: Asserting these things are equal would be good. Currently, they must be equal, just by the way the predict functions creates things...

pypesto/prediction/prediction.py

pypesto/visualize/dimension_reduction.py

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

paulstapor added 15 commits January 27, 2021 12:11

fix ensemble identifiability plot if one or more categories are empty

dcdf258

add dimension reduction functionality via umap and pca

3a982fc

make function usable from the outside

4691ad6

add first version of covariance stuff

d814cc5

add visualizations for dimension reduction

d9f115f

rewrite reader and wrtier functions for ensembles and predictions to …

8bc0303

…have an easier was to save ensemble predictions later one

add covariance matrix analysis code

04c07b8

cleanup pep8, little bugfixes in ensembles

e549a1b

update extra requirements for machine learning like tools

1b531f4

Merge branch 'develop' into ensemble_viz

86fd535

change Type hints for Umap and sklearn objects, to not ruin dependencies

eb279cd

Merge branch 'ensemble_viz' of github.com:ICB-DCM/pyPESTO into ensemb…

bc5fd9a

…le_viz

fix import

89da9a7

fix spurious variable name

f44b595

fix spurious variable name

646bc87

paulstapor added 5 commits February 4, 2021 14:44

improve doc

9a5970c

add mltools to installation, as used for tests

3a3ca7c

make flake8 happy

8af43ed

fix doc

94381a1

trailing whitespace... need that pre-commit thing

f117f37

paulstapor requested review from dilpath and jvanhoefer February 4, 2021 20:38

Merge branch 'develop' into ensemble_viz

2f1c233

dilpath reviewed Feb 5, 2021

View reviewed changes

dilpath approved these changes Feb 5, 2021

View reviewed changes

paulstapor and others added 14 commits February 8, 2021 13:40

Update pypesto/ensemble/covariance_analysis.py

363a62f

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/ensemble/covariance_analysis.py

68f0c52

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/ensemble/covariance_analysis.py

c60eec8

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/ensemble/covariance_analysis.py

5fc7f29

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/ensemble/covariance_analysis.py

dff371b

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/ensemble/ensemble.py

5ab6c86

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/ensemble/ensemble.py

51c8c28

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/ensemble/ensemble.py

6f6aa9c

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/visualize/dimension_reduction.py

cc7555c

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/visualize/dimension_reduction.py

d73984f

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

Update pypesto/visualize/dimension_reduction.py

058c2f6

Co-authored-by: Dilan Pathirana <59329744+dilpath@users.noreply.github.com>

address code review

74f4eec

fix flake

a1f74e5

arg, dammit, forgot an unused import

976b817

paulstapor merged commit e9a5d5e into develop Feb 8, 2021

paulstapor deleted the ensemble_viz branch February 8, 2021 20:00

yannikschaelte mentioned this pull request Mar 17, 2021

Release 0.2.4 #596

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensemble viz #565

Ensemble viz #565

paulstapor commented Feb 4, 2021

codecov-io commented Feb 4, 2021 •

edited

Loading

paulstapor commented Feb 5, 2021 •

edited

Loading

yannikschaelte commented Feb 5, 2021

paulstapor commented Feb 5, 2021 •

edited

Loading

yannikschaelte commented Feb 5, 2021

paulstapor commented Feb 5, 2021

yannikschaelte commented Feb 5, 2021

dilpath commented Feb 5, 2021 •

edited by paulstapor

Loading

paulstapor commented Feb 5, 2021

dilpath left a comment

dilpath left a comment

dilpath Feb 5, 2021

paulstapor Feb 8, 2021

dilpath Feb 8, 2021

paulstapor Feb 8, 2021

paulstapor Feb 8, 2021

Ensemble viz #565

Ensemble viz #565

Conversation

paulstapor commented Feb 4, 2021

codecov-io commented Feb 4, 2021 • edited Loading

Codecov Report

paulstapor commented Feb 5, 2021 • edited Loading

yannikschaelte commented Feb 5, 2021

paulstapor commented Feb 5, 2021 • edited Loading

yannikschaelte commented Feb 5, 2021

paulstapor commented Feb 5, 2021

yannikschaelte commented Feb 5, 2021

dilpath commented Feb 5, 2021 • edited by paulstapor Loading

paulstapor commented Feb 5, 2021

dilpath left a comment

Choose a reason for hiding this comment

dilpath left a comment

Choose a reason for hiding this comment

dilpath Feb 5, 2021

Choose a reason for hiding this comment

paulstapor Feb 8, 2021

Choose a reason for hiding this comment

dilpath Feb 8, 2021

Choose a reason for hiding this comment

paulstapor Feb 8, 2021

Choose a reason for hiding this comment

paulstapor Feb 8, 2021

Choose a reason for hiding this comment

codecov-io commented Feb 4, 2021 •

edited

Loading

paulstapor commented Feb 5, 2021 •

edited

Loading

paulstapor commented Feb 5, 2021 •

edited

Loading

dilpath commented Feb 5, 2021 •

edited by paulstapor

Loading