added option for weighted mean in ensemble #702

PaulJonasJost · 2021-07-12T16:38:21Z

This for now is only a draft.
It assumes, that the post processor either returns only a np.ndarray or a dict/dict like object.

Perhaps want to add another default post processor, that would only need to receive AMICI_outputs do creat a return dict of things that should be returned (like 'x', 'y', 'llh' etc).

Also need to add sigmas.

Currently the weighting is done in terms of llh of each condition separately. should I go with the llh for one parameter across all conditions?

codecov-commenter · 2021-07-12T16:46:23Z

Codecov Report

Merging #702 (afc6aa6) into develop (160c2a8) will increase coverage by 1.36%.
The diff coverage is 84.85%.

@@             Coverage Diff             @@
##           develop     #702      +/-   ##
===========================================
+ Coverage    88.16%   89.53%   +1.36%     
===========================================
  Files           79       97      +18     
  Lines         5257     6613    +1356     
===========================================
+ Hits          4635     5921    +1286     
- Misses         622      692      +70

Impacted Files	Coverage Δ
pypesto/engine/mpi_pool.py	`0.00% <0.00%> (ø)`
pypesto/ensemble/covariance_analysis.py	`18.36% <0.00%> (-0.39%)`	⬇️
pypesto/optimize/__init__.py	`100.00% <ø> (ø)`
pypesto/profile/util.py	`95.74% <ø> (ø)`
pypesto/sample/diagnostics.py	`100.00% <ø> (+30.76%)`	⬆️
pypesto/sample/geweke_test.py	`94.36% <ø> (ø)`
pypesto/store/hdf5.py	`79.16% <0.00%> (-3.45%)`	⬇️
pypesto/visualize/clust_color.py	`91.25% <42.85%> (ø)`
pypesto/predict/task.py	`43.75% <43.75%> (ø)`
pypesto/petab/__init__.py	`63.63% <50.00%> (ø)`
... and 72 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c013b9a...afc6aa6. Read the comment docs.

pypesto/predict/amici_predictor.py

dilpath

I think a test would be good

dilpath · 2021-07-13T09:38:57Z

pypesto/ensemble/ensemble.py

@@ -138,7 +140,8 @@ def condense_to_arrays(self):
        }

    def compute_summary(self,
-                        percentiles_list: Sequence[int] = (5, 20, 80, 95)
+                        percentiles_list: Sequence[int] = (5, 20, 80, 95),
+                        weighting: bool = True


Suggested change

weighting: bool = True

weigh: bool = True

Option can be weigh and values can be weights

Perhaps this should be False by default?

is just a matter of taste I think. As LLH is not included in a predictor by default, this will get turned to False in line 168. The idea behind that for me was that if you include llh in predictor for ensembles the default you probably would want is True.

So in the default setting, weighting==True but no weights are used? Might be confusing for users. Could call it something more descriptive, e.g. weights_if_llh_provided, or make it False by default?

dilpath · 2021-07-13T09:40:55Z

pypesto/ensemble/ensemble.py

+        # check if weightings shall be used or not depending on whether llh
+        # is part of the prediciton output


Suggested change

# check if weightings shall be used or not depending on whether llh

# is part of the prediciton output

# Check whether LLH-based weights can be used.

dilpath · 2021-07-13T09:47:32Z

pypesto/ensemble/ensemble.py

+        if AMICI_LLH not in self.prediction_results[0].conditions[0].output:
+            weighting = False


A user might explicitly want weights but not realize they're not used because of this. A warning might help?

That warning would then be problematic if the weighting flag is True by default, because the ensemble prediction would issue a warning by default. But default settings should not be chosen such that they issue warnings, I think.

dilpath · 2021-07-13T09:54:28Z

pypesto/ensemble/ensemble.py

+            summary[STANDARD_DEVIATION] = np.average(
+                (tmp_array.T-summary[MEAN].T).T**2,
+                axis=-1, weights=weights)


oh yes, there needs to be an additional np.sqrt() 👍

Then equivalent to np.average(np.abs(tmp_array - summary[MEAN]), axis=-1, weights=weights) I guess

no, at least that should not be the case mathematically...

paulstapor · 2021-07-14T10:59:42Z

pypesto/ensemble/ensemble.py

+        if AMICI_LLH not in self.prediction_results[0].conditions[0].output:
+            weighting = False


That warning would then be problematic if the weighting flag is True by default, because the ensemble prediction would issue a warning by default. But default settings should not be chosen such that they issue warnings, I think.

paulstapor · 2021-07-14T11:02:24Z

pypesto/ensemble/ensemble.py

+                output_list = [prediction.conditions[ic].output
+                               for prediction in self.prediction_results]
+            else:
+                output_list = [prediction.conditions[ic].output[AMICI_Y]


I just saw below, that it seems you intend to change the type of prediction.conditions[ic].output. However, this is a data member of the class PredictionConditionResult. I'd strongly argue that data members of classes should not switch type, if possible, and if inevitable: Only during major refactorings, as this will typically break backwards compatibility. Rather add a new data member for the weights...

paulstapor · 2021-07-14T11:07:53Z

pypesto/ensemble/ensemble.py

+            output_list = [prediction.conditions[ic].output[AMICI_LLH]
+                           for prediction in self.prediction_results]


Hm... I have the impression that this will be problematic. By default, a PredictionConditionResult has a member "output", which is supposed to be an ndarray. You're making a dict out of it, which might brake quite some dependencies...
I would rather suggest you add a new member "output_weight" to the PredictionConditionResult. This way, you can just use this new member here, and an additional field will not affect the functioning of old script, hence not destroy backwards compatibility.

…/pyPESTO into feature_ensemble_weights

…eature_ensemble_weights

…emble.

jvanhoefer

Since I am not really into that part of the code, I did only review code style (otherwise I would have to get an overview of the rest of the ensemble code first). So please let someone else review regarding functionality/structrue, but from style point of view: Looks fine to me :)

dilpath

Looks good 👍

Maybe an example in a notebook?

pypesto/ensemble/ensemble.py

dilpath · 2021-09-24T16:15:05Z

pypesto/ensemble/ensemble.py

+                        compute_weighted_sigma=True)
+            except TypeError:
+                raise ValueError('Computing a summary failed.')
+        n_conditions = len(self.prediction_results[0].conditions)


Can conditions differ between predictions? If so, can simply raise a NotImplementedError if there are different conditions between the prediction results.

they should not be able to, as the ensembles currently are created from the same amici_model which should generate the prediction the same no matter what the actual parameter values are.

Might be interesting way down the train, when we allow for different model structures to contribute to the same ensemble.

Conditions are provided via amici.ExpData (a single amici.Model can simulate different sets of conditions).

pypesto/ensemble/ensemble.py

dilpath · 2021-09-24T16:23:03Z

pypesto/ensemble/ensemble.py

+            if y_meas.shape != mean_traj.shape:
+                raise ValueError('Shape of trajectory and shape '
+                                 'of measurements does not match.')


I guess this assumes equal number of timepoints -> same timepoints. However, given the pypesto.objective.AmiciObjective.set_custom_timepoints method, maybe this can't be assumed. Could check timepoints explicitly.

not sure, but shouldn't the whole optimization fail if we set custom timepoints as we could not compute a likelihood anymore? 🤔 correct me if I am wrong.

Ah, often used for predictions rather than methods that require a likelihood.

in that case you wouldn't compute the chi^2 value either I'd say.

pypesto/ensemble/ensemble.py

pypesto/ensemble/constants.py

added option for weighted mean in ensemble

d66aa25

PaulJonasJost requested a review from paulstapor July 12, 2021 16:38

Merge branch 'develop' into feature_ensemble_weights

3aaee3b

PaulJonasJost commented Jul 12, 2021

View reviewed changes

pypesto/predict/amici_predictor.py Outdated Show resolved Hide resolved

Update pypesto/predict/amici_predictor.py

a2f9456

PaulJonasJost commented Jul 12, 2021

View reviewed changes

pypesto/predict/amici_predictor.py Outdated Show resolved Hide resolved

Update pypesto/predict/amici_predictor.py

74769f7

PaulJonasJost requested review from dilpath and yannikschaelte July 13, 2021 06:47

dilpath reviewed Jul 13, 2021

View reviewed changes

paulstapor reviewed Jul 14, 2021

View reviewed changes

PaulJonasJost and others added 14 commits July 15, 2021 08:08

Merge branch 'develop' into feature_ensemble_weights

1c4c388

Merge branch 'develop' into feature_ensemble_weights

a1cdd5e

reworked weightings, added sigmas.

bc3f26c

Merge branch 'feature_ensemble_weights' of https://github.com/ICB-DCM…

66ee92d

…/pyPESTO into feature_ensemble_weights

Merge branch 'develop' into feature_ensemble_weights

7380a92

added default values for ouputs_weights and outputs_sigmay

f875e05

Merge remote-tracking branch 'origin/feature_ensemble_weights' into f…

b2d41ba

…eature_ensemble_weights

readded AMICI_X and AMICI_SX to amici_output fields.

76a6f99

Merge branch 'develop' into feature_ensemble_weights

50f84e1

Merge branch 'develop' into feature_ensemble_weights

0bcd313

Merge branch 'develop' into feature_ensemble_weights

04a5058

added weighted sigmas and chi2 calculation for mean trajectory of ens…

fd48573

…emble.

resolved weighted sigma issue ig no sigmas are provided.

e7267d1

removed weighted_sigmas from cond list if not computed.

5a62fd9

PaulJonasJost requested a review from dilpath September 13, 2021 10:37

PaulJonasJost marked this pull request as ready for review September 13, 2021 10:37

PaulJonasJost requested a review from jvanhoefer September 13, 2021 10:37

jvanhoefer reviewed Sep 24, 2021

View reviewed changes

dilpath approved these changes Sep 24, 2021

View reviewed changes

PaulJonasJost and others added 5 commits September 27, 2021 10:05

added suggestions from review.

211e7d8

raise ValueError if no weights available.

38598db

Merge branch 'develop' into feature_ensemble_weights

6e3ce1d

added changes from review.

669a55b

Merge branch 'develop' into feature_ensemble_weights

afc6aa6

PaulJonasJost merged commit 363c091 into develop Sep 30, 2021

PaulJonasJost deleted the feature_ensemble_weights branch September 30, 2021 10:58

yannikschaelte mentioned this pull request Oct 28, 2021

Release 0.2.8 #740

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added option for weighted mean in ensemble #702

added option for weighted mean in ensemble #702

PaulJonasJost commented Jul 12, 2021

codecov-commenter commented Jul 12, 2021 •

edited

Loading

dilpath left a comment

dilpath Jul 13, 2021

dilpath Jul 13, 2021

PaulJonasJost Jul 13, 2021

dilpath Jul 13, 2021

dilpath Jul 13, 2021

dilpath Jul 13, 2021

paulstapor Jul 14, 2021

dilpath Jul 13, 2021

PaulJonasJost Jul 13, 2021

dilpath Jul 13, 2021

PaulJonasJost Jul 15, 2021

paulstapor Jul 14, 2021

paulstapor Jul 14, 2021

paulstapor Jul 14, 2021

jvanhoefer left a comment

dilpath left a comment

dilpath Sep 24, 2021

PaulJonasJost Sep 27, 2021

dilpath Sep 27, 2021

dilpath Sep 24, 2021

PaulJonasJost Sep 27, 2021

dilpath Sep 27, 2021

PaulJonasJost Sep 28, 2021

		# check if weightings shall be used or not depending on whether llh
		# is part of the prediciton output

	# check if weightings shall be used or not depending on whether llh
	# is part of the prediciton output
	# Check whether LLH-based weights can be used.

		if AMICI_LLH not in self.prediction_results[0].conditions[0].output:
		weighting = False

		output_list = [prediction.conditions[ic].output[AMICI_LLH]
		for prediction in self.prediction_results]

added option for weighted mean in ensemble #702

added option for weighted mean in ensemble #702

Conversation

PaulJonasJost commented Jul 12, 2021

codecov-commenter commented Jul 12, 2021 • edited Loading

Codecov Report

dilpath left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jvanhoefer left a comment

Choose a reason for hiding this comment

dilpath left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 12, 2021 •

edited

Loading