Adding a helper method to check model gradients before optimization. #690

stephanmg · 2021-07-01T07:59:46Z

Check for the gradients of the model equations and returns False if gradients are incompatible / not well-suited for gradient-based optimizers otherwise returns True.

codecov-commenter · 2021-07-01T08:08:14Z

Codecov Report

Merging #690 (50930d7) into develop (16639a4) will increase coverage by 55.52%.
The diff coverage is 48.83%.

@@             Coverage Diff              @@
##           develop     #690       +/-   ##
============================================
+ Coverage    34.58%   90.10%   +55.52%     
============================================
  Files           96       96               
  Lines         6466     6507       +41     
============================================
+ Hits          2236     5863     +3627     
+ Misses        4230      644     -3586

Impacted Files	Coverage Δ
pypesto/objective/base.py	`81.95% <11.76%> (+29.14%)`	⬆️
pypesto/objective/amici.py	`82.69% <20.00%> (+62.98%)`	⬆️
pypesto/petab/importer.py	`85.08% <85.71%> (+85.08%)`	⬆️
pypesto/predict/constants.py	`100.00% <0.00%> (+4.34%)`	⬆️
pypesto/optimize/optimize.py	`94.00% <0.00%> (+6.00%)`	⬆️
pypesto/startpoint/uniform.py	`100.00% <0.00%> (+7.14%)`	⬆️
pypesto/optimize/optimizer.py	`90.09% <0.00%> (+7.37%)`	⬆️
pypesto/engine/multi_process.py	`100.00% <0.00%> (+7.40%)`	⬆️
pypesto/optimize/options.py	`100.00% <0.00%> (+10.52%)`	⬆️
pypesto/ensemble/constants.py	`91.83% <0.00%> (+12.24%)`	⬆️
... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16639a4...50930d7. Read the comment docs.

This reverts commit 285aad2.

pypesto/petab/importer.py

Co-authored-by: Paul Jonas Jost <70631928+PaulJonasJost@users.noreply.github.com>

…into check_and_log_gradients

stephanmg · 2021-07-01T11:23:26Z

Added your suggestion @PaulJonasJost - thanks for the pointers.

dweindl

Generally useful. I am only wondering if it might be better to move it to a different place . It doesn't really depend on PEtab. pypesto.Problem? @yannikschaelte ?

dweindl · 2021-07-01T15:32:28Z

pypesto/petab/importer.py

+                            free_indices,
+                            multi_eps=[1e-3, 1e-4, 1e-5],
+                            mode=mode))
+            except (RuntimeError, ValueError):


Short comment why/when this is expected to happen?

Added in commit, see below.

dweindl · 2021-07-01T15:33:02Z

pypesto/petab/importer.py

@@ -78,6 +80,38 @@ def from_yaml(yaml_config: Union[dict, str],
            output_folder=output_folder,
            model_name=model_name)

+    def check_and_log_gradients(self, RTOL: float = 1e-2, ATOL: float = 1e-3):


Please use lower-case argument names

dweindl · 2021-07-01T15:34:26Z

pypesto/petab/importer.py

+        """Check and log if gradients are compatible with gradient-based optimizers
+        Parameters


Suggested change

"""Check and log if gradients are compatible with gradient-based optimizers

Parameters

"""Check and log if gradients match finite differences

Parameters

Changed the documentation brief.

dilpath · 2021-07-01T22:17:03Z

Generally useful. I am only wondering if it might be better to move it to a different place . It doesn't really depend on PEtab. pypesto.Problem? @yannikschaelte ?

Perhaps the ObjectiveBase class, where the gradient check methods are implemented? However, the convenience of automatically using petab_problem.x_nominal_scaled as the test vector would then be lost. This could still be possible by redefining check_and_log_gradients in AmiciObjective, as e.g.:

def check_and_log_gradients(self, x: np.ndarray = None, *args, **kwargs):
    if x is None and 'petab_problem' in dir(self.amici_object_builder):
        x = self.amici_object_builder.petab_problem.x_nominal_scaled
    return super(ObjectiveBase, self).check_and_log_gradients(x=x, *args, **kwargs)

dilpath

👍

Mostly suggestions, feel free to ignore, thanks for the helper method!

dilpath · 2021-07-01T22:21:33Z

pypesto/petab/importer.py

+            try:
+                dfs.append(objective.check_grad_multi_eps(
+                            free_indices,
+                            multi_eps=[1e-3, 1e-4, 1e-5],


Could remove multi_eps from here (check_grad_multi_eps already has a default multi_eps) and replace with *args and **kwargs, so that the user has more control over the gradient check.

dilpath · 2021-07-01T22:23:28Z

pypesto/petab/importer.py

+        modes = [MODE_FUN, MODE_RES]
+        for mode in modes:


This loop over modes could be by default, but also allow users to only specify one of MODE_FUN or MODE_RES, to reduce computation (perhaps not a significant issue except for large problems).

Okay, default is to consider both modes but give the user control over it.

dilpath · 2021-07-01T22:24:40Z

pypesto/petab/importer.py

+                            multi_eps=[1e-3, 1e-4, 1e-5],
+                            mode=mode))
+            except (RuntimeError, ValueError):
+                dfs.append(dfs[0])


Not sure, what does this achieve? Also, it will raise an error if dfs == [] too.

If this is to ensure len(dfs) == 2 for the return statement, then see my suggestion there for an alternative.

dilpath · 2021-07-01T22:26:55Z

pypesto/petab/importer.py

+        objective.amici_solver.setSensitivityMethod(
+            amici.SensitivityMethod_forward)
+        objective.amici_solver.setAbsoluteTolerance(1e-10)
+        objective.amici_solver.setRelativeTolerance(1e-12)


These hard-coded problem-dependent settings should instead be default values that the user can customize (some models will fail at these tolerances, or may be overly computationally-expensive with forward sensitivities). Alternatively, the function could accept an objective that is already customized by the user.

Made these hardcoded values (1e-10 and 1e-12) a user input, allow for pre-configured objective function as user input as well.

dilpath · 2021-07-01T22:27:41Z

pypesto/petab/importer.py

+        RTOL: relative error tolerance (default 1e-2)
+        ATOL: absolute error tolerance (default 1e-3)
+        """
+        par = np.asarray(self.petab_problem.x_nominal_scaled)


Could be by default but also accept a user-supplied vector.

Made as default and allow user defined tolerances.

dilpath · 2021-07-01T22:36:24Z

pypesto/petab/importer.py

+        return np.all((dfs[0].rel_err.values < RTOL) |
+                      (dfs[0].abs_err.values < ATOL) |
+                      (dfs[1].rel_err.values < RTOL) |
+                      (dfs[1].abs_err.values < ATOL))


Suggested change

return np.all((dfs[0].rel_err.values < RTOL) |

(dfs[0].abs_err.values < ATOL) |

(dfs[1].rel_err.values < RTOL) |

(dfs[1].abs_err.values < ATOL))

# Gradients are correct if any of the error types are within their

# respective tolerance, for any of the modes.

return any([

any([

np.all(mode_df.rel_err.values < RTOL),

np.all(mode_df.abs_err.values < ATOL),

])

for mode_df in dfs

])

For another suggestion.

dilpath · 2021-07-01T22:38:03Z

test/petab/test_petab_import.py

@@ -76,6 +76,13 @@ def test_3_optimize(self):
            self.assertTrue(np.isfinite(
                result.optimize_result.get_for_key('fval')[0]))

+    def test_4_gradients(self):


Suggested change

def test_4_gradients(self):

def test_check_and_log_gradients(self):

Ah I thought you want to enumerate the test in the class, as the previous tests are enumerated. Changed!

Ah, didn't notice sorry, fine as before then.

dilpath · 2021-07-01T22:39:27Z

pypesto/petab/importer.py

@@ -1,4 +1,6 @@
+from pypesto.objective.constants import MODE_FUN, MODE_RES


Suggested change

from pypesto.objective.constants import MODE_FUN, MODE_RES

from ..objective.constants import MODE_FUN, MODE_RES

Relative imports are a few lines below, could move this there.

Cleaned up imports.

dilpath · 2021-07-01T22:42:08Z

pypesto/petab/importer.py

@@ -78,6 +80,38 @@ def from_yaml(yaml_config: Union[dict, str],
            output_folder=output_folder,
            model_name=model_name)

+    def check_and_log_gradients(self, RTOL: float = 1e-2, ATOL: float = 1e-3):
+        """Check and log if gradients are compatible with gradient-based optimizers


What is the log functionality of this method?

You are right, the logging is implictly done if log level is specified to check_grad_multi_eps.

stephanmg · 2021-07-02T07:40:12Z

@dilpath @dweindl @PaulJonasJost please compare if I added all your suggestions.

yannikschaelte

Generally useful. I am only wondering if it might be better to move it to a different place . It doesn't really depend on PEtab. pypesto.Problem? @yannikschaelte ?

Perhaps the ObjectiveBase class, where the gradient check methods are implemented? However, the convenience of automatically using petab_problem.x_nominal_scaled as the test vector would then be lost. This could still be possible by redefining check_and_log_gradients in AmiciObjective, as e.g.:
def check_and_log_gradients(self, x: np.ndarray = None, *args, **kwargs):
    if x is None and 'petab_problem' in dir(self.amici_object_builder):
        x = self.amici_object_builder.petab_problem.x_nominal_scaled
    return super(ObjectiveBase, self).check_and_log_gradients(x=x, *args, **kwargs)

sounds good to me, i.e. defining generically in the base class and for amici/petab allowing to use the reference parameter.

yannikschaelte · 2021-07-02T07:46:38Z

pypesto/petab/importer.py

+        objAbsoluteTolerance: float = 1e-10,
+        objRelativeTolerance: float = 1e-12,
+        multi_eps=None,
+        **kwargs


Suggested change

**kwargs

**kwargs,

yannikschaelte · 2021-07-02T07:52:19Z

pypesto/petab/importer.py

+            modes = [mode]
+
+        if multi_eps is None:
+            multi_eps = [1e-3, 1e-4, 1e-5]


in ODE problems, we often find FDs of step size between 1e-1 and 1e-8 stable (see also https://github.com/ICB-DCM/pyPESTO/blob/develop/pypesto/objective/finite_difference.py#L66), so one could define a broader default regime.

dweindl · 2021-07-02T08:04:19Z

pypesto/petab/importer.py

+        """Check if gradients match finite differences (FDs)
+        Parameters


Suggested change

"""Check if gradients match finite differences (FDs)

Parameters

"""Check if gradients match finite differences (FDs)

Parameters

dweindl · 2021-07-02T08:05:05Z

pypesto/petab/importer.py

+        ----------
+        rtol: relative error tolerance
+        atol: absolute error tolerance
+        mode: unction values or residuals


Suggested change

mode: unction values or residuals

mode: function values or residuals

dweindl · 2021-07-02T08:06:44Z

pypesto/petab/importer.py

+        objAbsoluteTolerance: float = 1e-10,
+        objRelativeTolerance: float = 1e-12,


Suggested change

objAbsoluteTolerance: float = 1e-10,

objRelativeTolerance: float = 1e-12,

obj_absolute_tolerance: float = 1e-10,

obj_relative_tolerance: float = 1e-12,

We use snake_case for everything but class names (PEP8). Please also adapt below.

dweindl · 2021-07-02T08:08:13Z

pypesto/petab/importer.py

+
+        Returns
+        -------
+        bool


I'd prefer using return type hints in the function signature instead of specifying it in the docstring.

agreed, call it "match" or something

dweindl · 2021-07-02T08:09:41Z

pypesto/petab/importer.py

+        rtol: float = 1e-2,
+        atol: float = 1e-3,
+        mode: Literal = None,
+        objective: Objective = None,


Suggested change

objective: Objective = None,

objective: Optional[Objective] = None,

dweindl · 2021-07-02T08:09:58Z

pypesto/petab/importer.py

+        *args,
+        rtol: float = 1e-2,
+        atol: float = 1e-3,
+        mode: Literal = None,


Suggested change

mode: Literal = None,

mode: Optional[Literal] = None,

Can't use plain Literal in Optional.

dweindl · 2021-07-02T08:12:56Z

pypesto/petab/importer.py

+                # fails for specified tolerances in forward sensitivities
+                pass
+
+        return any([


Suggested change

return any([

return all([

Shouldn't that be all? If it fails for one mode, I'd consider that a failure.

Yes it should be all, you are right.

dweindl · 2021-07-02T08:15:04Z

pypesto/petab/importer.py

+            except (RuntimeError, ValueError):
+                # Might happen in case PEtab problem not well defined or
+                # fails for specified tolerances in forward sensitivities
+                pass


What will the function return in this case? Or will it raise during the check? I wouldn't not want it return True.

Returning False now instead of pass.

dweindl · 2021-07-02T08:17:18Z

pypesto/petab/importer.py

+                np.all(mode_df.rel_err.values < rtol),
+                np.all(mode_df.abs_err.values < atol),


Hm, if one row passes only the relative tolerance and another row passes only the absolute tolerance, this report failure, right? This would be rather unexpected for me.

So you suggest to make this an or statement?

Suggested change

np.all(mode_df.rel_err.values < rtol),

np.all(mode_df.abs_err.values < atol),

np.all((mode_df.rel_err.values < rtol) | (mode_df.abs_err.values < atol)),

Yes, that's what I just added.

stephanmg · 2021-07-02T08:37:45Z

To which class should the method be moved? ObjectiveBase?

dweindl · 2021-07-02T08:49:26Z

To which class should the method be moved? ObjectiveBase?

Meanwhile it has an AMICI dependency, so ObjectiveBase is rather not suitable anymore. Either that part needs to be thrown out, or it has to go somewhere pypesto.objective.amici*. I think others have a better sense of where it fits than I do.

stephanmg · 2021-07-02T08:50:25Z

To which class should the method be moved? ObjectiveBase?

Meanwhile it has an AMICI dependency, so ObjectiveBase is rather not suitable anymore. Either that part needs to be thrown out, or it has to go somewhere pypesto.objective.amici*. I think others have a better sense of where it fits than I do.

Okay, maybe @yannikschaelte could clarify? Then I will make the changes accordingly. Thank you!

…into check_and_log_gradients

yannikschaelte · 2021-07-02T08:57:19Z

To which class should the method be moved? ObjectiveBase?

Meanwhile it has an AMICI dependency, so ObjectiveBase is rather not suitable anymore. Either that part needs to be thrown out, or it has to go somewhere pypesto.objective.amici*. I think others have a better sense of where it fits than I do.

Okay, maybe @yannikschaelte could clarify? Then I will make the changes accordingly. Thank you!

What amici dependency do you mean? The amici specific forward sensi + tolerance instruction obviously only makes sense for amici, so the base method would not have those, but the rest should work?

dweindl · 2021-07-02T09:01:06Z

What amici dependency do you mean? The amici specific forward sensi + tolerance instruction obviously only makes sense for amici, so the base method would not have those, but the rest should work?

Exactly those. Breaks encapsulation and adds a cyclic dependency.

stephanmg · 2021-07-02T09:02:49Z

What amici dependency do you mean? The amici specific forward sensi + tolerance instruction obviously only makes sense for amici, so the base method would not have those, but the rest should work?

Exactly those. Breaks encapsulation and adds a cyclic dependency.

Yes that's what I noticed (cyclic dependency) as well.

dweindl · 2021-07-02T09:05:19Z

P.S.: I'd rather remove the AMICI part. In my opinion, these settings should have been applied consciously by the user before running any checks here.

stephanmg · 2021-07-02T09:15:11Z

So remove the AMICI part and add a required argument for an AMICIObjective or ObjectiveBase instead to the method?

dilpath · 2021-07-02T09:36:16Z

What amici dependency do you mean? The amici specific forward sensi + tolerance instruction obviously only makes sense for amici, so the base method would not have those, but the rest should work?

Exactly those. Breaks encapsulation and adds a cyclic dependency.

Yes that's what I noticed (cyclic dependency) as well.

This suggestion resolves this right?

#690 (review)

stephanmg · 2021-07-02T13:20:06Z

Removed the Amici dependencies, still have to move functionality to BaseObjective.

stephanmg · 2021-07-05T11:50:50Z

Functionality now moved to BaseObjective, added delegate to amici.py for convenient usage.

yannikschaelte

looks good to me, thanks! feel free to merge when you think it's fine.

yannikschaelte · 2021-07-05T17:40:32Z

pypesto/petab/importer.py

+
+        Returns
+        -------
+        bool


agreed, call it "match" or something

test/petab/test_petab_import.py

yannikschaelte · 2021-07-05T17:44:46Z

test/petab/test_petab_import.py

+        objective.amici_solver.setAbsoluteTolerance(1e-10)
+        objective.amici_solver.setRelativeTolerance(1e-12)
+
+        self.assertFalse(petab_problem.check_gradients(


why is this False again? maybe add a scenario where it's true? this https://github.com/ICB-DCM/pyPESTO/blob/develop/test/util.py#L122 should be stable enough no matter what.

Co-authored-by: Yannik Schälte <31767307+yannikschaelte@users.noreply.github.com>

Adding a helper method to check model gradients before optimization.

d96494b

stephanmg added 4 commits July 1, 2021 10:35

Adding unit test for gradient helper function.

02d6ac3

Fix unit test.

e6512b8

Refactoring return statement.

285aad2

Revert "Refactoring return statement."

e27520b

This reverts commit 285aad2.

PaulJonasJost reviewed Jul 1, 2021

View reviewed changes

pypesto/petab/importer.py Outdated Show resolved Hide resolved

stephanmg and others added 3 commits July 1, 2021 12:42

Update pypesto/petab/importer.py

b258b4d

Co-authored-by: Paul Jonas Jost <70631928+PaulJonasJost@users.noreply.github.com>

Update pypesto/petab/importer.py

51da6a1

Co-authored-by: Paul Jonas Jost <70631928+PaulJonasJost@users.noreply.github.com>

Merge branch 'check_and_log_gradients' of github.com:ICB-DCM/pyPESTO …

9fa10cc

…into check_and_log_gradients

Fix for IndexError.

0b4efb5

stephanmg requested a review from dweindl July 1, 2021 14:38

dweindl reviewed Jul 1, 2021

View reviewed changes

dilpath reviewed Jul 1, 2021

View reviewed changes

stephanmg added 2 commits July 2, 2021 09:33

Addressing reviewers' comments for PR #690.

55dfe21

Providing a default multi_eps.

06fae51

yannikschaelte reviewed Jul 2, 2021

View reviewed changes

dweindl reviewed Jul 2, 2021

View reviewed changes

Addressing most of Yannik's and Daniel's suggestions.

b51c72a

Addressing most of Yannik's and Daniel's suggestions.

0477de3

Merge branch 'check_and_log_gradients' of github.com:ICB-DCM/pyPESTO …

06fb67b

…into check_and_log_gradients

Removing Amici dependency from method.

af1a0ef

stephanmg added 4 commits July 2, 2021 15:25

Small correction.

2e31a74

Moving function to pypesto/objective.base.py

c1a3f57

Merge branch 'develop' into check_and_log_gradients

9355e04

Small refactoring: Names and use of super(...).

2d57e25

stephanmg requested a review from yannikschaelte July 5, 2021 14:32

yannikschaelte approved these changes Jul 5, 2021

View reviewed changes

Update test/petab/test_petab_import.py

50930d7

Co-authored-by: Yannik Schälte <31767307+yannikschaelte@users.noreply.github.com>

stephanmg merged commit c0df8d6 into develop Jul 6, 2021

stephanmg deleted the check_and_log_gradients branch July 6, 2021 06:58

yannikschaelte mentioned this pull request Aug 2, 2021

Release 0.2.7 #672

Merged

		"""Check and log if gradients are compatible with gradient-based optimizers
		Parameters

	def test_4_gradients(self):
	def test_check_and_log_gradients(self):

		@@ -1,4 +1,6 @@
		from pypesto.objective.constants import MODE_FUN, MODE_RES

	from pypesto.objective.constants import MODE_FUN, MODE_RES
	from ..objective.constants import MODE_FUN, MODE_RES

		"""Check if gradients match finite differences (FDs)
		Parameters

	mode: unction values or residuals
	mode: function values or residuals

		objAbsoluteTolerance: float = 1e-10,
		objRelativeTolerance: float = 1e-12,

	objective: Objective = None,
	objective: Optional[Objective] = None,

		np.all(mode_df.rel_err.values < rtol),
		np.all(mode_df.abs_err.values < atol),

	np.all(mode_df.rel_err.values < rtol),
	np.all(mode_df.abs_err.values < atol),
	np.all((mode_df.rel_err.values < rtol) \| (mode_df.abs_err.values < atol)),

Adding a helper method to check model gradients before optimization. #690

Adding a helper method to check model gradients before optimization. #690

Conversation

stephanmg commented Jul 1, 2021 • edited Loading

codecov-commenter commented Jul 1, 2021 • edited Loading

Codecov Report

stephanmg commented Jul 1, 2021 • edited Loading

dweindl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilpath commented Jul 1, 2021 • edited Loading

dilpath left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanmg commented Jul 2, 2021

yannikschaelte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanmg Jul 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanmg Jul 2, 2021 • edited Loading

Choose a reason for hiding this comment

stephanmg commented Jul 2, 2021

dweindl commented Jul 2, 2021

stephanmg commented Jul 2, 2021

yannikschaelte commented Jul 2, 2021

dweindl commented Jul 2, 2021

stephanmg commented Jul 2, 2021 • edited Loading

dweindl commented Jul 2, 2021 • edited Loading

stephanmg commented Jul 2, 2021 • edited Loading

dilpath commented Jul 2, 2021

stephanmg commented Jul 2, 2021 • edited Loading

stephanmg commented Jul 5, 2021

yannikschaelte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephanmg commented Jul 1, 2021 •

edited

Loading

codecov-commenter commented Jul 1, 2021 •

edited

Loading

stephanmg commented Jul 1, 2021 •

edited

Loading

dilpath commented Jul 1, 2021 •

edited

Loading

dilpath left a comment •

edited

Loading

stephanmg Jul 2, 2021 •

edited

Loading

stephanmg Jul 2, 2021 •

edited

Loading

stephanmg commented Jul 2, 2021 •

edited

Loading

dweindl commented Jul 2, 2021 •

edited

Loading

stephanmg commented Jul 2, 2021 •

edited

Loading

stephanmg commented Jul 2, 2021 •

edited

Loading