Custom Optimizer and LR schedulers #290

wistuba · 2023-06-09T08:40:01Z

Optimizer and LR scheduler can be defined via Renate config file.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

github-actions · 2023-06-09T17:38:35Z

Coverage report

The coverage rate went from 85.68% to 85.16% ⬇️

100% of new lines are covered.

Diff Coverage details (click to unfold)

src/renate/updaters/experimental/repeated_distill.py

100% of new lines are covered (100% of the complete file).

src/renate/utils/module.py

100% of new lines are covered (93.75% of the complete file).

src/renate/shift/detector.py

100% of new lines are covered (89.74% of the complete file).

src/renate/updaters/experimental/fine_tuning.py

100% of new lines are covered (100% of the complete file).

src/renate/updaters/experimental/gdumb.py

100% of new lines are covered (94.44% of the complete file).

src/renate/utils/optimizer.py

100% of new lines are covered (100% of the complete file).

src/renate/cli/parsing_functions.py

100% of new lines are covered (79.18% of the complete file).

src/renate/cli/run_training.py

100% of new lines are covered (96.92% of the complete file).

src/renate/updaters/experimental/offline_er.py

100% of new lines are covered (81.08% of the complete file).

src/renate/updaters/model_updater.py

100% of new lines are covered (95.2% of the complete file).

src/renate/updaters/experimental/joint.py

100% of new lines are covered (95.83% of the complete file).

src/renate/updaters/learner.py

100% of new lines are covered (100% of the complete file).

src/renate/defaults.py

100% of new lines are covered (99% of the complete file).

src/renate/updaters/avalanche/model_updater.py

100% of new lines are covered (94.57% of the complete file).

src/renate/memory/storage.py

100% of new lines are covered (46.42% of the complete file).

src/renate/updaters/experimental/er.py

100% of new lines are covered (91.61% of the complete file).

src/renate/utils/distributed_strategies.py

100% of new lines are covered (90.9% of the complete file).

src/renate/shift/ks_detector.py

100% of new lines are covered (100% of the complete file).

src/renate/cli/parsing_functions.py

prabhuteja12 · 2023-06-12T16:56:06Z

src/renate/updaters/avalanche/model_updater.py

+        lr_scheduler_plugin = None
+        optimizers_scheduler = self._dummy_learner.configure_optimizers()
+        if isinstance(optimizers_scheduler, tuple):
+            optimizer, scheduler_config = optimizers_scheduler


What do these two lines do?

Return value can either be Tuple[Optimizer, Scheduler] or Optimizer. We are checking for this case and conditionally add a LRCallback.

prabhuteja12

The self.save_hyperparameters must now include the optimizer object.

prabhuteja12 · 2023-06-12T17:12:53Z

test/conftest.py

@@ -421,6 +290,8 @@ def get_simple_updater(
    metric=None,
    deterministic_trainer=False,
 ):
+    if learner_kwargs is None:


Why this form?

I don't understand. Are you asking why we compare with is None?

Yes. Is there a test that populates different learner_kwargs

yes, e.g., test_joint.py L27. This change is not changing the behavior but rather avoiding the unwanted behavior when using dicts and the likes as defaults in functions.

prabhuteja12 · 2023-06-12T17:16:37Z

src/renate/updaters/avalanche/model_updater.py

-        learning_rate_scheduler_step_size: int = defaults.LEARNING_RATE_SCHEDULER_STEP_SIZE,
-        momentum: float = defaults.MOMENTUM,
-        weight_decay: float = defaults.WEIGHT_DECAY,
+        learning_rate_scheduler: Optional[partial] = None,


Type Callable?

wistuba · 2023-06-12T18:42:36Z

The self.save_hyperparameters must now include the optimizer object.

I've also ignored the lr_scheduler

prabhuteja12 · 2023-06-14T08:17:55Z

doc/getting_started/how_to_run_training.rst

@@ -47,6 +47,11 @@ instantiating the method you selected. See :doc:`supported_algorithms` for more



+..  note::
+    If you have defined the ``optimizer_fn`` function in your Renate config, do not pass values for the keys


Is there a check for this in the code?

Yes. run_training_job.py will not add these params in the argparse code and therefore the script will fail if these arguments are passed but not expected.

custom optimizer and lr schedulers can be added to the config file

0c39d18

wistuba changed the title ~~custom optimizer and lr schedulers can be added to the config file~~ Custom Optimizer and LR schedulers Jun 9, 2023

wistuba added 10 commits June 9, 2023 11:16

fix problem with fine tuning

8fba5e6

remove lr scheduler argparse args

f3d9fc0

not in

66d7911

flake

05430d1

flake

ddbcec7

update tests

3c175b0

update test

8dcbe88

fix tests

4002e08

final fix

a632a33

final fix

7bcb83f

test config file with custom optimizer and lr scheduler

0851a06

wistuba marked this pull request as ready for review June 10, 2023 16:35

wistuba requested review from lballes and prabhuteja12 June 10, 2023 16:35

prabhuteja12 reviewed Jun 12, 2023

View reviewed changes

src/renate/cli/parsing_functions.py Show resolved Hide resolved

prabhuteja12 reviewed Jun 12, 2023

View reviewed changes

prabhuteja12 requested changes Jun 12, 2023

View reviewed changes

prabhuteja12 reviewed Jun 12, 2023

View reviewed changes

wistuba added 2 commits June 12, 2023 20:28

partial -> Callable

78b41de

generator -> list, ignore optimizer and lr_scheduler

65a85c8

wistuba added 2 commits June 12, 2023 21:04

fixing docs

f097de6

fix typing

3478623

wistuba requested a review from prabhuteja12 June 13, 2023 09:10

prabhuteja12 reviewed Jun 14, 2023

View reviewed changes

wistuba requested a review from prabhuteja12 June 14, 2023 09:54

prabhuteja12 approved these changes Jun 14, 2023

View reviewed changes

wistuba merged commit 15541cf into dev Jun 14, 2023
18 checks passed

wistuba deleted the mw-custom-opt-scheduler branch June 14, 2023 14:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom Optimizer and LR schedulers #290

Custom Optimizer and LR schedulers #290

wistuba commented Jun 9, 2023

github-actions bot commented Jun 9, 2023 •

edited

src/renate/updaters/experimental/repeated_distill.py

src/renate/utils/module.py

src/renate/shift/detector.py

src/renate/updaters/experimental/fine_tuning.py

src/renate/updaters/experimental/gdumb.py

src/renate/utils/optimizer.py

src/renate/cli/parsing_functions.py

src/renate/cli/run_training.py

src/renate/updaters/experimental/offline_er.py

src/renate/updaters/model_updater.py

src/renate/updaters/experimental/joint.py

src/renate/updaters/learner.py

src/renate/defaults.py

src/renate/updaters/avalanche/model_updater.py

src/renate/memory/storage.py

src/renate/updaters/experimental/er.py

src/renate/utils/distributed_strategies.py

src/renate/shift/ks_detector.py

prabhuteja12 Jun 12, 2023

wistuba Jun 12, 2023

prabhuteja12 left a comment

prabhuteja12 Jun 12, 2023

wistuba Jun 12, 2023

prabhuteja12 Jun 14, 2023

wistuba Jun 14, 2023

prabhuteja12 Jun 12, 2023

wistuba Jun 12, 2023

wistuba commented Jun 12, 2023

prabhuteja12 Jun 14, 2023

wistuba Jun 14, 2023

		@@ -47,6 +47,11 @@ instantiating the method you selected. See :doc:`supported_algorithms` for more



		.. note::
		If you have defined the ``optimizer_fn`` function in your Renate config, do not pass values for the keys

Custom Optimizer and LR schedulers #290

Custom Optimizer and LR schedulers #290

Conversation

wistuba commented Jun 9, 2023

github-actions bot commented Jun 9, 2023 • edited

Coverage report

src/renate/updaters/experimental/repeated_distill.py

src/renate/utils/module.py

src/renate/shift/detector.py

src/renate/updaters/experimental/fine_tuning.py

src/renate/updaters/experimental/gdumb.py

src/renate/utils/optimizer.py

src/renate/cli/parsing_functions.py

src/renate/cli/run_training.py

src/renate/updaters/experimental/offline_er.py

src/renate/updaters/model_updater.py

src/renate/updaters/experimental/joint.py

src/renate/updaters/learner.py

src/renate/defaults.py

src/renate/updaters/avalanche/model_updater.py

src/renate/memory/storage.py

src/renate/updaters/experimental/er.py

src/renate/utils/distributed_strategies.py

src/renate/shift/ks_detector.py

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prabhuteja12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wistuba commented Jun 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 9, 2023 •

edited