Skip to content

Commit

Permalink
Synchronize updates; fix AdamW lr_t (keras)
Browse files Browse the repository at this point in the history
**BUGFIXES**:
 - Last weight in network would be updated with `t_cur` one update ahead, desynchronizing it from all other weights
 - `AdamW` in `keras` (optimizers.py, optimizers_225.py) weight updates were _not_ mediated by `eta_t`, so cosine annealing had no effect. Pardon the mishap

**FEATURES**:
 - Added `lr_t` to tf.keras optimizers to track "actual" learning rate externally; use `K.eval(model.optimizer.lr_t)` to get "actual" learning rate for given `t_cur` and `iterations`
 - Added `lr_t` vs. iterations plot to README, and source code in `example.py`

**MISC**:
 - Added `test_updates` to ensure all weights update synchronously, and that `eta_t` first applies on weights as-is and _then_ updates according to `t_cur`
 - Fixes #47
  • Loading branch information
OverLordGoldDragon committed Jun 4, 2020
1 parent 8dc42e0 commit 8d362e2
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion keras_adamw/__init__.py
Expand Up @@ -28,4 +28,4 @@
from .utils import get_weight_decays, fill_dict_in_order
from .utils import reset_seeds, K_eval

__version__ = '1.31'
__version__ = '1.32'

0 comments on commit 8d362e2

Please sign in to comment.