Update adlala.py #7
Conversation
pinot/inference/adlala.py
Outdated
| Perform the steps in the following order: | ||
|
|
||
| ``` | ||
| A^1_(h/2) |
There was a problem hiding this comment.
You might note the initialization step(s) as well, since this is a formulation in which the weights and momenta are half a step out of sync during each step.
pinot/inference/adlala.py
Outdated
| # E_(h/2) step: \xi := \xi + 0.5 * h * \epsilon * (p^T p - N * \tao) | ||
| state['xi'].add_( | ||
| 0.5 * group['h'] * group['epsilon'] *\ | ||
| (torch.sum(torch.pow(state['p'].flatten(), 2)) - state['p'].shape[0] * group['tau'])) |
There was a problem hiding this comment.
point of uncertainty: N here should be "number of parameters". is state['p'].shape[0] always equal to the number of parameters? may be safer to do np.prod(state['p'].shape) instead, in case state['p'] is not flat
There was a problem hiding this comment.
p might not always be flat. in this case I'll switch to torch.prod
asked the authors here for further clarification
|
Sorry, wasn't quite done yet! Would still like to make sure there's a test or two here. My changes so far just refactor the substeps and their order in a way that makes them more amenable to review and later generalization. |
No description provided.