0.0.6

Highlights

🚀 Ensure equal importance of loss penalties in terms of grad norms in a specific layer 🚀

Other weights can then be applied, e.g. to have a loss term that influences the weights 50% more than the other.

# Given two losses 
loss_a, loss_b

# Compute the loss_b weight
loss_b_weight = calculate_adaptive_weight(loss_a, loss_b, layer)
loss_b_weighted = loss_b_weight * loss_b

# So that loss_a and loss_b have equal importance in the specified layer grads
loss_a_gradnorm = torch.norm(torch.autograd.grad(loss_a, layer, retain_graph=True)[0])
loss_b_weighted_gradnorm = torch.norm(torch.autograd.grad(loss_b_weighted, layer, retain_graph=True)[0])
assert torch.allclose(loss_a_gradnorm, loss_b_weighted_gradnorm)

What's Changed

• Disable pivot sorting in latex df by @lucmos in #12
• Add function to compute adaptive loss weights by @lucmos in #13

Full Changelog: 0.0.5...0.0.6