You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
But differ in how the compute the mean and variance across the input:
Batch Normalization - calculated for each individual channel across all samples and spatial dimensions.
reduction_axes: [:batch, :height, :width, ...]
Instance Normalization - calculated for each individual channel for each individual sample across both spatial dimensions.
reduction_axes: [:height, :width, ...]
Layer Normalization - calculated for each individual sample across all channels and both spatial dimensions.
reduction_axes: [:channels, :height, :width, ...]
Group Normalization - calculated across groups of channels and both spatial dimensions for the given group size.
reduction_axes: [:groups, :height, :width, ...] (after some reshaping to get :groups)
Additionally, some of these layers are stateful (batch/instance norm) and some are stateless (layer/group norm). Stateful normalization layers return the transformed input and a running average mean and variance adjusted with momentum, relying on the state to compute the next iteration of normalization. Stateless normalization layers return just the transformed input.
In order to unify these normalization layers under the lower-level functional API, rather than have individualized functions for each layer we will instead have:
The current API has 4 normalization layers:
All of these implementations are built on a fundamental formula:
But differ in how the compute the mean and variance across the input:
reduction_axes: [:batch, :height, :width, ...]
reduction_axes: [:height, :width, ...]
reduction_axes: [:channels, :height, :width, ...]
reduction_axes: [:groups, :height, :width, ...]
(after some reshaping to get:groups
)Additionally, some of these layers are stateful (batch/instance norm) and some are stateless (layer/group norm). Stateful normalization layers return the transformed input and a running average mean and variance adjusted with momentum, relying on the state to compute the next iteration of normalization. Stateless normalization layers return just the transformed input.
In order to unify these normalization layers under the lower-level functional API, rather than have individualized functions for each layer we will instead have:
In the layers API:
normalize
- see aboveIn a separate module:
batch_norm_stats(input, ra_mean, ra_var, opts \\ [])
- returns{mean, var}
instance_norm_stats(input, ra_mean, ra_var, opts \\ [])
- returns{mean, var}
group_norm_stats(input, opts \\ [])
- returns{mean, var}
layer_norm_stats(input, opts \\ [])
- returns{mean, var}
In a separate module (probably an
updates.ex
or something that has gradient/parameter transforms):ema(x, momentum)
- returns a scaledx
, exponential moving averageI think this limits code reuse and still enables us to easily build these normalization layers into a high level API
The text was updated successfully, but these errors were encountered: