Skip to content

Commit

Permalink
added docs about practical aspect of discretized steerable filters as…
Browse files Browse the repository at this point in the history
… mentioned in #39
  • Loading branch information
Gabri95 committed Mar 13, 2023
1 parent a15d6a9 commit b101341
Show file tree
Hide file tree
Showing 8 changed files with 105 additions and 13 deletions.
82 changes: 82 additions & 0 deletions docs/source/api/conv_notes.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@

Practical Notes on Steerable Dense Convolution
----------------------------------------------

In this section, we include a few notes and practical tips for using
:ref:`discretized convolution layers <steerable-dense-conv>`.

These modules implement **discretized convolution** operators over discrete grids.
This means that **equivariance** to continuous symmetries is **not perfect**.
In practice, by using sufficiently band-limited filters, the equivariance error introduced by the
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.

.. contents::
:local:
:backlinks: top

Data augmentation to prevent propagation of equivariance error
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Unfortunately, the small equivariance error can be leveraged during optimization to break the equivariance in case the
training data does not present the chosen symmetry.
The use of padding can have a similar effect, allowing the neural network to detect the vertical and horizontal axis.
For these reasons, **we recommend to use data augmentation** when training models equivariant to continuous symmetries
on discretized data.


Downsampling can break equivariance to the exact symmetries of the grid
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the architecture involves **downsampling** of intermediate features (e.g. via *strided convolution*),
some design choices **may also break the equivariance** to the discrete symmetries of the grid.
For instance, using odd-sized convolutional filters with ``stride=2`` over an even-sized image will break the
equivariance to the 90 degrees rotations of the input, see Figure 2 `here <https://arxiv.org/abs/2004.09691>`_ for
more details.
We recommend working with *odd-sized images/features* in such cases.


Default steerable basis
~~~~~~~~~~~~~~~~~~~~~~~

The parameters ``sigma``, ``frequencies_cutoff`` and ``rings`` of
:class:`~escnn.nn.modules.conv._RdConv` (and its subclasses :class:`~escnn.nn.R2Conv` or
:class:`~escnn.nn.R3Conv`) are optional parameters used to control how the basis for the steerable filters is built,
how it is sampled on the filter grid and how it is expanded to build the filter.
These **default values** work well in most cases since we tuned them manually in a few practical settings
(odd-sized filters, with relatively small kernel size), so we don't recommend changing them.
In case you may want to adapt them for different use cases (e.g. even-sized or really wide filters), we provide
more details about them below.

The default steerable basis used by the convolution layers is built by using :class:`~escnn.kernels.CircularShellsBasis`
or :class:`~escnn.kernels.SphericalShellsBasis` so steerable filters are in general split in multiple rings,
see Figure 2 `here <https://arxiv.org/pdf/1711.07289.pdf>`_ .

The parameter ``rings`` is a list of float values, defining the radius of the different rings the filter is split into.
The parameter ``sigma`` defines the width of each of them.
You can pass a list containing a different sigma per radius or just a singular float value which is used for all rings.
The parameter ``frequencies_cutoff`` regulates the maximum frequency on each ring.
By default, it uses a policy we manually tuned.
If you pass a float value ``frequencies_cutoff=F``, the maximum frequency at radius ``r`` becomes ``int(r * F)``.
Alternative, you can pass a generic function accepting a radius ``r`` in input and return the maximum frequency
accepted at that radius.



Even-sized filters and ``kernel_size=2``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

While even-sized convolutional filters (``kernel_size`` is an even number) are supported,
they are generally less stable to continuous rotations.
This is partially because the default parameters of the steerable basis were manually tuned for odd-sized filters.
In particular, if ``kernel_size=2``, the resulting grid is too small to sample any filter in the constructed
steerable basis: as a result, the sampled basis is empty and an error is raised.
**We don't recommend using this grid** if your goal is achieving equivariance to all rotations.

A ``kernel_size=2`` filter can be useful if you only care about equivariance to :math:`90\deg` rotations.
In that case, you can play with the ``sigma``, ``rings`` and ``frequencies_cutoff`` parameters of
:class:`~escnn.nn.modules.conv.R2Conv` or :class:`~escnn.nn.modules.conv.R3Conv` to build a non-empty basis.
By tuning these parameters, you might be able to get a basis stable enough to continuous rotations for
your application, but we didn't experiment with this setting enough to provide any recommendation.
You can find a useful discussion about this in `this issue <https://github.com/QUVA-Lab/e2cnn/issues/18>`_ .

19 changes: 6 additions & 13 deletions docs/source/api/escnn.nn.rst
Original file line number Diff line number Diff line change
Expand Up @@ -133,26 +133,19 @@ Linear
:members:
:show-inheritance:


.. _steerable-dense-conv:

Steerable Dense Convolution
---------------------------

The following modules implement *discretized* convolution operators over discrete grids.
This means that equivariance to continuous symmetries is *not* perfect.
The following modules implement **discretized convolution** operators over discrete grids.
This means that **equivariance** to continuous symmetries is **not perfect**.
In practice, by using sufficiently band-limited filters, the equivariance error introduced by the
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.

Unfortunately, the small equivariance error can be leveraged during optimization to break the equivariance in case the
training data does not present the chosen symmetry.
For this reason, we recommend to use data augmentation when training models equivariant to continuous symmetries on
discretized data.

Additionally, if the architecture involves downsampling of intermediate features (e.g. via *strided convolution*),
some design choices may also break the equivariance to the discrete symmetries of the grid.
For instance, using odd-sized convolutional filters with ``stride=2`` over an even-sized image will break the
equivariance to the 90 degrees rotations of the input, see Figure 2 `here <https://arxiv.org/abs/2004.09691>`_ for
more details.
We recommend working with odd-sized images/features in such cases.
We also provide some :doc:`practical notes <conv_notes>` on using these discretized convolution modules.


.. contents::
Expand Down
3 changes: 3 additions & 0 deletions escnn/nn/modules/conv/r2_transposed_convolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ def __init__(self,
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.
We provide some :doc:`practical notes <conv_notes>` on using this discretized
convolution module.
.. warning ::
Expand Down
3 changes: 3 additions & 0 deletions escnn/nn/modules/conv/r2convolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,9 @@ def __init__(self,
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.
We provide some :doc:`practical notes <conv_notes>` on using this discretized
convolution module.
During training, in each forward pass the module expands the basis of G-steerable kernels with learned weights
before calling :func:`torch.nn.functional.conv2d`.
When :meth:`~torch.nn.Module.eval()` is called, the filter is built with the current trained weights and stored
Expand Down
3 changes: 3 additions & 0 deletions escnn/nn/modules/conv/r3_transposed_convolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ def __init__(self,
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.
We provide some :doc:`practical notes <conv_notes>` on using this discretized
convolution module.
.. warning ::
Transposed convolution can produce artifacts which can harm the overall equivariance of the model.
Expand Down
2 changes: 2 additions & 0 deletions escnn/nn/modules/conv/r3convolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ def __init__(self,
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.
We provide some :doc:`practical notes <conv_notes>` on using this discretized
convolution module.
During training, in each forward pass the module expands the basis of G-steerable kernels with learned weights
before calling :func:`torch.nn.functional.conv2d`.
Expand Down
3 changes: 3 additions & 0 deletions escnn/nn/modules/conv/rd_convolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ def __init__(self,
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.
We provide some :doc:`practical notes <conv_notes>` on using this discretized
convolution module.
During training, in each forward pass the module expands the basis of G-steerable kernels with learned weights
before performing the convolution.
When :meth:`~torch.nn.Module.eval()` is called, the filter is built with the current trained weights and stored
Expand Down
3 changes: 3 additions & 0 deletions escnn/nn/modules/conv/rd_transposed_convolution.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ def __init__(self,
discretization of the filters and the features is contained, but some design choices may have a negative
effect on the overall equivariance of the architecture.
We provide some :doc:`practical notes <conv_notes>` on using this discretized
convolution module.
.. warning ::
Expand Down

0 comments on commit b101341

Please sign in to comment.