Wavelet packet transform -urgent help required

[urja02]

Hello, i was working on a research project and i wanted to ask you what is the exact output of the wavelet packet transform . Do we get outputs such that they can directly be concatenatd with convolution layer? I need to fuse the frequency domain features with spatial domain.I need more information about how the wavelet packet transform outputs look like.
Thank you in advance for your reply

[v0lta]

Dear @urja02,
Are you working with the two-dimensional packet transform?

If you do, you could use an example to explore the output shapes. I.e.:

from itertools import product
import ptwt, pywt, torch
import numpy as np
import scipy.misc
 
face = np.transpose(scipy.misc.face()[:256, :256, :],
                [2, 0, 1]).astype(np.float64)
pytorch_face = torch.tensor(face).unsqueeze(1)
packets = ptwt.WaveletPacket2D(pytorch_face, pywt.Wavelet("Haar"))

level = 3
paths = list(product(["a", "h", "v", "d"], repeat=level))
packet_list = []
for node in paths:
     packet = torch.squeeze(packets["".join(node)], dim=1)
     packet_list.append(packet)
wp_pt = torch.stack(packet_list, dim=1)

Should give you the shape: [3, 64, 32, 32] for wp_pt.
You can concatenate wavelet packets and convolution filters.
See https://rdcu.be/cUIRt for an example of where this is useful. Table 1 in particular presents a possible architecture.

[urja02]

Hello, thank you so much for your reply. Yes i have gone through this paper,could you please give me and insight on how the the packets are combined with the filters? The paper talks about wavelet coefficients which are compared,I used the same code as in the deepfake and it worked but im not getting the intuitive idea behind it.

…

On Sun, 23 Oct 2022, 01:06 Moritz Wolter, ***@***.***> wrote: Dear @urja02 <https://github.com/urja02>, Are you working with the two-dimensional packet transform? You could use an example to explore the output shapes. I.e.: from itertools import productimport ptwt, pywt, torchimport numpy as npimport scipy.misc face = np.transpose(scipy.misc.face()[:256, :256, :], [2, 0, 1]).astype(np.float64)pytorch_face = torch.tensor(face).unsqueeze(1)packets = ptwt.WaveletPacket2D(pytorch_face, pywt.Wavelet("Haar")) level = 3paths = list(product(["a", "h", "v", "d"], repeat=level))packet_list = []for node in paths: packet = torch.squeeze(packets["".join(node)], dim=1) packet_list.append(packet)wp_pt = torch.stack(packet_list, dim=1) Should give you the shape: [3, 64, 32, 32] for wp_pt. You can concatenate wavelet packets and convolution filters. See https://rdcu.be/cUIRt for an example of where this is useful. Table 1 in particular presents a possible architecture. — Reply to this email directly, view it on GitHub <#43 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRUSM2PMSVNYRG5DWIABJTWEQ63HANCNFSM6AAAAAARLFPMLI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[urja02]

Yes i am working with 2d packet transform.

…

On Sun, 23 Oct 2022, 08:05 urja giridhar, ***@***.***> wrote: Hello, thank you so much for your reply. Yes i have gone through this paper,could you please give me and insight on how the the packets are combined with the filters? The paper talks about wavelet coefficients which are compared,I used the same code as in the deepfake and it worked but im not getting the intuitive idea behind it. On Sun, 23 Oct 2022, 01:06 Moritz Wolter, ***@***.***> wrote: > Dear @urja02 <https://github.com/urja02>, > Are you working with the two-dimensional packet transform? > > You could use an example to explore the output shapes. I.e.: > > from itertools import productimport ptwt, pywt, torchimport numpy as npimport scipy.misc > face = np.transpose(scipy.misc.face()[:256, :256, :], > [2, 0, 1]).astype(np.float64)pytorch_face = torch.tensor(face).unsqueeze(1)packets = ptwt.WaveletPacket2D(pytorch_face, pywt.Wavelet("Haar")) > level = 3paths = list(product(["a", "h", "v", "d"], repeat=level))packet_list = []for node in paths: > packet = torch.squeeze(packets["".join(node)], dim=1) > packet_list.append(packet)wp_pt = torch.stack(packet_list, dim=1) > > Should give you the shape: [3, 64, 32, 32] for wp_pt. > You can concatenate wavelet packets and convolution filters. > See https://rdcu.be/cUIRt for an example of where this is useful. Table > 1 in particular presents a possible architecture. > > — > Reply to this email directly, view it on GitHub > <#43 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APRUSM2PMSVNYRG5DWIABJTWEQ63HANCNFSM6AAAAAARLFPMLI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[v0lta]

The wavelet transforms in CNN create an invariant representation. I think the idea first appeared in https://arxiv.org/pdf/1203.1513.pdf . Adding an invariant representation can speed up convergence if the final result was going to resemble wavelets. In the deep-fake detection paper linked above, we also saw drops in the amount of training data required.

[urja02]

Thank you for the explanation. I have a question, what does invariant representation mean?

…

On Mon, 24 Oct 2022, 16:21 Moritz Wolter, ***@***.***> wrote: The wavelet transforms in CNN create an invariant representation. I think the idea first appeared in https://arxiv.org/pdf/1203.1513.pdf . Adding an invariant representation can speed up convergence if the final result was going to resemble wavelets. In the deep-fake detection paper, we also saw drops in the amount of training data required. — Reply to this email directly, view it on GitHub <#43 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRUSMYK4XC7ZYVI3CALWSTWEZSZHANCNFSM6AAAAAARLFPMLI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[v0lta]

The representation is not optimized, so all filter values are constant.

[urja02]

I have added the wavelet packet transform units in a residual manner in efficientdet. I have added it just before the biFPN layer. Does the invariant representation help in object detection.

…

On Mon, 24 Oct 2022, 18:53 Moritz Wolter, ***@***.***> wrote: The representation is not optimized, so all filter values are constant. — Reply to this email directly, view it on GitHub <#43 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRUSMYBTYOAYWEURLWWHP3WE2EU7ANCNFSM6AAAAAARLFPMLI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[urja02]

Does that mean that an object can be detected even when the scene changes

…

On Mon, 24 Oct 2022, 18:53 Moritz Wolter, ***@***.***> wrote: The representation is not optimized, so all filter values are constant. — Reply to this email directly, view it on GitHub <#43 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRUSMYBTYOAYWEURLWWHP3WE2EU7ANCNFSM6AAAAAARLFPMLI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[v0lta]

It means the wavelet filters are not optimized. The transformation is invertible it does not lose information. When the scene changes the output of the wavelet-transform changes accordingly. I haven't tried wavelet representations for object recognition tasks. Generally, I have seen that static representations can improve convergence. In terms of performance, they tend to help more on smaller data sets.