如何理解 TensorFlow 中的 tf.extract_image_patches，在 PyTorch 中如何实现？

[daa233]

在 TensorFlow 中有一个 tf.extract_image_patches 函数，这个函数究竟是做了什么操作呢？每一个参数控制着什么？该如何实现对应的 PyTorch 版本？

[daa233]

TF 文档中对函数的介绍如下：

tf.image.extract_image_patches(
    images,
    ksizes,
    strides,
    rates,
    padding,
    name=None
)

• images: A Tensor. Must be one of the following types: float32, float64, int32, uint8, int16, int8, int64, bfloat16, uint16, half, uint32, uint64.
  4-D Tensor with shape [batch, in_rows, in_cols, depth].
• ksizes: A list of ints that has length >= 4.
  The size of the sliding window for each dimension of images.
• strides: A list of ints that has length >= 4.
  1-D of length 4. How far the centers of two consecutive patches are in
  the images. Must be: [1, stride_rows, stride_cols, 1].
• rates: A list of ints that has length >= 4.
  1-D of length 4. Must be: [1, rate_rows, rate_cols, 1]. This is the
  input stride, specifying how far two consecutive patch samples are in the
  input. Equivalent to extracting patches with
  patch_sizes_eff = patch_sizes + (patch_sizes - 1) * (rates - 1), followed by
  subsampling them spatially by a factor of rates. This is equivalent to
  rate in dilated (a.k.a. Atrous) convolutions.
• padding: A string from: "SAME", "VALID".
  The type of padding algorithm to use.

We specify the size-related attributes as:

    ksizes = [1, ksize_rows, ksize_cols, 1]
    strides = [1, strides_rows, strides_cols, 1]
    rates = [1, rates_rows, rates_cols, 1]

name: A name for the operation (optional).

---

上面的函数接口基本上和卷积的操作是一样的，实际上，这种滑动窗口取 patch 的方法就是属于卷积操作的一部分，只不过这里没有 filter，也没有将 patch 和 filter 作卷积的操作。

卷积操作的动态示意图：https://github.com/vdumoulin/conv_arithmetic

返回值是理解这个函数的一个关键。上面函数的返回值是一个维度为 [N, H_L, W_L, k*k*C]，其中 L = H_L * W_L 是总的 patch 数目。

[daa233]

在 PyTorch 中，torch.nn.Unfold 可以实现类似的功能。

torch.nn.Unfold(kernel_size, dilation=1, padding=0, stride=1)

Extracts sliding local blocks from a batched input tensor.

• kernel_size (int or tuple) – the size of the sliding blocks
• stride (int or tuple, optional) – the stride of the sliding blocks in the input spatial dimensions. Default: 1
• padding (int or tuple, optional) – implicit zero padding to be added on both sides of input. Default: 0
• dilation (int or tuple, optional) – a parameter that controls the stride of elements within the neighborhood. Default: 1

---

它的返回值的维度为 [N, C*k*k, L]，其中 L 为 patch 总数。

[daa233]

下面用一个具体的例子来说明：

import tensorflow as tf
tf.enable_eager_execution()
import torch
import numpy as np

# 待提取 patch 的图像，在 TF 和 PyTorch 中有不同的组织方式
# images channel 0
# [[ 0.  1.  2.  3.  4.]
#  [ 5.  6.  7.  8.  9.]
#  [10. 11. 12. 13. 14.]
#  [15. 16. 17. 18. 19.]
#  [20. 21. 22. 23. 24.]]
# images channel 1
# [[25. 26. 27. 28. 29.]
#  [30. 31. 32. 33. 34.]
#  [35. 36. 37. 38. 39.]
#  [40. 41. 42. 43. 44.]
#  [45. 46. 47. 48. 49.]]

# TensorFlow image tensor
tf_images = np.concatenate((np.arange(25).reshape(1, 5, 5, 1).astype(np.float),
                            np.arange(25, 50).reshape(1, 5, 5, 1).astype(np.float)),
                            axis=3)
tf_images = tf.constant(tf_images, dtype=tf.float32)

# PyTorch image tensor
torch_images = np.concatenate((np.arange(25).reshape(1, 1, 5, 5).astype(np.float),
                               np.arange(25, 50).reshape(1, 1, 5, 5).astype(np.float)),
                               axis=1)
torch_images = torch.tensor(torch_images, dtype=torch.float32)

# 指定提取 patch 时的滑动窗口参数
ksizes = 2
strides = 2
rates = 1

# TensorFlow extract patches
tf_patches = tf.extract_image_patches(tf_images,
                                      ksizes=[1, ksizes, ksizes, 1],
                                      strides=[1, strides, strides, 1],
                                      rates=[1, rates, rates, 1],
                                      padding='SAME')  # [N, H_L, W_L, k*k*C]

# PyTorch unfold
def same_padding(images, ksizes, strides, rates):
    assert len(images.size()) == 4
    batch_size, channel, rows, cols = images.size()
    out_rows = (rows + strides[0] - 1) // strides[0]
    out_cols = (cols + strides[1] - 1) // strides[1]
    effective_k_row = (ksizes[0] - 1) * rates[0] + 1
    effective_k_col = (ksizes[1] - 1) * rates[1] + 1
    padding_rows = max(0, (out_rows-1)*strides[0]+effective_k_row-rows)
    padding_cols = max(0, (out_cols-1)*strides[1]+effective_k_col-cols)
    # Pad the input
    padding_top = int(padding_rows / 2.)
    padding_left = int(padding_cols / 2.)
    padding_bottom = padding_rows - padding_top
    padding_right = padding_cols - padding_left
    paddings = (padding_left, padding_right, padding_top, padding_bottom)
    images = torch.nn.ZeroPad2d(paddings)(images)
    return images

# A little different because of the padding
# Maybe we can use a 'SAME' padding like here:
# https://github.com/DAA233/learning-notes/issues/16
torch_images = same_padding(torch_images, (ksizes, ksizes), (strides, strides), (rates, rates))
unfold = torch.nn.Unfold(kernel_size=ksizes,
                         dilation=rates,
                         padding=0,
                         stride=strides)
torch_patches = unfold(torch_images)  # [N, C*k*k, L]


# tf_patches 和  torch_patches 对数据组织的方式不同，但其中的 patch 是一一对应的
# 比如，我们想得到第一个 patch 中第一个通道的值，两种方法可以分别通过：
print(tf.reshape(tf_patches[0, 0, 0, :], (ksizes, ksizes, 2))[:, :, 0])
# tf.Tensor(
# [[0. 1.]
#  [5. 6.]], shape=(2, 2), dtype=float32)
print(torch_patches[0, :, 0].view(2, ksizes, ksizes)[0])
# tensor([[0., 1.],
#         [5., 6.]])

[DamperHa]

您好，通过关键字Unflod查到这里，有个问题想请教您！！

我在实现Dynamic Filter Net的过程出现以下错误，但一直找不出错误原因，错误如下：
#################################
`==> Training Epoch: [1/1000]... Learning Rate: 0.000100
Epoch: [1/1000]: 0%| | 0/37 [00:00<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 136, in
main()
File "train.py", line 73, in main
iter_loss = solver.train_step()
File "/home/ubuntu/fzh/SR/0000/solvers/SRSolver.py", line 98, in train_step
outputs = self.model(self.LR)
File "/home/ubuntu/fzh/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/ubuntu/fzh/anaconda3/envs/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 88, in forward
raise NotImplementedError
NotImplementedError
########################################
这部分的网络模型如下：
##############################
Dynamic_Res(
(conv): Sequential(
(0): ConvTranspose2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): PReLU(num_parameters=1)
)
(con_makefilter): Sequential(
(0): Conv2d(64, 9, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): BatchNorm2d(9, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): Softmax()
)
(compress_out): Sequential(
(0): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1))
(1): PReLU(num_parameters=1)
)
)
################################
这个网络结构块与残差网络类似，主要不同是：

1. 第一个分支的卷积核是通过其他信息产生的，这部分操作通过Unfold+Mul+Fold实现，产生一个featuremap
2. 第二部分是一个反转卷积，产生一个featuremap
3. 以上两部分cat，在做降维
   结构我个人，认为没有什么问题，但是在实现过程中出现上面错误！！！
   苦苦寻找，未有眉目，若能指点，万分感激！！！

[daa233]

@DamperHa 从报错看，像是在运行 outputs = self.model(self.LR) 这句话的时候触发了一个 NotImplementedError，但是你没有给出 self.model 和 self.LR 是什么，所以上面也看不出来到底是哪里出错了。

[DamperHa]

非常感谢，我再试一试~
+——+