Relevance embedding

[taeyeopl]

TTSR/model/SearchTransfer.py

Lines 32 to 33 in 2836600

	R_lv3 = torch.bmm(refsr_lv3_unfold, lrsr_lv3_unfold) #[N, Hr*Wr, H*W]
	R_lv3_star, R_lv3_star_arg = torch.max(R_lv3, dim=1) #[N, H*W]

As I understood related to the equation 4 (in the main paper), your relevance matrix is calculating normalized inner product. r_{i,j} = norm-inner{q_i,k_j}. ( Query is from the up-sampled low-resolution image and Key is from the down/up-sampled reference image. )

My understanding is like the below code.
Q1. Can you explain why your code is opposed?? (Usually, transformer makes the scores using the equation scores = (Q, K^T)).

R_lv3 = torch.bmm(lrsr_lv3_unfold, refsr_lv3_unfold) #[N, H*W, Hr*Wr] 
R_lv3_star, R_lv3_star_arg = torch.max(R_lv3, dim=2) #[N, H*W]

• Reference code from attention-is-all-you-need-PyTorch
  https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/132907dd272e2cc92e3c10e6c4e783a87ff8893d/transformer/Modules.py#L17

[FuzhiYang]

It is ok to change the parameter position of lrsr_lv3_unfold and refsr_lv3_unfold. You can just instead permute lrsr_lv3_unfold in this line:

TTSR/model/SearchTransfer.py

Line 27 in 2836600

refsr_lv3_unfold = refsr_lv3_unfold.permute(0, 2, 1)

.
And then you can apply your code above to get the equal results.

[taeyeopl]

Due to my limited understanding, I do not clearly understand your points.
My understanding is matrix multiplication is not permutation invariant.
Therefore, those operations will make a different relation matrix, even it considers the permute.
Can you explain why at the end of the results you get equal results?
Am I missing something??

R_lv3 = torch.bmm(lrsr_lv3_unfold, refsr_lv3_unfold) #[N, H*W, Hr*Wr] (Oirginal)
R_lv3 = torch.bmm(refsr_lv3_unfold, lrsr_lv3_unfold) #[N, Hr*Wr, H*W] (My Understanding)

[FuzhiYang]

I mean the key point is to get "R_lv3_star_arg" and "R_lv3_star" in

TTSR/model/SearchTransfer.py

Line 33 in 2836600

R_lv3_star, R_lv3_star_arg = torch.max(R_lv3, dim=1) #[N, H*W]

Therefore, how to permute the tensor is not important. But you need to adjust "dim" parameter to get the correct "R_lv3_star_arg" and "R_lv3_star" in this line:

TTSR/model/SearchTransfer.py

Line 33 in 2836600

R_lv3_star, R_lv3_star_arg = torch.max(R_lv3, dim=1) #[N, H*W]

[taeyeopl]

I see, thanks for the detailed explanation !!