About the covariance matrix

[Lzyin]

Thank you for your interesting work!

The section on Cross-covariance Loss states: "The cross-covariance is expected to exhibit an identity matrix, as the feature pair should contain identical information." I am confused as to why a feature pair containing identical information implies that the optimal solution for the covariance matrix of the feature pair is an identity matrix?

I sincerely hope to get your reply.

[wjahn]

Thank you for your interest in our research.

It will be helpful to explain it first for single image. Given an image $x$ and its normalized feature $\bar{F}$, the diagonal component of self-covariance $\bar{F} \cdot \bar{F}^T$ should be 1, since the feature vectors are being inner-product with itself.

Expanding the concept, given the image $x$ 와 augmented image $x_a$ (same content different style). The objective is for the network to extract the same information from both $x$ and $x_a$. To do so, we use the cross-cross-covariance to be identity matrix (only for the diagonal component) to make only the contents the same.

$\bar{F}$ = $\bar{F}_a$ is the best solution, but when we used this as the objective function, the network's performance decreased (as far as i remember). So, we approached the core of DG by focusing on style and content.

[rsleishaui]

Does this apply to other modalities of data, such as point clouds?

[wjahn]

@rsleishaui Could you be more specific please? For 3D-CNN features, I would say yes, but I'm not sure if point clouds have "style information".

[rsleishaui]

I can create style data through data augmentation. I have written covariance matching loss and cross-covariance matching loss based on your paper. I did not find the implementation of these two loss functions in your code, and now there is a gradient explosion problem. Do you have any suggestions?

[wjahn]

@rsleishaui Please refer BlindNet/network/deepv3.py line 629 (CML, CCL).

I don't know much about point clouds, but if there is a paper on style transfer functions in point clouds, I think it can be modified and used.

[rsleishaui]

Could you tell me where is your code for the covariance matching loss?