Skip to content

Matrix Calculus via Differentials, Matrix Derivative, 矩阵求导教程

Notifications You must be signed in to change notification settings

LynnHo/Matrix-Calculus-Tutorial

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

90 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Matrix Calculus

In this page, we introduce a differential based method for vector and matrix derivatives (matrix calculus), which only needs a few simple rules to derive most matrix derivatives. This method is useful and well established in mathematics; however, few documents clearly or detailedly describe it. Therefore, we make this page aiming at the comprehensive introduction of matrix calculus via differentials.

* If you want results only, there is an awesome online tool Matrix Calculus. If you want "how to," let's get started.

0. Notation

  • , , and denote , , and respectively.
  • The first half of the alphabet denote constants, and the second half denote variables.
  • denotes matrix transpose, is the trace, is the determinant, and is the adjugate matrix.
  • is the Kronecker product and is the Hadamard product.
  • Here we use numerator layout, while the online tool Matrix Calculus seems to use mixed layout. Please refer to Wiki - Matrix Calculus - Layout Conventions for the detailed layout definitions, and keep in mind that different layouts lead to different results. Below is the numerator layout,

1. Matrix Calculus via Differentials

1.1 Differential Identities

  • Identities 1

  • Identities 2

  • Identities 3 - chain rules

  • Identities 4 - total differential. Actually, all identities 1 are the matrix form of the total differential in eq. (24).

1.2 Deriving Matrix Derivatives

To derive a matrix derivative, we repeat using the identities 1 (the process is actually a chain rule) assisted by identities 2.

1.2.1 Proof of chain rules (identities 3)

finally from eq. (2), we get .

finally from eq. (3), we get .

finally from eq. (1), we get .

finally from eq. (5), we get .

1.2.2 Practical examples

E.g. 1,

finally from eq. (2), we get .

E.g. 2,

finally from eq. (3), we get . From line 3 to 4, we use the conclusion of , that is to say, we can derive more complicated matrix derivatives by properly utilizing the existing ones. From line 6 to 7, we use to introduce the in order to use eq. (3) later, which is common in scalar-by-matrix derivatives.

E.g. 3,

finally from eq. (3), we get .

E.g. 4,

finally from eq. (3), we get .

E.g. 5 - two layer neural network, , is a loss function such as Softmax Cross Entropy and MSE, is an element-wise activation function such as Sigmoid and ReLU

For ,

finally from eq. (3), we get .

For ,

finally from eq. (3), we get .

E.g. 6, prove

Since

then

therefore

* See examples.md for more examples.

2. Conclusion

Now, if we fully understand the core mind of the above examples, I believe we can derive most matrix derivatives in Wiki - Matrix Calculus by ourself. Please correct me if there is any mistake, and raise issues to request the detailed steps of computing the matrix derivatives that you are interested in.

About

Matrix Calculus via Differentials, Matrix Derivative, 矩阵求导教程

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published