How to calculate derivative when multiplying matrixes with Eigen?

[manuelreis]

Considering the following snippet of code:

  #include <iostream>
  #include <dCpp.h> 
  #include <dEigen.h>
  #include <Eigen/Dense>

  #define BATCH_SIZE 3
  #define NUM_FEATURES 2
  #define NUM_CLASSES 4

  template<typename A, typename B>
  void print_derivative(A &input, B &w) {
      auto nRows = input.rows();
      auto nCols = input.cols();
      Eigen::MatrixXd toprint(nRows, nCols);
      for (size_t i = 0; i < nRows; ++i){
          for (size_t j = 0; j < nCols; ++j) {
              toprint(i, j) = input(i, j).d(&w);
          }
      }
      std::cout << toprint << std::endl;
  }

  int main() {
      dCpp::initSpace(1);

      // x.W + b
      Eigen::Matrix<dCpp::var, BATCH_SIZE, NUM_FEATURES> x;
      x = Eigen::Matrix<dCpp::var, BATCH_SIZE, NUM_FEATURES>::Ones(BATCH_SIZE, 
  NUM_FEATURES);
      dCpp::init(x);

      Eigen::Matrix<dCpp::var, NUM_FEATURES, NUM_CLASSES> w;
      w = Eigen::Matrix<dCpp::var, NUM_FEATURES, NUM_CLASSES>::Zero(NUM_FEATURES, NUM_CLASSES);
      dCpp::init(w);

      Eigen::Matrix<dCpp::var, BATCH_SIZE, NUM_CLASSES> b;
      b = Eigen::Matrix<dCpp::var, BATCH_SIZE, NUM_CLASSES>::Zero(BATCH_SIZE, NUM_CLASSES);
      dCpp::init(b);


      Eigen::Matrix<dCpp::var, BATCH_SIZE, NUM_CLASSES> y = x*w + b;
      print_derivative(y, w);

      return 0;
  }

The previous snippet of code does not compile because of the following line:

  toprint(i, j) = input(i, j).d(&w);

My question is how, while leveraging dCpp, can I obtain a matrix representing the first derivate of y in respect to w?

[ZigaSajovic]

Think about the assignment

toprint(i, j) = input(i, j).d(&w)

now, toprint(i,j) is a double. But you are trying to assign to it a derivative of a scalar by a matrix. This means that this dInput(i,j)/dW is a matrix. Here input(,j) is the scalar and w is the matrix. If this is not clear to you, I advise you look into matrix calculus. Here is a good start on wiki, showing differences between, differentiating a scalar by a vector/matrix, differentiating a vector/matrix by a scalar, etc.

For this reason, dCpp does not allow you to differentiate with respect to a matrix, but only with respect to a scalar (var). This is clear from the signature of the function _.d(var &).

In other words, you need to differentiate with respect to each element separately, and store those derivatives where you'd like. It should be fairly simple to implement such functionality. Another point is that _.d(var &) returns a var. If you want to have a double, you need to call .id on it, as seen in the examples.