Passing in function from Python and using it is very slow

[mluogh]

So basically my code more or less looks like this:
It's basically just mapping over a bunch of numbers and summing them up, but the user can provide the map as a python function.

double fn(py::function mapper, .......) {
  for(int i = 0; i < .....){
    // calculate some stuff in c++
    stuff += mapper(variable).cast<double>();
    // calculate some more stuff
  }
}

I've looked over the documentation, and this seems to be the way to do it, but why is it so slow? I think it's due to the overhead associated with casting back and forth, and for the reasons Python is slow (interpreted, dynamic typing, etc). The other parts of the code have sped up significantly, but that function call is dragging it all down.

Is there some way other than writing it in C++ that can make the calling the Python function faster?

Sorry for the Stackoverflow-ish question.

[jagerman]

The slowdown here is mostly due to the python function call overhead; casting adds a bit more. As long as you want to be able to handle Python function calls you can't avoid the Python machinery. You could do something like making mapper a keyword argument that defaults to None with a base implementation in pure C++ for the common case -- but like you said, that's basically writing it in C++, while keeping some flexibility when someone needs the Python function override.

[mluogh]

Thanks a lot!