Time per epoch increases substantially for high order (4th) targets

[PieterGimbel]

I observed that the training time increases a lot if I added 4th order targets. During a quick check roughly the following training times were observed for the corresponding targets (1000 epochs):

• 3x 0th order: < 1s
• 2x 0th + 1x 4th: ~ 30s
• 1x 0th + 2x 4th: ~ 100s

Is having high order targets that much more computational demanding? Are there ways to improve on this? Enabling GPU does not seem to matter.

[sciann]

This is normal. Note that 4th differentiation results in an extensive graph and therefore computational time increase.
To avoid this, you can reduce the required order of differentiation by introducing new variables.

[PieterGimbel]

Thank you for your response. Do you mean introducing new variables in the following way?

x = sn.Variable(...)
w = sn.Functional(...)
w1x = sn.diff(w, x, order=1)
w2x = sn.diff(w1x, x, order=1)
w3x = sn.diff(w2x, x, order=1)
w4x = sn.diff(w3x, x, order=1)

This is how I was taking the 4th order derivative, because I noticed a time reduction in comparison to taking the 4th order derivative directly (w4x = sn.diff(w, x, order=4)). However, it was not a very impressive reduction.

[sciann]

That might help but it still deals with 4th order derivatives that result in long computational graphs because of the chain rule.

Not sure if possible for your problem, but I meant to introduce intermediate functions. For instance, if you want to limit the differentiation order to two, you may do the following:

`
x = sn.Variable(...)
w = sn.Functional(...)
w1x = sn.diff(w, x, order=1)
w2x = sn.diff(w1x, x, order=1)

y = sn.Functional(...)
L1 = w2x - y

w3x = sn.diff(y, x, order=1)
w4x = sn.diff(y, x, order=1)
`