Subs in Derivatives give incorrect results

[jsharkey13]

I have an expression containing symbols and derivatives parsed from text input which amounts to something like Derivative(y * sin(x), x), where y is supposed to be a function of x but is a plain Symbol object due to the limitations of parsing.

In SymPy 1.1 and prior, it was possible to use the .subs(...) method to replace a Symbol with a Function and then get the expected result of performing the derivative:

>>> from sympy import Function, Derivative, sin, __version__
>>> from sympy.abc import x, y
>>> from sympy.abc import x, y
>>> __version__
'1.1'
>>>
>>> Fy = Function('Fy')(x)
>>>
>>> d1 = Derivative(Fy * sin(x), x)  # This is what we'd expect as result of substitution,
>>> d2 = Derivative(y * sin(x), x).subs(y, Fy)  # try starting with Symbol, sub in Function.
>>>
>>> d1 == d2
True
>>>
>>> d1
Derivative(Fy(x)*sin(x), x)
>>> d2
Derivative(Fy(x)*sin(x), x)
>>> d2.doit()
Fy(x)*cos(x) + sin(x)*Derivative(Fy(x), x)  # Correctly performed differentiation!

where Fy is used for clarity to mean y(x).

In SymPy 1.2 onwards, this is no longer possible, and it is not possible to correctly simplify this derivative:

>>> from sympy import Function, Derivative, sin, __version__
>>> from sympy.abc import x, y
>>> __version__
'1.4'
>>>
>>> Fy = Function('Fy')(x)
>>>
>>> d1 = Derivative(Fy * sin(x), x)  # This is what we expect to get after sub,
>>> d2 = Derivative(y * sin(x), x).subs(y, Fy)  # try starting with Symbol, sub in Function.
>>>
>>> d1 == d2
False
>>>
>>> d1
Derivative(Fy(x)*sin(x), x)
>>> d2
Subs(Derivative(y*sin(x), x), y, Fy(x))
>>> d2.doit()
Fy(x)*cos(x)  # What happened to the product rule?

The substitution happens after the Derivative.doit() call.
There is no way from outside SymPy to ensure it does the substitution before the derivative when calling .doit() since that method does as much as it can, recursively. The result is then incorrect. In fact, when you call .doit() on the outer Subs it explicitly simplifies anything arguments first; unlike some classes you cannot pass deep=False as a hint for Subs to prevent this.

A workaround exists by iterating through every Derivative object in the expression and performing the substitution on args[0] and rebuilding the Derivative (but that won't work well for nested derivatives).

If the outer Subs contains variables in any inner Derivative, then should it not perform the substitution before simplifying the inner value? Or can Subs support the deep=False hint that e.g. Derivative supports in the .doit() method?
I couldn't quite track the regression down to a specific commit, because I don't quite understand all the behaviour under the hood; but it seems to be unaffected by, and unrelated to, the recent changes to how Derivative and Subs work together.

[jsharkey13]

I checked this behaviour occurred in a clean install of 1.2, 1.3 and 1.4 using the python:3.7-slim Docker image (docker run --rm -it python:3.7-slim bash then pip install sympy==1.2, repeat with new container for 1.3 and 1.4).

I then verified that a clean 1.1 install behaved as I expected it to.

[oscarbenjamin]

Not sure about the cause but you might find using replace instead of subs a useful workaround:

In [9]: Fy = Function('Fy')(x)                                                                                                    

In [10]: Derivative(y * sin(x), x).subs(y, Fy).doit()                                                                             
Out[10]: Fy(x)⋅cos(x)

In [11]: Derivative(y * sin(x), x).replace(y, Fy).doit()                                                                          
Out[11]: 
                      d        
Fy(x)⋅cos(x) + sin(x)⋅──(Fy(x))
                      dx

[oscarbenjamin]

Bisect to d71b9e6 from #13803

[jsharkey13]

Thanks for this; .replace() and .xreplace() do seem to do exactly what I want. Perhaps they are in fact the right thing in this particular case? I'm just not clear on the purpose of .replace() vs .subs(), and I'm not sure the docstrings are so clear on when you'd prefer one over the other and why.

The result of .subs() does seem to be what you'd expect if you're swapping one Function of x for another Function of x, so would work fine in more mathematical use cases like change of variable in a derivative or suchlike.

[oscarbenjamin]

I think that the "proper" solution is that y should have been a function of x in the first place. Then subs and doit commute and all is mathematically correct:

In [14]: f1, f2 = symbols('f1, f2', cls=Function)                                                                                 

In [15]: Derivative(f1(x)*sin(x), x).doit().subs(f1, f2)                                                                          
Out[15]: 
                      d        
f₂(x)⋅cos(x) + sin(x)⋅──(f₂(x))
                      dx       

In [16]: Derivative(f1(x)*sin(x), x).subs(f1, f2).doit()                                                                          
Out[16]: 
                      d        
f₂(x)⋅cos(x) + sin(x)⋅──(f₂(x))
                      dx

In [17]: Derivative(f1(x)*sin(x), x).subs(f1, Lambda(x, x**2)).doit()                                                             
Out[17]: 
 2                    
x ⋅cos(x) + 2⋅x⋅sin(x)

Having y as an implicit function of x is always going to be flaky:

In [19]: Derivative(y, x).doit()                                                                                                  
Out[19]: 0