Overlay() calculations such as "add", "subtract", not working properly with float values #461
Description
Internal Overlay incorrect, but works ok with OverlayPlus
https://forum.doom9.org/showthread.php?p=2024698
AVS+ x64 r4289
w=blankclip(pixel_type="RGBPS", colors=[1,1,1])
g=blankclip(pixel_type="RGBPS", colors=[0.5,0.5,0.5])
lg=blankclip(pixel_type="RGBPS", colors=[0.25,0.25,0.25])
#overlay(w,w, mode="add") #1.0 wrong; ? clamp or return first value ?
#overlayplus(w,w, mode="add") #2.0 correct
#overlay(w,lg, mode="subtract") #1.0 wrong; return first value ?
#overlay(lg,w, mode="subtract") #0.25 wrong; return first value ?
#overlayplus(w,lg, mode="subtract") #0.75 correct
#overlayplus(lg,w, mode="subtract") #-0.75 correct
#overlay(w,g, mode="subtract") #1.0 wrong; return first value ?
#overlay(g,w, mode="subtract") #0.5 wrong; return first value ?
#overlayplus(w,g, mode="subtract") #0.5 correct
#overlayplus(g,w, mode="subtract") #-0.5 correct