In the SwiftUI demo the dragging feels stuttery while dragging and buttery smooth while animating. Possibly this was on purpose because the line responsible is offsetAnimator.mode = .nonAnimated that being said I do not think it creates a good example.
I was originally confused that this was the only demo with this problem and thought it was some limitation of SwiftUI and Promotion. However I looked at the internals and everything you're doing with DisplayLink seems correct.
I just copied your pattern from the PIP demo to use an interactive spring while dragging and an animating spring after release and it works great.
AnimationPatch.patch