pdCompositor: finalize rotated layer support

- Layer opacity now works
- Custom blend modes now work
- Mad optimizations are now active.  PD's extensive viewport caching
mechanism is now compatible with affine-transformed layers, so the
viewport should be back to excellent performance, even on images with
many rotated layers.  Huge win here.
- I've added rudimentary antialiasing along rotated borders.  There's
not really a good way to do this without murdering performance, so for
extremely high rotation quality, a destructive transform is inevitable.
But the current solution is still way more than other software offers,
so I'm content to compromise for now.
- Program-wide viewport performance settings are now respected by the
affine transform rendering path.  This should make non-destructive
rotation usable even on terrible hardware, at some (inevitable) cost to
viewport rendering quality.