rsx: Implement separable MRT masks

[kd-11]

• Allows modifying write masks individually for each RTT as is done on PS3. Discovered when investigating Missing health bars on Jojo [BLUS31405] on AMD after #4265 #4380
• Also sets up explicit memory pools for GL buffers in preparation for compute integration.
• Clean up unused OGL code.
• Hotfix for crashing capture replay.

Fixes #4380

[MarioSonic2987]

Fixes black textures, weird repeated image and black holes in GTA V.

Before:

After:

[incognitoh]

regression with Vulkan: Toy Story 3

Now has basically a black screen on the main menu/hub area. Ingame graphics also regressed. Cleared shader cache and tried different settings like blit, wcb, rcb, disable async etc..same result. however OGL seems to have not regressed.

This game is stubborn as it also still has the original regression from about a year ago #5526

gpu: gtx1070

[stride21]

This PR fixes black textures in Infamous 2 though some of them only partially . The transparent parts of the partial fixed textures change color and look as if their flickering as the camera moves.

Before:

After:

Before:

After:

[kd-11]

@incognitoh RSX capture needed.

[MarioSonic2987]

It also fixes black textures in Infamous: Festival of Blood.

Before:

After:

[toccata10]

JoJo health bars: problem solved here, on linux. Thanks.

[MarioSonic2987]

There's a black part of the screen in Just Cause 2 with Vulkan. With OpenGL, it looks like master in both backends.

RSX Capture (Vulkan PR)

[Xcedf]

Dante's Inferno regressed on the last commit
Before:

After:

Rsx capture
https://mega.nz/#!aQZzASrI!GafxJhcOsRq418xXq8bmahWSpwhe9OjzuIalYoHsv8o

[incognitoh]

@kd-11 rsx capture https://mega.nz/#!lNVgzSAY!s0l1djjm6KUM9e_GUlvuSDLR1vEyWXtZSJDoSkobBbU

[kd-11]

Should be fixed now. There was an existing regression in surface cache that got uncovered by this PR.

[incognitoh]

yup, Toy Story 3 vulkan works again. The original regression from last year remains but I'm sure that will be addressed someday by a different pr.

[MarioSonic2987]

Just Cause 2 regression is fixed:

[ghost]

@kd-11

Issue #6259 is semi-fixed with this, and by semi-fixed I mean that the effects work for a limited amount of time before they becomes absent (20-30 seconds of gameplay)

You are onto something but the issue is not entirely fixed.

Huge thanks though!

[Xcedf]

Dante's Inferno fixed

[incognitoh]

@kd-11

Found another regression. Total black screen now after this pr was merged. both ogl and vk are affected.

previous master:

master with this pr:

rsx cap: https://mega.nz/#!EQ8SlQwJ!JnWo8LueuSWRdOfdbAkd4txvfkhYhFeHaqhPanehEEc

[kd-11]

@incognitoh Should be fixed by #6436. I messed up the setup for a certain type of framebuffer that is rarely used.

[stachuATX7]

I need to admit that older version works good. 0.0.4/3