rsx: Fix unknown vertex base types

[elad335]

• Clamp vertex type field into 3-bit instead of 4-bit value, vertex type 0 is the same as ub256.

Testcase : https://github.com/elad335/myps3tests/tree/master/rsx_tests/vertex%20base%20type%200

Testing conditions:

• all vertex base types are being ORed with 0x8, this doesnt affect the results on realhw but crashes on rpcs3.

• Vertex base type 0 is used for POSITION array, values of ub256 are being passed and normalized manually into [-1, +1] by doing new_value = ((value) - 128) / 128 in the vertex shader.
  Rendering works and produces normal image.

Fixes #5582
Fixes #2408

[xYaroslavGTx]

[elad335]

To make things clear this is not a graphical fix, this only affects games which used to throw the exception mentioned in the title.

[digitaldude555]

The error is gone, it's now ingame.

The error is gone from Absolute Supercars and Ferrari Race Experience., they reach menu now.

[digitaldude555]

Correction they go ingame.

[xYaroslavGTx]

To make things clear this is not a graphical fix, this only affects games which used to throw the exception mentioned in the title.

Yep, but this PR added 0.4 - 0.6 FPS to this game.