GS: Handle higher TH/TW for STQ calculation #7104
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of Changes
Modify the clamps/FixedTEX0 calculations to allow TEX0 to set values higher than 10 (max texture size on the GS)
Rationale behind Changes
When using STQ, the calculations are done with the TH and TW sizes in mind that were written to the TEX0 register, clamping them to 10 causes the calculation result to be incorrect.
Suggested Testing Steps
test games (Especially those where textures looked accidentally blown up), make sure nothing is busted. Affects both SW and HW.
TODO: Need to do a hardware test to confirm the clamp behaviour, I suspect it will clamp/wrap at 1024, but we will need to do a hardware test to confirm this.
Fixes the text size in
Simple 2000 Series Vol. 105 - The Maid Fuku to Kikanjuu
,there is a further issue where the 4bit pixels are in the wrong order, however the PS2 also shows this error, so it could be another GS problem with an upload that only happens once, or it's a core problem.Before:
After: