openfl_TextureCoordv not returning correct values in ShaderFilters

[MiniBobbo]

• Haxe version: 3.4.7
• Flixel version: 4.5.1 (first noticed on 4.4.1)
• OpenFL version: 8.5.0 (first noticed on 8.1.0)
• Lime version: 7.1.1 (first noticed on 6.3.1)
• Affected targets: HTML5, CPP, maybe others.

---

Code snippet reproducing the issue:
Shader:

class TestShader extends FlxShader
{
	@:glFragmentSource('
		#pragma header
			
		void main()
		{
			if (openfl_TextureCoordv.y > 0.5)
				gl_FragColor = vec4(1.0,0.0,0.0,1.0);
			else
				gl_FragColor =vec4(0.0,0.0,1.0,1.0);
		}'
	)
	public function new() 
	{
		super();
	}	
}

Playstate:

import flixel.FlxG;
import flixel.FlxSprite;
import flixel.FlxState;
import openfl.filters.ShaderFilter;

class PlayState extends FlxState
{
	override public function create():Void
	{
		super.create();
		var filter = new ShaderFilter(new TestShader());
		FlxG.game.setFilters([filter]);
	}

	override public function update(elapsed:Float):Void
	{
		super.update(elapsed);
	}
}

---

Observed behavior:
When calling openfl_TextureCoordv.x the left side of the screen is 0, but the right side of the screen is not 1. It resolves to something like .6 instead. openfl_TextureCoordv.y also has problems. The bottom half of the window should be red, but instead openfl_TextureCoordv.y > 0.5 is true about 10% of the way down.

Also, if you wait a few seconds or click on the screen, it will change by itself:

This is closer to the correct location, but still wrong.

If the shader is applied directly to a sprite it behaves correctly.

I've tested in Chrome, Internet Explorer and Edge on the HTML5 target, and the Windows CPP target and they all have the same behavior. You can see a sample project here: http://codingadventure.net/html5/WeirdShader/

Expected behavior:
The bottom half of the screen should be red while the top half should be blue. It should not change.

[Gama11]

Did you get around to testing this with pure OpenFL? Shaders are more of an OpenFL than a Flixel feature, so Flixel doesn't really control the value of openfl_TextureCoordv.

[MiniBobbo]

I did test in OpenFL and it worked, but I'm not sure my test was good. I'm having trouble finding the equivalent of the HaxeFlixel game and camera object in OpenFL. I think my OpenFL test was the equivalent of a sprite shader in Haxeflixel which is working like expected.

I was going to track back the HaxeFlixel objects back and try to figure out what the OpenFL equivalent is.

[Gama11]

Maybe @jgranick has an idea what this could be.

[MiniBobbo]

I did some more testing and investigation. I found that this problem is true of anything in HaxeFlixel where a sub area of a larger BitmapData is displayed.

For instance, here's my sample image.

Same basic shader, but looking at the x instead of the y.

if (openfl_TextureCoordv.x > 0.5)
	gl_FragColor = vec4(1.0,0.0,0.0,1.0);
else
	gl_FragColor =vec4(0.0,0.0,1.0,1.0);

s = new FlxSprite();
s.loadGraphic('assets/images/multitexture.png', true);
s.animation.frameIndex = 0;
s.shader = new TestShader();

Frame 0 turns all blue and Frame 1 is all red.

So I'm expecting the openfl_TextureCoordv.x to look like this since I'm only fetching a subset of the whole BitmapData.

What I actually get is this:

I think this is what is happening with the camera and game filters also. I think that the camera and game BitmapData objects are larger than what is rendered, but openfl_TextureCoordv looks at the entire BitmapData instead of just the subset that should be displayed.

This affects a lot of other things. Animated sprites, Texture Atlases, TileMaps, etc. Shaders don't work correctly on them all because of this issue.

I'm not quite sure how to fix this. I was looking around deep in the HaxeFlixel/OpenFL display code but I was getting lost. I think that this problem needs to be solved at the HaxeFlixel level because, from my understanding, OpenFL has no concept of cameras or frames or subtextures.

Maybe HaxeFlixel should calculate and pass another uniform with the frame specific uv coordinates? Something like flixel_TextureCoordv so both are available to the shader?

[starry-abyss]

This affects a lot of other things. Animated sprites, Texture Atlases, TileMaps, etc.

I think this is normal behaviour in OpenGL.

Maybe HaxeFlixel should calculate and pass another uniform with the frame specific uv coordinates?

If you need to know in-frame coordinates, you can pass your own vec4 uniform (hmm... or rather an attribute is wanted?) with rect of the frame currently shown. Maybe FlxSprite.frame.uv would help with that?

[MiniBobbo]

I think this is normal behaviour in OpenGL.

@starry-abyss I think you are correct :) I'm still learning the whole "programmable graphics pipeline" thing. I think the behavior makes a ton of sense in a 3D world, so you get some unexpected behavior when using it in 2D.

If you need to know in-frame coordinates, you can pass your own vec4 uniform (hmm... or rather an attribute is wanted?) with rect of the frame currently shown. Maybe FlxSprite.frame.uv would help with that?

That's what I ended up doing to work as a workaround. I added

		var rect = s.frame.uv;
		shader.subTexturePosition.value = [rect.x, rect.y, rect.width, rect.height];

to my update code and writing a function in my shader

	uniform vec4 subTexturePosition;
	
	vec2 subtextureCoord(vec2 coord) 
	{
		float width = subTexturePosition.z - subTexturePosition.x;
		float height = subTexturePosition.a - subTexturePosition.y;
		float newX = (coord.x - subTexturePosition.x) / width;
		float newY = (coord.y - subTexturePosition.y) / height;
		
		return vec2(newX, newY);
	}
	
	
	void main()
	{
		
	vec2 subcoord = subtextureCoord(openfl_TextureCoordv);
	if (subcoord.y > 0.5)
			gl_FragColor = vec4(1.0,0.0,0.0,1.0);
		else
			gl_FragColor =vec4(0.0,0.0,1.0,1.0);
	}

and it is working as expected. I don't think this will work when I'm trying to use a texture atlas with any rotated sprites though. That could get complicated, but Flixel obviously is handling this currently because the sprites animate correctly.

I would think that if I have a 2D animated sprite it would be nice to have the shader values be only for the frame I am currently displaying, but I understand why it is not.

I'm going to close this issue since it isn't an error (it just isn't doing what I expect). There is a definite error with the Camera filter when scrolling but I captured that in a different issue (#2194)