FlxScaleMode w/ customizable Shader ($150 Bounty)

[larsiusprime]

I have a pretty specific use case that I'm having trouble implementing myself, and I'd like to put up for bounty. What I'm specifically looking for is an easy way to upscale the entire final, actually rendered set of pixels to an arbitrarily defined size, in conjuction with a user-defined shader.

Just using existing FlxCamera/FlxGame scaling is not ideal, because it doesn't actually upscale the actual, final rendered pixels, instead it increases the size of the entire drawTiles rendering area and applies scaling operations to each object during the drawing process itself. (At least, that's what it seems to be doing to me)

Example:
Say my world size is 100x100 and my scale factor is (2.0 , 2.0). Currently, flixel renders each object at object.(x , y) * 2.0 with a scale factor of (2.0 , 2.0). In a loose mathematical sense this is close to what I want, but the final rendered result is different. What I want instead is to just render those 100x100 pixels exactly as-is, and post-process them with a shader that blows them up to 200x200 pixels.

Right now my only concern is to get this working on OpenFL+Next+CPP+drawTiles, but if it can also be generalized to HTML5+WebGL target that's cool too.

Requirements:

• First render the game 1:1 at world size with no global scaling
• Take the final outputted pixels and upscale them by user-defined scale (x/y), and display transformed pixels
• Allow the user to pass in a shader of type Shader that does the upscaling work
  • probably define a subclass or interface that enforces the right sort of shader for this (e.g. has a user-supplied scaleX/scaleY, or whatever). Maybe use enums even to guarantee functionality.
  • Extra credit: expose user-definable "strength" parameter that lets you blend out the effect of the shader from 0.0-1.0
• Do proper logic under the hood to synchronize mouse position, etc to match upscaled pixels on screen
• Easy to use, something like e.g:

FlxG.scaleMode = new ShaderScaleMode(ShaderScaleEnum.BILINEAR);
FlxG.scaleMode.scale.x = scaleX;
FlxG.scaleMode.scale.y = scaleY;

Not only would this solve my personal problems, it would open up a lot of powerful features for really precise pixel-perfect upscaling operations for other developers.

(If I'm totally misunderstanding things and this is already possible / implemented, do still let me know as I'll also pay a portion of the bounty just for taking 10 minutes to show me how to achieve all this trivially with existing methods.)

The obvious starting point here is just to implement this with a dumb nearest-neighbor shader, and once that works, test that it also works with Bilinear, Bicubic, SuperSal, Scanlines, etc.

[JoeCreates]

I would find this useful, too. I'm not sure about making it a scale mode, though, as this functionality would be useful in conjunction with existing scale modes. For example, you may want the game to keep a fixed ratio?

This might be useful: http://community.openfl.org/t/render-to-texture/1155/6

[Beeblerox]

@JoeCreates thanks for the link, seems very useful!

[larsiusprime]

@JoeCreates -- yeah, I'm not super duper familiar with exactly what Flixel primitive should be used here, whether it's a scale mode or a camera or whatever. I just want the experience of:

FlxMakeItScaleTheWayIWantIt = new ShaderScalingThingy(scaleX,scaleY,SomeEnum.THIS_METHOD);

So yeah, keeping the game at a fixed 1:1 ratio underneath and then just add this on top would suit my needs perfectly so long as the mouse coordinates could be lined up.

[larsiusprime]

Heads up: https://twitter.com/_Sean_Whiteman_/status/722892870760304641 claims to be working on it. I'll post updates.

[Seanw265]

Hey guys! Progress is being made! Right now it's close to the original spec where it acts as a ScaleMode. Works perfectly visually right now. I just need to abstract it a little bit.

Unfortunately I couldn't find a way around making a few changes to other existing classes but nothing drastic.

I'll post more in the morning!

[larsiusprime]

@Seanw265 : excellent! And it takes care of updating the mouse coordinates, too? So that button click areas match up with what the user sees on the screen?

[Seanw265]

@larsiusprime Yup! Handles the mouse coordinates, button clicks and all that. I'd say it's about finished now. How should I go about uploading all the changes?

[larsiusprime]

Okay so what I would do is fork flixel if you haven't already, and then create a feature branch, name it "fancyscaling" or whatever, and push that branch to your fork. Then make a pull request to flixel, and post some sample code here about how to use / test it.

[Seanw265]

Ok I can do that in about half an hour.

[larsiusprime]

EXCELLENT!

[MSGhero]

I need to learn shaders so I can do stuff like this. Luckily, I updated haxe to git, so I can compile to cpp once again (after 11 months).

[larsiusprime]

@Seanw265: I take it this is the code?

https://github.com/HaxeFlixel/flixel/pull/1823/files

Looking forward to testing this :)

[Seanw265]

Ok just submitted the pull. Kind of new to git in general but I think I did it correctly.
#1823

Usage:

var scaleMode = new ShaderScaleMode(ShaderScaleMode.ShaderScaleEnum.BILINEAR);
// Alternatively, right now we also have a shader for nearest neighbor sampling, ShaderScaleEnum.NEAREST
FlxG.scaleMode = scaleMode;
scaleMode.setScale(2,2);
scaleMode.activate();


// OR, you can pass in a shader object
var scaleMode = new ShaderScaleMode(new Shader(...));

Currently when you want to use this scale mode you should set that before any other PostProcess effects.

Also, the window size should be larger than FlxGame size unless your scale is meant to be 1.
For example, if your game runs at 200x200 but you want to scale it up to 400x400:

<!--In project.xml-->
<window width="400" height="400" antialiasing="0" />

// In main.hx
new FlxGame(200, 200, MenuState);

Currently you also must calculate the scale of your game manually (which makes sense).

Let me know if there are any other questions!

[larsiusprime]

Thanks! Very excited to try it.

First question:
I see you're using a post-process shader to implement this, IIRC that's only used in legacy. Are you compiling with the "-Dnext" flag? (Flixel defaults to openfl-legacy unless you specify the "-Dnext" flag).

[Seanw265]

@larsiusprime That may have been an oversight. I'll see if I can find a way to fix that.
In the mean time it works with legacy pretty well.

[larsiusprime]

Not a problem, hopefully you've cracked the meat of it with your current work and it will be pure gravy to have this available on legacy too!

[Seanw265]

Working on a possible solution. I'll post again when I know more.

[larsiusprime]

Great! Can't wait :)

[Seanw265]

Ok I finally got it working on Openfl Next but it's just way too late/early for me to be fiddling with Git right now. I'll submit a pull request in the morning!

[larsiusprime]

No prob!

[Seanw265]

Ok here it is:
#1826

Usage is the same as before except you pass in an openfl.display.Shader object or one of the enums instead of a flixel.effects.postProcess.Shader object.

The enums still work the same.

[larsiusprime]

Evaluating now!

[larsiusprime]

@Seanw265 :

Hey there! Trying to use this now, not sure I'm doing it right. I'm trying to add the example of the shader to the https://github.com/HaxeFlixel/flixel-demos/tree/dev/Features/ScaleModes flixel demo.

Here's my modified code:

package;

import flixel.FlxG;
import flixel.FlxSprite;
import flixel.FlxState;
import flixel.math.FlxMath;
import flixel.math.FlxRandom;
import flixel.system.scaleModes.FillScaleMode;
import flixel.system.scaleModes.FixedScaleMode;
import flixel.system.scaleModes.RatioScaleMode;
import flixel.system.scaleModes.RelativeScaleMode;
import flixel.system.scaleModes.ShaderScaleMode;
import flixel.text.FlxText;
import flixel.util.FlxColor;

class PlayState extends FlxState
{
    private var currentPolicy:FlxText;
    private var scaleModes:Array<ScaleMode> = [RATIO_DEFAULT, RATIO_FILL_SCREEN, FIXED, RELATIVE, FILL, SHADER];
    private var scaleModeIndex:Int = 0;

    override public function create():Void
    {
        add(new FlxSprite(0, 0, "assets/bg.png"));

        for (i in 0...20)
        {
            add(new Ship(FlxG.random.int(50, 100), FlxG.random.int(0, 360)));
        }

        currentPolicy = new FlxText(0, 10, FlxG.width, ScaleMode.RATIO_DEFAULT);
        currentPolicy.alignment = CENTER;
        currentPolicy.size = 16;
        add(currentPolicy);

        var info:FlxText = new FlxText(0, FlxG.height - 40, FlxG.width, "Press space or click to change the scale mode");
        info.setFormat(null, 14, FlxColor.WHITE, CENTER);
        info.alpha = 0.75;
        add(info);
    }

    override public function update(elapsed:Float):Void
    {
        if (FlxG.keys.justPressed.SPACE || FlxG.mouse.justPressed)
        {
            scaleModeIndex = FlxMath.wrap(scaleModeIndex + 1, 0, scaleModes.length - 1);
            setScaleMode(scaleModes[scaleModeIndex]);
        }

        super.update(elapsed);
    }

    private function setScaleMode(scaleMode:ScaleMode)
    {
        currentPolicy.text = scaleMode;

        FlxG.scaleMode = switch (scaleMode)
        {
            case ScaleMode.RATIO_DEFAULT:
                new RatioScaleMode();

            case ScaleMode.RATIO_FILL_SCREEN:
                new RatioScaleMode(true);

            case ScaleMode.FIXED:
                new FixedScaleMode();

            case ScaleMode.RELATIVE:
                new RelativeScaleMode(0.75, 0.75);

            case ScaleMode.FILL:
                new FillScaleMode();

            case ScaleMode.SHADER:
                new ShaderScaleMode(ShaderScaleEnum.BILINEAR, 2, 2);
        }
    }
}

@:enum
abstract ScaleMode(String) to String
{
    var RATIO_DEFAULT = "ratio";
    var RATIO_FILL_SCREEN = "ratio (screenfill)";
    var FIXED = "fixed";
    var RELATIVE = "relative 75%";
    var FILL = "fill";
    var SHADER = "shader";
}

Using either BILINEAR or NEAREST enum constant with 2.0,2.0 results in a 1:1 scale (small postage stamp @ original size) game in both legacy and next for me.

Maybe I did something wrong?

[Seanw265]

@larsiusprime You have to call scalemode.activate(); in order to get it to run. It also has a scalemode.deactivate(); method to turn it off.

Let me know if you have any other questions!

[larsiusprime]

Figured it was something simple like that. Thanks!

[larsiusprime]

Okay got it working with the flixel scalemodes demo. Really well done, exactly what I wanted.

Last test is to try it out in Defender's Quest and see how that works :)

[larsiusprime]

Quick heads up --- if you run this test code, you can see that if you cycle quickly through the scale modes, your frame rate will drop -- this doesn't happen in the old version of the demo. Wonder if there's a memory leak in the ShaderScaleMode somewhere, or if deactivate() needs more cleanup logic or something. The functionality itself is marvelous though!

package;

import flixel.FlxG;
import flixel.FlxSprite;
import flixel.FlxState;
import flixel.math.FlxMath;
import flixel.math.FlxRandom;
import flixel.system.scaleModes.FillScaleMode;
import flixel.system.scaleModes.FixedScaleMode;
import flixel.system.scaleModes.RatioScaleMode;
import flixel.system.scaleModes.RelativeScaleMode;
import flixel.system.scaleModes.ShaderScaleMode;
import flixel.text.FlxText;
import flixel.util.FlxColor;

class PlayState extends FlxState
{
    private var currentPolicy:FlxText;
    private var scaleModes:Array<ScaleMode> = [RATIO_DEFAULT, RATIO_FILL_SCREEN, FIXED, RELATIVE, FILL, SHADER_NEAREST, SHADER_BILINEAR];
    private var scaleModeIndex:Int = 0;

    override public function create():Void
    {
        add(new FlxSprite(0, 0, "assets/bg.png"));

        for (i in 0...20)
        {
            add(new Ship(FlxG.random.int(50, 100), FlxG.random.int(0, 360)));
        }

        currentPolicy = new FlxText(0, 10, FlxG.width, ScaleMode.RATIO_DEFAULT);
        currentPolicy.alignment = CENTER;
        currentPolicy.size = 16;
        add(currentPolicy);

        var info:FlxText = new FlxText(0, FlxG.height - 40, FlxG.width, "Press space or click to change the scale mode");
        info.setFormat(null, 14, FlxColor.WHITE, CENTER);
        info.alpha = 0.75;
        add(info);
    }

    override public function update(elapsed:Float):Void
    {
        if (FlxG.keys.justPressed.SPACE || FlxG.mouse.justPressed)
        {
            scaleModeIndex = FlxMath.wrap(scaleModeIndex + 1, 0, scaleModes.length - 1);
            setScaleMode(scaleModes[scaleModeIndex]);
        }

        super.update(elapsed);
    }

    private function setScaleMode(scaleMode:ScaleMode)
    {
        currentPolicy.text = scaleMode;

        if (Std.is(FlxG.scaleMode, ShaderScaleMode))
        {
            cast(FlxG.scaleMode, ShaderScaleMode).deactivate();
        }

        FlxG.scaleMode = switch (scaleMode)
        {
            case ScaleMode.RATIO_DEFAULT:
                new RatioScaleMode();

            case ScaleMode.RATIO_FILL_SCREEN:
                new RatioScaleMode(true);

            case ScaleMode.FIXED:
                new FixedScaleMode();

            case ScaleMode.RELATIVE:
                new RelativeScaleMode(0.75, 0.75);

            case ScaleMode.FILL:
                new FillScaleMode();

            case ScaleMode.SHADER_NEAREST:
                new ShaderScaleMode(ShaderScaleEnum.NEAREST, 2.0, 2.0);

            case ScaleMode.SHADER_BILINEAR:
                new ShaderScaleMode(ShaderScaleEnum.BILINEAR, 2.0, 2.0);
        }

        if (Std.is(FlxG.scaleMode, ShaderScaleMode))
        {
            cast(FlxG.scaleMode, ShaderScaleMode).activate();
        }

    }
}

@:enum
abstract ScaleMode(String) to String
{
    var RATIO_DEFAULT = "ratio";
    var RATIO_FILL_SCREEN = "ratio (screenfill)";
    var FIXED = "fixed";
    var RELATIVE = "relative 75%";
    var FILL = "fill";
    var SHADER_NEAREST = "shader (nearest)";
    var SHADER_BILINEAR = "shader (bilinear)";
}

[larsiusprime]

Okay just tested it in DQ!

So that's what happens when I set the game's native vertical resolution to 450, and then load up a 1600x900 window. The red line was inserted in photoshop to point out the glitch -- the screen is being vertically cut off.

The good news is that everything looks exactly as I expect and the mouse position seems to work flawlessly! My only issue is whatever's going on with the verticality here. I didn't notice this issue in the FlxScaleMode demo so it could be a side effect of something stupid I'm doing in DQDX. I'll let you know as I investigate more.

Happy to pay the bounty out right now as long as you're willing to help me work out the remaining issues:

• Memory leak?
• Vertical cutoff glitch

[larsiusprime]

Messing around a little, it looks like the problem is somewhere in the offset functions that you've overriden. If I un-override those and fall back to the default values I get a different cutoff, which also happens to be wrong, so maybe it's all about playing with that logic. I'm going to see if I can reproduce this error outside of DQDX so it's easier for you to fix. I might start by just setting up an 800x450 game canvas in FlxScaleModes demo with a 1600x900 window.

[MSGhero]

The scale modes never get destroyed or anything, and the game still holds a reference to each one due to FlxG.game.setFilters(filters); and FlxG.signals.postDraw.add(postDraw); in ShaderScaleMode.

But does your framerate drop when you cycle through the non-shader scale modes?

[larsiusprime]

The framerate does not drop when I cycle through the non-shader scale modes, only if I use the new ones. So is the problem just improper cleanup in my handling of the Shader scale modes?

[larsiusprime]

@MSGhero so perhaps the deactivate() function needs this?

    public function deactivate():Void
    {
        filters.remove(this.filter);
        FlxG.game.setFilters([]);
    }

[Seanw265]

@larsiusprime The issue with the vertical cutoff is why it took so long to get it working on Next. I thought I had figured it out but evidently my solution is not applicable for all cases. The problem is that when filters are applied to the FlxGame object, they are applied to the ENTIRE FlxGame openfl Sprite object, which usually happens to be larger than the screen space. This means that when the shader (filter) is applied, the vertex(?) coordinates (0,0) (0,1) (1,0) (1,1) do not map to the corners of the screen. Often they map to areas outside of it. I will revisit the issue when I have some time tonight and I'll look into solving it with the offset functions like you said.

Also, just as a quick question, do you add any openfl sprites to the FlxGame Sprite? For example with addChildBelowMouse()?

All that being said, it may be best to open a new issue to look into having filters applied to the FlxGame object only work in the screen space. I can't see why you'd want to waste processing time to, for example, apply a blur to areas of the screen which aren't visible.

As for the memory leak, I'll look into that. The problem with setting the FlxGame filters to an empty array is that it will remove whatever other filters the user has applied to the FlxGame. The reason I made the filters field of ShaderScaleMode public is so that other filters can still be applied. I'll try and look for a solution tonight but rapid hot-swapping between ScaleModes is a pretty specific use case that most likely will not be needed by the average HaxeFlixel user.

[larsiusprime]

@Seanw265 : I don't believe I add anything, but one thing to note with Flixel games is that the debugger overlay is implemented as a regular ol' OpenFL sprite pasted onto the screen.

I can try real quick in release mode (sans flixel debugger) and see if that helps.

[larsiusprime]

@Seanw265 : update: still there in release mode, so it's likely not the HaxeFlixel debugger. But it is good to know it's working off the FlxGame sprite object, because it's very likely DQ is doing some special case logic to resize the window and the game object just before applying this filter, that you're very unlikely to reproduce anywhere else.

[larsiusprime]

@Seanw265 : update!

So I traced out everything that's attached to FlxGame as an openfl displayObject. Turns out quite a bit!

private function dumpDisplayList()
{
    trace("FlxGame x,y,width,height = " + FlxG.game.x + "," + FlxG.game.y + "," + FlxG.game.width + "," + FlxG.game.height);
    trace("...children : " + FlxG.game.numChildren);
    for (i in 0...FlxG.game.numChildren){
        var child = FlxG.game.getChildAt(i);
        trace("...child(" + i + ") = " + child + " x,y,width,height = " + child.x + "," + child.y + "," + child.width + "," + child.height);
    }
}

State_Title.hx:478: FlxGame x,y,width,height = 0,0,1431,813
State_Title.hx:479: ...children : 6
State_Title.hx:482: ...child(0) = [object Sprite] x,y,width,height = 640,360,1282,722
State_Title.hx:482: ...child(1) = [object Sprite] x,y,width,height = 208,501,24,32
State_Title.hx:482: ...child(2) = [object Sprite] x,y,width,height = 0,0,0,0
State_Title.hx:482: ...child(3) = [object FlxDebugger] x,y,width,height = -0,-0,1430,719
State_Title.hx:482: ...child(4) = [object FlxSoundTray] x,y,width,height = 560,-92,160,92
State_Title.hx:482: ...child(5) = [object FlxFocusLostScreen] x,y,width,height = -0,-0,1280,720

[larsiusprime]

So probably the biggest culprit for messing things up here is the sound tray, which is located off the top of the screen (it's designed to slide down). I'm going to compile with this in my project:

<haxedef name="FLX_NO_SOUND_TRAY"/>

And see if that sorts it out.

[larsiusprime]

Well, that didn't solve the visual glitch, but here's the output anyway. Killing the sound tray, and running in release mode -- which kills the debugger -- I get this:

FlxGame x,y,width,height = 0,0,1281,721
...children : 4
...child(0) = [object Sprite] x,y,width,height = 400,225,802,452
...child(1) = [object Sprite] x,y,width,height = 544.375,97.5,24,32
...child(2) = [object Sprite] x,y,width,height = 0,0,0,0
...child(3) = [object FlxFocusLostScreen] x,y,width,height = -0,-0,1280,720

(using sys.println here instead of trace, of course)

So now my FlxGame is exactly the size it's supposed to be, + 1 pixel for some reason.

child(0) looks like the FlxGame (I'm doing a test where I cap the max native resolution at 450 vertical and upscale from there), I have no idea what child(1) is supposed to be, likewise child(2), but child(3) tells us outright it's the focus lost screen.

[Seanw265]

@larsiusprime I just pushed a commit that should fix your issue with the strange offset. Killing the sound tray should not be necessary. Let me know if it works for you!

[Seanw265]

@larsiusprime Any updates? Is it working for you?

[larsiusprime]

Oh shoot, I didn't see the update. Gonna test it right now!

[larsiusprime]

It looks like it works!

[larsiusprime]

A few tweaks I made:

You need this at the top of Nearest.hx after the package:

#if sys

And at the bottom:

#end

Same deal for Bilinear.hx. Without those changes, you break flash target compilation. (Maybe it should be if sys || webgl ? Maybe @Gama11 should weigh in here.

[larsiusprime]

Anyways, I'm considering this bounty fulfilled. Well done, sir! I'll send the money now if you like.

[Seanw265]

@larsiusprime Great! Glad I could be of service! I sent you an email a week or two ago with my preferred method of payment.

[larsiusprime]

@Seanw265 Can you bump that email thread? Searching by your user name or your full name doesn't seem to bring it up.

[Seanw265]

Bounty received! Thanks @larsiusprime !