Integrate sub-pixel morphological antialiasing (SMAA) in the Vulkan renderer

[Calinou]

Related to #3401.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

Right now, we have MSAA and FXAA available in both 3.x and master. These algorithms serve their intended purposes, but each of them has their own limitations:

• MSAA looks good, but it is very expensive on master due to the added complexity of shaders. This is especially noticeable when using GIProbe or SDFGI. It also doesn't smooth out aliasing that originates from fragment shaders such as specular aliasing. (Alpha-tested surfaces are now well-handled thanks to alpha antialiasing and alpha-to-coverage support.)
• FXAA is cheap and looks decent on 1440p and higher. Unfortunately, the added blurriness makes it hard to use on 1080p and below. Post-processing sharpening algorithms make it possible to recover some of that lost sharpness, but they also add more aliasing on their own, so it's a tradeoff.

It's possible to use both MSAA and FXAA at the same time, but this has a significant cost and it won't improve the image's sharpness after it has been reduced by FXAA.

It's great that MSAA is still supported in master (thanks to the clustered forward renderer), but we have to admit that it's difficult to use in production with today's shader complexity. Therefore, this makes MSAA mostly suited to games that use few visual effects, which generally implies a stylized art direction.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Sub-pixel morphological antialiasing (SMAA) has been around since 2011 and is now a proven post-processing-based antialiasing algorithm. It's still used in a lot of AAA games.

Performance-wise, SMAA is more expensive than FXAA, but it does a good job at antialiasing despite introducing less blurriness. Either way, SMAA is likely to be less expensive than even 2× MSAA on the Vulkan renderer (despite providing better edge smoothing overall).

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

One downside of SMAA is that it's notoriously difficult to implement. Not only it's a multi-pass algorithm (unlike FXAA which is single-pass), there are also many quality settings to tinker with.

We can probably go with the recommended SMAA settings (luma-based edge detection), and only see if there are real benefits to exposing quality knobs in the project settings (such as changing the edge detection algorithm).

SMAA also offers an optional temporal component, but that comes with added complexity and motion trails commonly associated with temporal antialiasing algorithms. Therefore, I suggest we leave out the temporal antialiasing part of SMAA and focus on spatial-only SMAA (also called SMAA 1×). SMAA won't look as smooth without this temporal component, but SMAA still looks better than FXAA without it.

I've looked around a bit and couldn't figure out how to add a multi-pass post processing shader in Godot's GLSL core, so feel free to give it a try.

Question: Can this be implemented in 3.x?
I haven't seen any implementation of SMAA on OpenGL ES 3.0 (and even less OpenGL ES 2.0), so probably not.
Edit: It's possible to implement SMAA in WebGL 2.0 (and therefore OpenGL ES 3.0): https://github.com/dmnsgn/glsl-smaa

If this enhancement will not be used often, can it be worked around with a few lines of script?

Since real-time multi-pass shaders aren't possible in "userland" Godot yet, no.

Is there a reason why this should be core and not an add-on in the asset library?

See above.

[clayjohn]

For reference, the original SMAA code can be found here and it is MIT licenced.

[LiveTrower]

I found this: https://github.com/dmnsgn/glsl-smaa, i hope it helps.

[Calinou]

This porting work may also be useful as a reference: libretro/slang-shaders#189

[jntesteves]

This porting work may also be useful as a reference: libretro/slang-shaders#189

Hey, if you want to take any reference from my work there, please check the aftermath: libretro/slang-shaders#192

I can't agree with the docs on luma by default. I tested it extensively and color based edge detection always gave me better results, with less aliasing but not blurrier. In the end, I exposed every config knob, so I could more easily tweak settings at runtime. Tweaking it for your content is worthwhile, this AA is powerful when properly setup.

SMAA_PRESET_ULTRA + SMAAColorEdgeDetectionPS seemed like a good default to me. I increased AA strength a nod (SMAA_LOCAL_CONTRAST_ADAPTATION_FACTOR 2.5) where I knew there would be sharpening in a following pass, like SMAA+FSR.

My code is in the public domain, no restrictions apply. Feel free to copy anything.

[Calinou]

I've started work on integrating SMAA: https://github.com/Calinou/godot/tree/add-smaa-antialiasing

It's far from functional yet, but feel free to take a look nonetheless.

[Lasuch69]

Is there any update on this?

[Calinou]

Is there any update on this?

I haven't managed to get it working, as I have very little time to work on this.

Note that Godot 4 now offers temporal antialiasing, which you could combine with sharpening to counteract the added blurriness. This would still result in some amount of visible ghosting depending on the scene, but TAA handles specular and shader-induced aliasing much better than SMAA (which barely has any positive effect on this kind of aliasing).

TAA and FXAA sharpness can also be improved by adjusting mipmap LOD bias automatically.

I still think supporting SMAA is valuable for at least 2 scenarios:

• You want to maximize antialiasing quality (even in motion) by using both SMAA and TAA at the same time, at the cost of some blurriness. This is fairly expensive, but it can look very good in my experience, with nearly no noticeable aliasing on moving edges.
• You can't afford MSAA, you find FXAA too blurry but you still have enough GPU time available for SMAA. In this case, SMAA is a good compromise. (FXAA + sharpening can counteract the blurriness issue, but it will not resolve the issue where FXAA makes sharp edges appear rounded.)

[jntesteves]

I don't have any stake in Godot at the moment, so my opinion might not be too relevant, but I think working on the multi-pass shaders proposal mentioned in the description is a more worthwhile use of time. With that, anyone could easily do SMAA or whatever other post-process effects they want, not limited by what the engine offers.

Edit: Maybe these kinds of shaders should be on the Asset Library.

[Jamsers]

CMAA2 may be a great alternative to consider, it's been used to great effect in the recently released Counter Strike 2 and seems to work well as an MSAAesque cheap post process fallback.

The original repo published by Intel: https://github.com/GameTechDev/CMAA2

Intel's article on CMAA2
An article describing a hybrid CMAA implementation for mobile

A ReShade implementation: https://github.com/LordOfLunacy/Insane-Shaders/blob/master/Shaders/CMAA_2.fx

[jams3223]

How is this coming along? I was going to open a proposal until I saw this one. SMAA is the perfect middle ground in terms of performance, quality, and artifacts.

Q: Describe the project you are working on
A: 3D games that make use of realistic art styles and complex shapes

Q: Describe the problem or limitation you are having in your project
A: TAA tends to create ghosting artifacts while in motion, and while TAA is better at removing aliasing than SMAA, it still does a good job without having these issues; it's a good middle ground between TAA and FXAA. TAA is more resource-intensive and prone to artifacts, while FXAA is less effective.

Q: Describe the feature / enhancement and how it helps to overcome the problem or limitation
A: Add it to the list of already-available anti-aliasing algorithms.

Q: If this enhancement will not be used often, can it be worked around with a few lines of script?
A: No, it can't be used with a few lines of code; it needs to be implemented into the core engine.

Q: Is there a reason why this should be core and not an add-on in the asset library?
A: It sits right along with the other anti-aliasing methods that Godot ships with, so it helps extend the user's preferred choice and ensure a balanced level of quality.

[Calinou]

How is this coming along? I was going to open a proposal until I saw this one. SMAA is the perfect middle ground in terms of performance, quality, and artifacts.

SMAA is generally considered to be a stale approach by 2023 standards, so I wouldn't hold my hopes for it being added as a core feature (in the age of ever-improving TAA solutions). However, the rendering engine will eventually be made more flexible so that SMAA can be implemented by an add-on.

Modern games are often full of geometric and specular detail that SMAA can't do anything about. The only real use case I can see for SMAA nowadays is antialiasing an image when you have no access to motion vectors (such as running old games, antialiasing screenshots, etc). Similar concerns apply to other techniques like CMAA.

Being able to use TAA + SMAA at the same time isn't so important anymore, now that we have access to FSR2 at native resolution. This provides high-quality antialiasing with a sharper result compared to native TAA.

TAA is pending optimizations in the motion vector generation part, which will make it significantly less demanding.

[Jamsers]

Modern games are often full of geometric and specular detail that SMAA can't do anything about

That's fine - the users wanting to use SMAA aren't expecting it to match or even come close to the antialiasing capability of TAA - the point is to get the best possible antialiasing possible today that isn't TAA (due to ghosting and artifacting that will never be solved no matter how improved TAA gets) or MSAA (due to prohibitive performance cost)

SMAA is generally considered to be a stale approach by 2023 standards

Irrelevant. Progress and development on non-temporal post process antialiasing methods has practically halted, so unless the Godot team is planning to create a bespoke new technique, SMAA is still the best there is. (again, if you don't want TAA or MSAA)

If anything, FXAA is the stale, outdated, irrelevant technique - why is that still in core?

[jams3223]

I have tested the anti-aliasing algorithm with every kind of game art style, and I can say that it all depends on the art style or visual style. Godot will benefit by giving people choices in how they want to customize their game.

FXAA:
(+) Better with cel-shaded and anime art styles
(-) Lower Quality
(+) Better Performance

MSAA:
(+) Better with realistic art styles
(+) Higher Quality
(-) Lower Performance

TAA:
(+) Better with realistic art styles
(+) Higher Quality
(+) Balanced Performance
(-) Visual Artifact

SMAA:
(+) Better with realistic, anime and voxel art styles
(+) Balanced Quality
(+) Balanced Performance

[mrjustaguy]

One thing I see forgotten in here is the fact that SMAA can optionally be temporal itself, also known as SMAAT2x

[Jamsers]

I think temporal SMAA is actually stale, since updated TAA and TAAU (FSR 2) techniques have largely superseded any of the benefits temporal SMAA can offer today.

But the single pass SMAA stays relevant and best in class due to the lack of development on non-temporal post process antialiasing methods.

[mrjustaguy]

SMAAT2x still beats our TAA implementation in quality and likely even performance (though that is largely because of just how totally borked it is) and FSR2 is really expensive at native res, though I don't know the diff between SMAAT2x and FSR2

[Jamsers]

Absolutely, but as you say, that's a testament to how terrible Godot's TAA implementation is, rather than SMAA T2x being the way to go. And the current FSR 2 implementation has a disproportionately high performance cost - it's damn near 12% of GPU time on a relatively heavy and complex scene - so some optimization work is definitely due there.

[mrjustaguy]

FSR2 performance matches what AMD states for it's Performance Costs, and it's really designed for Upscaling, where you're rendering at a lower internal resolution (reducing the FSR2 cost) and Upscaling, improving the performance due to significantly less shading per frame

[Xynonners]

Hey all,

just dropping in to add that it is actually possible to implement / port reshade AA to gdshader.

here is my LXAA port:
https://gist.github.com/Xynonners/6e50a27d562829469281be45b039ec93
and here is HHAA (a weird mishmash bespoke algorithm I made for 3D):
https://gist.github.com/Xynonners/ddf30a5215a59db5690f6420621f0623

someone smarter than me could probably do a CMAA2 port if they wanted to.

pretty tired of waiting for AA from godot, considering that FXAA is stale, TAA is broken (motion smear).

[jams3223]

Hey all,

just dropping in to add that it is actually possible to implement / port reshade AA to gdshader.

here is my LXAA port: https://gist.github.com/Xynonners/6e50a27d562829469281be45b039ec93 and here is HHAA (a weird mishmash bespoke algorithm I made for 3D): https://gist.github.com/Xynonners/ddf30a5215a59db5690f6420621f0623

someone smarter than me could probably do a CMAA2 port if they wanted to.

pretty tired of waiting for AA from godot, considering that FXAA is stale, TAA is broken (motion smear).

I've never heard of HHAA, did you make it yourself?

[Xynonners]

Hey all,
just dropping in to add that it is actually possible to implement / port reshade AA to gdshader.
here is my LXAA port: https://gist.github.com/Xynonners/6e50a27d562829469281be45b039ec93 and here is HHAA (a weird mishmash bespoke algorithm I made for 3D): https://gist.github.com/Xynonners/ddf30a5215a59db5690f6420621f0623
someone smarter than me could probably do a CMAA2 port if they wanted to.
pretty tired of waiting for AA from godot, considering that FXAA is stale, TAA is broken (motion smear).

I've never heard of HHAA, did you make it yourself?

Yup, the idea was to be more aggressive than LXAA (which tends to be extremely conservative for the sake of anti-blur), but less blurry than the blur-fest that is FXAA. HHAA is a weird hybrid/combined antialiasing shader, but imo it works pretty well (I do believe it could still be more aggressive - though I don't think I'm smart enough to make something better than this).

Because of its completely custom nature though, it may have a sizeable performance impact, though I roughly tested it and it seems to be fine. Also may have bugs in implementation.

[Jamsers]

It produces pretty SMAAesque results to my eyes when I tested it on Bistro-Demo-Tweaked. Any chance you could port CeeJayDK's SMAA? 👉👈🥺

I tried it myself but I have zero shader knowledge and only got as far as porting the UI variables 😅

[Beanibirb]

Hey all,

just dropping in to add that it is actually possible to implement / port reshade AA to gdshader.

here is my LXAA port: https://gist.github.com/Xynonners/6e50a27d562829469281be45b039ec93 and here is HHAA (a weird mishmash bespoke algorithm I made for 3D): https://gist.github.com/Xynonners/ddf30a5215a59db5690f6420621f0623

someone smarter than me could probably do a CMAA2 port if they wanted to.

pretty tired of waiting for AA from godot, considering that FXAA is stale, TAA is broken (motion smear).

Inexperienced with setting up shaders for Godot here, how would one implement either of these in a 3D template?

[mrjustaguy]

See https://docs.godotengine.org/en/stable/tutorials/shaders/custom_postprocessing.html

[Beanibirb]

Hey all,

just dropping in to add that it is actually possible to implement / port reshade AA to gdshader.

here is my LXAA port: https://gist.github.com/Xynonners/6e50a27d562829469281be45b039ec93 and here is HHAA (a weird mishmash bespoke algorithm I made for 3D): https://gist.github.com/Xynonners/ddf30a5215a59db5690f6420621f0623

someone smarter than me could probably do a CMAA2 port if they wanted to.

pretty tired of waiting for AA from godot, considering that FXAA is stale, TAA is broken (motion smear).

Tested the two out, HHAA is good but you were right about it being very aggressive as it does not seem to like particle effects whatsoever.

This is using the default values for the hhaa-d shader.

[Xynonners]

Hey all,
just dropping in to add that it is actually possible to implement / port reshade AA to gdshader.
here is my LXAA port: https://gist.github.com/Xynonners/6e50a27d562829469281be45b039ec93 and here is HHAA (a weird mishmash bespoke algorithm I made for 3D): https://gist.github.com/Xynonners/ddf30a5215a59db5690f6420621f0623
someone smarter than me could probably do a CMAA2 port if they wanted to.
pretty tired of waiting for AA from godot, considering that FXAA is stale, TAA is broken (motion smear).

Tested the two out, HHAA is good but you were right about it being very aggressive as it does not seem to like particle effects whatsoever.

This is using the default values for the hhaa-d shader.

yeah, after making HHAA-D, I did a whole bunch more testing and ironed out the bugs. The edge detection in HHAA originally had a ton of false positives that were being dropped later on by the blending technique.

I made an improved version called AHAA (actually being used in a shipgame now), maybe that would be better?
https://gist.github.com/Xynonners/0edfc4ce5cab61f545041752e53b3f5b

also, another thing is that it is VERY important to set the render_priority of the ShaderMaterial to -128 (also causes the noise issue).

[Beanibirb]

Hey all,
just dropping in to add that it is actually possible to implement / port reshade AA to gdshader.
here is my LXAA port: https://gist.github.com/Xynonners/6e50a27d562829469281be45b039ec93 and here is HHAA (a weird mishmash bespoke algorithm I made for 3D): https://gist.github.com/Xynonners/ddf30a5215a59db5690f6420621f0623
someone smarter than me could probably do a CMAA2 port if they wanted to.
pretty tired of waiting for AA from godot, considering that FXAA is stale, TAA is broken (motion smear).

Tested the two out, HHAA is good but you were right about it being very aggressive as it does not seem to like particle effects whatsoever.

This is using the default values for the hhaa-d shader.

yeah, after making HHAA-D, I did a whole bunch more testing and ironed out the bugs. The edge detection in HHAA originally had a ton of false positives that were being dropped later on by the blending technique.

I made an improved version called AHAA (actually being used in a shipgame now), maybe that would be better? https://gist.github.com/Xynonners/0edfc4ce5cab61f545041752e53b3f5b

also, another thing is that it is VERY important to set the render_priority of the ShaderMaterial to -128 (also causes the noise issue).

The render_priority was the only thing that was messing me up, thank you so very much for this

[Jamsers]

The recent fantastic progress on implementing motion blur as a Compositor effect has got me thinking - how hard would it be to implement SMAA as a compositor effect? ᵃⁿᵈ ʷᵒᵘˡᵈ ᵃⁿʸᵒⁿᵉ ᵇᵉ ʷⁱˡˡⁱⁿᵍ ᵗᵒ ⁱᵐᵖˡᵉᵐᵉⁿᵗ ⁱᵗ ᵖˡᵉᵃˢᵉ 😁

[Calinou]

how hard would it be to implement SMAA as a compositor effect?

I think it's within the realm of possibility, as multipass effects can be implemented this way. In fact, it's probably easier than motion blur since SMAA only depends on the color buffer (at least in its higher quality modes, which are most relevant in 2024).