Use GPU to compute and edit maps

[Zylann]

Map = texture used for terrain data

Calculating heightmap normals, painting or sculpting the terrain is basically down to processing images, in ways that extend a bit further than using the classic drawing functions.
This is utterly slow in GDScript (but relatively usable on small maps and currently works fine with undo/redo etc).

Using C++ would improve this, and is a reason why issue #14 exists, however it can't get around the fact it needs to reupload changes to the graphics card, which takes time and memory since there has to be a copy of the images in RAM.

An alternative to using C++ would be to use render target Viewports. The GPU is many orders of magnitude faster at this than C++, and it would dispense the need for the plugin to have a GDNative library to maintain and ship with.

However there are limitations in the engine that prevent this to be used more widely in this plugin:

• Blending modes treating alpha channel like regular channel (A way to erase pixels or overwrite alpha channel of pixels using the draw_line method() and a new blend mode godotengine/godot#10255)
• A way to upload AND download partial textures sub-rects, which would be useful for undo/redo and bandwidth optimization (Need to be able to upate a texture partially godotengine/godot#9571). Current workaround would be to use multiple viewports rendering sub-regions.
• A way to render on viewports having a custom amount of channel and bit depth, which would optimize bandwidth when sculpting heightmaps that need only 1 channel out of 4
• 32-bit support. That would be really nice to not constantly get artifacts due to progressive precision loss of 16-bit.
• Have a way to disregard sRGB conversion, since apparently custom blending modes would only be available in 2D, while the output textures are used in 3D
• Have a way to do broader kinds of operations that do not solely involve individual pixels, such as computing the min and max of an area of pixels. This is required for the culling system.
• Nice to have: a way to render a viewport immediately, forced to wait for 1 update loop, which makes computing slower and forces an entirely different code design (note: Unity has this feature).

[BastiaanOlij]

For future reference, my terrain editor takes this approach so feel free to use whatever you can from this:
https://github.com/BastiaanOlij/godot-terrain-edit

I agree with many points Zylann raises here.

The first one (blend modes) was added a week or two ago (render_mode blend_disabled).

The second one, you can actually "upload" them with a trick, just place a TextureRect in your viewport that you size to the portion you want to update and set a texture to it. With the above blend_disabled and a nifty shader you could even make it just update one channel. In a way that is exactly how I end up painting terrain. Download is a problem, I might see if I can come up with something and submit a PR.

The 3rd is a pain. I've accepted it, it just uses more memory in the editor though careful choosing of how you use your channels can mitigate it. But yeah, would love to be able to specify I want a 32bit, 1 channel texture for my heightmap instead of mucking about with packing the height into multiple 8bit channels.

The 4th one (potential issues with color space conversion) only applies to 3D viewports, there is no color conversion on 2D viewports. There can be a conversion on loading/unloading textures. So far it's worked fine for me. For 3D viewports I've added a "keep_3d_linear" setting on the viewport that disables the color conversion provided you have not set anything in the environments tone mapper.

The 5th one, this is difficult in the architecture of Godot especially if you turn multithreading on and the viewport look runs in a separate thread. I haven't found it difficult but what I really miss is not being able to query whether the viewport was updated so I can react on it.
.

[Zylann]

I found a way to sculpt with a Viewport at small cost (i.e without having to render the entire terrain each time), so I'm now working on rewriting the brush system using the GPU and shaders.

How it works

• The terrain keeps rendering as it does now, and edited textures are turned into ImageTexture like before.
• When painting, a Viewport the size of the brush is created. Under this viewport, a single Sprite is rendered with the heightmap as texture (or another map if painting splats, colors or grass).
• Depending on the type of brush, this Sprite is given a ShaderMaterial. Brush operations are implemented in this shader, which uses custom blending with render_mode blend_disabled;. The shader knows the source pixels (TEXTURE), brush shape and parameters with various uniforms, and outputs the final color.
• The frame after, we get the result using Viewport.get_texture().get_data(), which is a tiny chunk of the edited texture.
• We blit that chunk back onto the final texture using VisualServer.set_data_partial(heightmap.get_rid(), ...)
• Extra calculations needed by the heightmap such as normalmap baking and chunk vertical bounds is deferred over frames or on mouse button release, to keep a smooth experience
• And we're done!

Doing things this way allow undo/redo to keep working as it does, and does not require large renders.
This is significantly faster than GDScript, and might be faster than GDNative too. This is considering GDNative still has to deal with API overhead anyways, but also we could now implement subpixel precision much more cheaply, as well as more customization such as slope detection when painting textures, or even noise and erosion brushes.

Minor issues with texture format

Ideally, the viewport's framebuffer should match the format of the texture we paint on, but Godot 3.x can only offer us either RGBA8 or RGBAH. So there is a bunch of unused bandwidth when calling get_data and conversion going on the CPU still, but things still appear to be way better than GDScript on my Ryzen 5.

Heightmaps require 16-bit or 32-bit rendering with a single channel, but only 16-bit and 4 channels are available. Besides, only the 3D mode allows HDR so we also have to waste a bit of memory for unused things like the depth buffer. Again, doesn't cause much trouble beyond memory usage in the end.

Problem with color painting

I'm having a problem with the colormap though:
When I try to paint on it using this system, the result keeps getting darker until it becomes black. This makes no sense because all the color brush does is to interpolate from a color to another, non-black color, without going below 0 or above 1, and not touching alpha.

Turns out, on all terrain shaders, the colormap was hinted with hint_color. So because of this, Godot apparently decides to convert the texture to sRGB every time we paint a stroke. This notoriously makes the texture darker in some scenarios. Here it snowballed making color painting unusable.
I suspect this is because in the brush viewport, the colormap renders darker (since we use a Sprite with no particular hints on TEXTURE). So what we just painted is fed back here in darker form. This editor must work on the source. Trying to "undo" this darkening brings further issues related to 8-bit precision so it.
I believe Godot 4 eliminates this problem, but I never tested yet.

Removing hint_color from terrain shaders fixed it, but I'm not very happy with that... that hint was correct, it would be incorrect to remove it, unless people are happy with the fact tints become lighter.

[Zylann]

The re-implementation is now available to test in branch gpu_painting

[Zylann]

This is now in master. The only thing remaining are the following:

• Some operations can't be done efficiently with a fragment shader (such as computing min and max of an area), but this would more likely happen either with GDNative (like it does now) or Compute Shaders.
• The very limited choice of framebuffer formats causes a waste of bandwidth, memory and CPU. So although the new implementation is still way faster than GDScript, it could be made even more efficient by supporting a few simple things in the API, such as more target formats and new Image methods to separate channels.
• 32-bit float is still not available, which is more minor but can still annoy people who make extremely high mountains.
• sRGB conversion is a pain when painting colors on GPU, if the texture is being used in 3D at the same time. However, with Vulkan, having this conversion occur on samplers instead of textures would fix it. Unfortunaley, GLES3 is still a thing in Godot 4 so it might be complicated to support both.
• Being able to render on-demand is no longer required by the painting system, because it is now implemented in such a way it can wait for one frame, but might be nice to have regardless if we want to simplify things.

[TokisanGames]

Color painting is still broken. I previously used the Nov 10th commit where I could paint on my snow texture and get the exact color I wanted. Grass has color map opacity set low so it doesn't impact it.

Now when I paint, it stays brown only where I'm holding down the mouse, like a spotlight, and everywhere else is black. If I let go that spot light of color will remain. It also makes it difficult to fully erase with white. My old color is untouched.

If I use a shader with hint_albedo removed, it messes up my old color and new colors are not accurate.

However, if I convert the colormap to linear, I'm able to get reasonably accurate colors with your choice of calculation speed:

uniform sampler2D u_terrain_colormap: hint_white;

vec4 toLinearFast(vec4 col) {
	return vec4(col.rgb*col.rgb, col.a);
}
vec4 toSRGBFast(vec4 col) {
	return vec4(sqrt(col.rgb), col.a);
}

vec4 toLinearMed(vec4 col) {
	return vec4(pow(col.rgb, vec3(2.2)), col.a);
}
vec4 toSRGBMed(vec4 col) {
	return vec4(pow(col.rgb, vec3(.4545)), col.a);
}

vec4 toLinearSlow(vec4 col) {
    bvec4 cutoff = lessThan(col, vec4(0.04045));
    vec4 higher = vec4(pow((col.rgb + vec3(0.055))/vec3(1.055), vec3(2.4)), col.a);
    vec4 lower = vec4(col.rgb/vec3(12.92), col.a);
    return mix(higher, lower, cutoff);
}

vec4 toSRGBSlow(vec4 col) {
    bvec4 cutoff = lessThan(col, vec4(0.0031308));
    vec4 higher = vec4(vec3(1.055)*pow(col.rgb, vec3(1.0/2.4)) - vec3(0.055), col.a);
    vec4 lower = vec4(col.rgb * vec3(12.92), col.a);
    return mix(higher, lower, cutoff);
}

...
vertex() {
	vec4 colormap_tint = toLinearSlow( texture(u_terrain_colormap, UV) );

[Zylann]

I don't really know what I can do here. It's not broken, but it's a compat-breaking change that happened because being forced to workaround the conversion is a pain, and I'd like to be able to paint in linear like before, but my last attempts failed...

[TokisanGames]

I would add my linear conversion to your default shaders and call it a day. It's unusable now. Things will probably change when vulkan is released, so you can figure out a more elegant solution then.