Rework the TileMap terrain system to improve its usability

[Portponky]

Describe the project you are working on

General 2D games, though I have a focus on puzzle or casual games which rely heavily on tilemaps.

Describe the problem or limitation you are having in your project

The terrain system in 4.0 is a replacement for the autotiling system in 3.x. Whilst it is designed to be more powerful and usable, it has a number of issues and behaviors that impact its usability. It also regresses on a number of features from autotiling.

Editing terrains affects neighboring tiles in unexpected ways, often altering their type.

odd-tile-behavior.mp4

odd-tile-behavior-2.mp4

Terrain sets do not play nicely at all.

terrain-sets-dont-mix.mp4

No substitute for _is_tile_bound in 3.x. No way to make different types of terrain interact like you could in 3.x with autotiling.

joining-tiles.mp4

Results depend on painting order.

order-dependent-painting.mp4

There are other similar behaviors, but hopefully you get the idea.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

There are four key changes which I believe will both improve the usability of the terrain system, and push it beyond what 3.x could do:

• Enforce that all of a terrain’s tiles have the center bit of that terrain type. Essentially, remove ‘centerless’ terrain tiles (extension of New TileSet terrains system should reintroduce "center" bit/terrain type #3875 )
• Remove the concept of terrain ‘sets’, instead just define individual terrains with connection types.
• Make terrain tiles’ peering bits match multiple terrains (duplicate of Allow multiple terrain types as a tile's peering bits #4689 )
• Tidy up some curious API (such as the removal of the ignore_empty_terrains flag)

Whilst each has a justification, the overall goal would be to reduce the complexity of the terrain matching algorithm, whilst also increasing its flexibility and stability of behavior. There are two key ideas here: solidifying the data-centric design of tilemaps, and linearizing the constraint satisfaction problem.

Enforcing that each terrain tile is correlated with a terrain type gives fixed constraints for fitting terrains. This is also generally good for the user, as the terrain types they place are invariant. If a user places a ‘water’ tile, for example, that should be adhered to by the engine, and not changed to a better fitting tile of a different terrain type at a later time. Any game with precision design is going to have that requirement (e.g. platform games).

Grouping terrains into terrain sets, as a feature, adds some complexity and constraint to the user, but terrain sets don’t really seem to offer any specific advantage. I understand that they are there to limit/define terrain interactions, but seeing as no such restriction existed with Godot 3’s autotiles, it feels like the terrain sets are only there because a similar feature exists in other editors such as Tiled.

Multiple peering bits to allow generalized connections between tiles will linearize the constraint satisfaction problem that occurs when trying to pick tiles for terrains. It would allow individual cells of the tilemap to select terrain tiles without needing to propagate constraints to their neighbors, nor care about how their neighbors connect to them (if at all). That way, tiles can be picked in any order, via a simple search of the possible options, and will behave consistently.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

When a tile needs to connect to its neighbors, it looks at all the tiles matching its current tile’s terrain type. For each tile, it checks the peering bit arrays against the neighboring tile terrain types. These are graded and the tiles that gets the best score is placed.

I have taken some time to mock up how this would appear:

masked-tile-connections.mp4

masked-tile-corner-connections.mp4

These were calculated using the following scores for peering bit (arrays) for connections to neighbors:

• Peering matches and neighbor is a different type: +5
• Peering matches and neighbor is the same type: +3
• No peering (array is empty): +1
• Peering does not match neighbor: -5

If this enhancement will not be used often, can it be worked around with a few lines of script?

The terrain system will be used in a significant amount of 2D games, both in terms of editing and designing levels, and in playing them too. It will be used very often.

Is there a reason why this should be core and not an add-on in the asset library?

The terrain system itself is a vital feature. For example, here is a user requesting that they want to make a layout in a similar (but simpler) way to the incredibly popular game Celeste: #4689 (comment) Godot should be able to do such a common and popular design out of the box.

[MewPurPur]

Really in support with these ideas! After testing tilemaps with my usual workflows, I think everything they can do is very powerful and ready for Godot 4 stable, but terrains stick out like a sore thumb in that regard.

I think you've identified the issue well. Making it so every autotile is part of a specific terrain covers every realistic use case of tilemaps I can think of, yet this workflow is super weird to use. I don't think different use cases should be supported to its detriment.

Even if UX is not good with this, for 4.0 stable I believe that having cohesive tilesets that can accomplish anything users commonly need should be a way bigger priority than having a good polished UX and UI, as those can be done in later versions. So IMO this proposal should be considered seriously while Godot 4 is still in beta.

I'm not sure about removing terrain sets, but I do agree they're a bit useless right now, at least I can't think of any reasons they should exist. But I think the ability to group terrains would be nice in order to make matching between big groups of tiles more straightforward and less error-prone. For example by making all tiles from the "Platforms" set always match with all tiles from the "Solid Tiles" set, instead of having to pick if you want them to connect with grass, sand, rock, bricks, dirt, etc... for every terrain type. Though I suppose this may be a rare use case, as most games would use different transition tiles.

[groud]

Enforce that all of a terrain’s tiles have the center bit of that terrain type. Essentially, remove ‘centerless’ terrain tiles

I don't think this can happen. Some border tiles may need such a behavior in the 3x3 mode, as a transition may cover either a third or two third of a tile, and depending on that the center bit might be set or no.

Remove the concept of terrain ‘sets’, instead just define individual terrains with connection types.

The problem with that is that different kind of matching mode, like "matching sides" and "matching corners and sides" should be able to co-exist with each other in a single atlas, they are use cases for it. The terrain sets make this simpler than setting it up per-tile, and make it clear about which tile can connect to each other.

Make terrain tiles’ peering bits match multiple terrains (duplicate of #4689 )

This shouldn't be done at the bit level, but at the terrain level. Otherwise it will be super tedious to setup.

Tidy up some curious API (such as the removal of the ignore_empty_terrains flag)

This flag is needed so that empty tiles do not become a constraint in some areas. It's a bit like the "ignore bit" in 3.x if I am not mistaking. But I am open to suggestions regarding this.

I'm not sure about removing terrain sets, but I do agree they're a bit useless right now, at least I can't think of any reasons they should exist.

This is mainly for performance, and because different sets are needed anyway when you use different matching modes.

[MewPurPur]

Ah, makes sense about the different matching modes. Would they still needed if center terrain bits were enforced?

I don't think this can happen. Some border tiles may need such a behavior in the 3x3 mode, as a transition may cover either a third or two third of a tile, and depending on that the center bit might be set or no.

May I see an example of this that would be impossible/cucumbersome to implement without enforced center bits?

[groud]

Ah, makes sense about the different matching modes. Would they still needed if center terrain bits were enforced?

Yes. Center bits used not to be here before and the algorithm could (kind of) work anyway, so that's unrelated.

I suppose that, maybe, it would be possible to get rid of the terrain sets, though I wonder how it would work when mixing the 3 different modes together. I guess the "matching sides" and the "matching corners" can work with the "matching sides and corners" mode, but "matching sides" and "matching corners" would not work together. I guess it's something that could be done and would simplify things, so I am not against it. (It's a huge refactor though)

May I see an example of this that would be impossible/cucumbersome to implement without enforced center bits?

Basically any tile with a thin border, which would be mainly empty but would serve as a transition from a terrains to void.
But well, enforcing the center bit does not make a lot of sense anyway, and I am not sure what that means and what that would solve. The "empty terrain" is kind of a terrain in the algorithm anyway, so I am not sure what it would be helpful for.

[MewPurPur]

Ah I see, so its use case is tiles that are from the "empty" terrain, but have content nonetheless. OK, I'm convinced their existence makes sense. But I think we should still somehow make it so the center terrain of each tile on the tilemap is always preserved unless the user changes it. Tiles from the empty terrain (no center bit) could be kept, but they should be the only scenario where a tile pops up without the user placing it, since in that case, they keep empty terrain empty.

[groud]

But I think we should still somehow make it so the center terrain of each tile on the tilemap is always preserved unless the user changes it.

I guess that could work. Maybe it's the most expected behavior but it has an implication. Right now, painting a terrain with this setup:

Leads to this:

Peek.12-10-2022.14-44.mp4

While with that new system it would lead to this:

Peek.12-10-2022.14-41.mp4

That means the algorithm would do no effort to make things match, but would be more predictable I guess.

Tiles from the empty terrain (no center bit) could be kept, but they should be the only scenario where a tile pops up without the user placing it, since in that case, they keep empty terrain empty.

I think this is kind of a bug, I agree that a tile should not be modified if its terrain pattern matches the one selected by the algorithm. So if a tile is not a terrain or is from the wrong terrain set, it should not be removed but stay as-is instead.

[Portponky]

I'm still unconvinced that centerless tiles make sense. Is there an actual concrete use case, rather than a hypothetical one? Such as a game or tileset from a user which can be best satisfied with centerless tile constraints.

If you're describing something like this:

That was thrown together quickly with Godot 3.5 and some tiles from Open Game Art. If you wanted to transition tiles to void (or between void and other tiles) then why wouldn't something like this be appropriate? It's easy to do and very logical.

With centerless tiles, what happens when you erase a tile? Should it paint a different terrain tile if there's a centerless tile with a valid connection?

[groud]

That was thrown together quickly with Godot 3.5 and some tiles from Open Game Art. If you wanted to transition tiles to void (or between void and other tiles) then why wouldn't something like this be appropriate? It's easy to do and very logical.

Sure it make sense for a simple layout, but you may want to have, for example, a tile with grass on a side, tall grass on the other, and void in the middle. The terrain system allows this flexibility and it should be kept. To make it clear, "no terrain" behaves almost exactly like any other terrain. There's no reason to remove it, it won't simplify anything.

With centerless tiles, what happens when you erase a tile? Should it paint a different terrain tile if there's a centerless tile with a valid connection?

Yes, exactly. It tries to make the center bit match your expectations, then adapt the surroundings (the tile itself but if needed the surrounding tiles also) to what you are asking of.

[Portponky]

Sure it make sense for a simple layout, but you may want to have, for example, a tile with grass on a side, tall grass on the other, and void in the middle. The terrain system allows this flexibility and it should be kept. To make it clear, "no terrain" behaves almost exactly like any other terrain. There's no reason to remove it, it won't simplify anything.

It seems like the simplest answer would be for the user to consider those tiles as one type or the other, and let the terrain handle the transition normally. That the centre of the tile happens to be empty (graphically) is not a use case that should be informing the design, or at least, I don't think so.

Is this use case worth all the complexity that centerless tiles bring to the system? Especially considering how easily it could be handled otherwise.

[groud]

Is this use case worth all the complexity that centerless tiles bring to the system? Especially considering how easily it could be handled otherwise.

I don't really understand what make you think that. No, I don't think it would make things simpler, the terrain algorithm has to handle any kind of transitions from one terrain to the other (that's the whole point of terrains vs autotiles). So removing one special terrain (as I explained the "no terrain" is considered like any other terrain, it's simply a terrain like the others) won't make things simpler, it's the exact same thing but with one terrain less.

[MewPurPur]

@groud I'm not sure what the gifs show exactly - I suppose they show examples of placing terrain in connect mode. In such a tileset, I still think the current behavior gets in the way, because it assumes what other tiles you're gonna plant. In the gif, the initially placed tile became the bottom right one, but maybe the user wanted it to be the top left one and was about to draw that little square, but then the tilemap got in the way. And then they try to erase those unintended tiles, probably leading to more artifacts.

So I think just defaulting to a wrong type of tile until valid neighbors are set up by the user would be the better behavior.

@Portponky Maybe erasing centerless tiles could be made invalid and only possible by changing the adjacent terrain that made the centerless tile appear.

[groud]

@groud I'm not sure what the gifs show exactly - I suppose they show examples of placing terrain in connect mode. In such a tileset, I still think the current behavior gets in the way, because it assumes what other tiles you're gonna plant.

Yeah, it's in the connect mode. Basically what it mean is that, if we don't modify the surrounding tiles' center bits at all cost, then there might be situations where no valid transitions can be found, and things will end up in an "invalid state", even in simple cases like that one. But if that's fine to everyone, I guess it should be quite straightforward to modify.