Very large tilemaps, loading tilemaps in parts ?

[gotenxds]

Hello, I'm working on a game which in the end size should be around 1,00000000 tiles, this is a big number.
I'm searching for way to load the tilemap part by part, here is what I tried so far.

1. I created small scenes, each scene consists of 100*100 grid.
   I then load the adjacent scenes by the position of the player with LoadSceneAsync.
   This works, but it leaves me with 10,000 scenes which even when empty require are around 1 hour per build.

2. I thought about saving the tiledata into files then loading the tiles by the data according to the position of the player, having no more then around 50 tiles loaded.
   I was unable to do this, as I cant find a way to access the tileData or tileAnimationData.
   Serializing TileBase does not seem to work.

3. I also thought about using the new SubScene, combining that with my first solution might work great, but alas it seems that SubScenes cannot render tilemaps or even sprites as of now.

I would love some suggestion about how to tackle this.

[edwardrowe]

I initially expected Tilemap to be a standalone class, so that we could pass the tile + position information around more easily without being coupled to MonoBehaviours. Unfortunately it's not that way, and it does make it a bit tricky to do stuff like you are wanting. (I'm sure there is a reason, would love to know why it is a MB MonoBehaviour.)

That said, it's possible to easily make containers to store tilemap information. You basically just need <Position, TileBase> pairs, which you would use to populate a Tilemap at runtime (the tilemap centered on the player). You'll need to serialize your references to TileBase, which is gonna be the biggest pain. But that's a whole other topic.

I'm not sure whether you are painting those bazillion tiles, or procedurally generating them. If you are painting them, and want to serialize the structure to be read from at runtime, you can get painted tiles with Tilemap.GetTile(position). You similarly would use Tilemap.SetTile(position, tile) to assign it. (Does this answer your 2 suggestion above?)

Am I understanding the question?

[gotenxds]

Hya @edwardrowe first of all thanks for trying to help me ^^
Not sure what you mean by MB or where I mentioned that size, If you are talking about the amount of tiles we are creating an open world 2d game, a relatively big one for a 2d game but not that big in terms of open world.

as for your suggestions, I have tried doing exactly that, I create dictionaries mapping from tileMap to <Position3int, TileBase> In theory this would have worked, If I could only serialize and deserialize the tileBase Class, to my dismay I was unable to do that, when ever I the deseralzation returned null, I tried using savegame pro to, maybe my serializing logic is flawed, I thought, but I got the same result using that library, that what brought me here, in the hope someone would have some knowledge about saving the tiles data into a file.

p.s love your work ^^, a few of your redbluegames blog posts helped me in the best. (you should really continue doing these =])

[edwardrowe]

Yeah sorry, corrected my post. MB was just shorthand for MonoBehaviour.

So it's sounding like the best solution to this is going to be getting tiles to serialize. Your Dictionary is exactly what I'd do (and what we do).

Serializing references to assets is really tricky. We haven't really figured out exactly how we're going to do it for Sparklite yet. But the easiest way I can think of would be to map tile names (or paths, or something you can easily save) to tiles, stored on a GameObject (let's call this the Library) that you can access wherever you load the tiles. Make your Dictionary class serialize as <position, string> instead of <position, TileBase>, where the string is the tile names (let's call this the tileID). So then when you are loading tiles you can get the actual TileBase by using the tileID to look it on up on the Library. Does that make sense? (this is kinda hasty so I apologize if it's glossing over too much).

To clarify, you'll have:

• A tile streamer which has a reference to a Library and tile data (serialized, or deserialized) to load from (this is placing the tiles around the player based on serialized data)
• A tile Library which has I'd say a List of <string, TileBase> pairs. This needs to be placed in the scene you will play in.
• SerializeTilemap - the structure you will store tiles in, which is Dictionary<position, string>, where the string is the "tileID"

And I've been meaning to do some more posts soon. I'm really glad you've found them helpful! Thanks so much for letting me know!

[gotenxds]

Not exactly sure what you mean, as I cant serialize the TileBase or the tileData(cant get a ref to tileData).

I was able to write something like so where on Load I created a Big (Hugh) dic by using the getTilesBlock method, this works, but It wont scale as the memory required to do it is massive, even If I use dic O(1) to get that data The memory footprint would be too big to maintain.

If you mean that I should create some kind of gameObject (a TileMap ?) containing all of the possible tiles and then referencing them, I think that might work, but again, that will scale poorly as the game gets larger and larger I think, I am not sure how many distinct tiles we will have by end of the game, but a few thousand is more then likely.

Then again, that would be the best idea I've seen so far.

I would love someone from unity, @ChuanXin-Unity, to give us some input on this ^^.

Edit: just talked to our artist, she said she can easily see us get to 25k unique tiles by end game.

If worse come to worse I can always make scenes that will only be those tile collection and then load them as we go near areas in the world using them. But if the unity team can step in and suggest a more elegant solution that would be lovely.

[edwardrowe]

With regard to serializing TileBase in the TileLibrary object - if you place it in a scene, and reference the tiles on an object, they will serialize and deserialize properly when Unity loads the scene.

With regard to serializing TileData - It looks like you can technically serialize everything that you could want to do on a placed tile (transformation). You'd also need to make your serialized Tile structure include that, in addition to the tileID. (Again, this is where a non-MonoBehaviour Tilemap would be useful).

This is a challenging problem, but at it's core this is mostly related to Serializing and Loading Assets.

You are right, loading all the tiles will be a lot of memory (for the textures, not as much the tiles themselves). If your Dictionary<pos, TileBase> is too big to keep in memory, you'll have to stream THAT in, which is possible but a pain.

My unsolicited advice would be to come up with a solution that can be refactored to scale if necessary, consider what that refactor would be, but then do it as quickly and simply as possible. Another approach is to identify now what data you will likely need to stream in the future, and put in the framework for it, but not flesh it out fully.

[gotenxds]

Hey, I know that unity serializes the objects for me, but I cannot control what parts it loads into the scene that way.

I would love to stream the data, but again, as I cant control the data loaded I cant stream it.

I'l explain this again to make sure we understand each outer.

• Serializing TileBase via a script is not possible.
• Serializing the TileData IS possible but the current tileMap api does not allow us to get hold of the TileData object.
• It's possible to create an Object holding all of the tiles and then referncing this object, the issue here is that Unity handles the deserializing and loading of the object and so you cannot stream it or load it in chunks.

Only current way I see is creating some kind of struct/class, lets call it WorldTile (as TileData is already used by unity)

That WorldTile would look somewhat like this:
{ Vector3Int position, Sprite sprite, string id, AnimationData // some kind of additional data for animated tiles. }
You would store this info inside an Object on the scene.
Then, using a saved data stracture of the form <position, id>
you would retrive the desired tiles and render them around the player.

Issues with this are both the heavy usage of memory for all the textures and the intial load time as unity does not allow you to load this data partially.

A possible modification could be saving the path to the texture on the object instead of the texture itself and the async load resource it. this will also require some kind of implementation of a "GC" to remove tiles textures that were not used in X last frames or something to that affect.

[gotenxds]

I've thought on another possibility, it's possible to create a scriptable tile that getTileData will expose the tile data(to a file/another collection)

[ChuanXin-Unity]

The Tile class is a simple wrapper asset for TileData. If you serialize a Tile, that is as good as serializing TileData.

If you check the Scripting API, the properties for both the Tile and the TileData are the same:
https://docs.unity3d.com/ScriptReference/Tilemaps.Tile.html
https://docs.unity3d.com/ScriptReference/Tilemaps.TileData.html

You could write an extension function that can retrieve the TileData from Tile, if you really need it:
`
using UnityEngine.Tilemaps;

public static class TileExtensions
{
public static TileData GetTileDataFromTile(this Tile tile)
{
var tileData = new TileData();
tileData.sprite = tile.sprite;
tileData.color = tile.color;
tileData.transform = tile.transform;
tileData.gameObject = tile.gameObject;
tileData.flags = tile.flags;
tileData.colliderType = tile.colliderType;
return tileData;
}
}
`

Using the Tile asset/s, you could use AssetBundles to stream in and build the parts of the Tilemap you are interested in.

[gotenxds]

To anyone still interested I managed this:

4 scenes, 5 tileMaps each, 10000 tiles each
So 200000 loading and unloading over time here

[wei398262294]

To anyone still interested I managed this:

4 scenes, 5 tileMaps each, 10000 tiles each
So 200000 loading and unloading over time here

hello，can you show your project in your github

[DaseinPhaos]

To anyone still interested I managed this:

4 scenes, 5 tileMaps each, 10000 tiles each
So 200000 loading and unloading over time here

hello，can you show your project in your github

One way to do this, I guess, is to pre-process the original big scene, splitting it into different sub scenes which then contains different sub-tilemaps.
At runtime tilemaps/scenes gets loaded/unloaded based on the viewing frustum of the main camera.

This is the process I'm using right now, but it has its problems. For one, the pre-proscessing takes too much time. As any modification done on the original map requires another pass of pre-processing, the iteration time slows down quite dramatically.

[Jackoberto]

I know I'm very late but I've found a quite a efficient way to store tile maps I couldn't tell you how to best load though

I save the positions and names of tiles as a file and by scanning from buttom left to upper right
The positions are stored in two int arrays one with X and one with Y and then it stores the names of tiles in a string array.

Then I have like you've mentioned a gameObject with a list of references to all tiles and in our game that's no problem due to tiles being 750 bytes each

One level of 2500 tiles is stored in a 36 kb file so with 1000000 total tiles it would only take up 14.4mb

The system is not perfect as I made it in a day but it works perfectly for my level editor to be able to make custom maps in our game after a build has been made

This combined with a algorithm to load only the necessary tiles could be a solution

Another idea is if you store the names as I do is that you can load with Resources.Load(tileName[i]) and then use this with a scan at certain points to see what needs to be loaded in

[Qusdrok]

I have the same question as you, I hope someone can help you and me

[bentaller-lion]

If you want to store that many tiles you should break your map into pieces.
Imagine this:

Files:
#1. #2. #3. #4. #5.
#6. #7. #8. #9. #10
#11. #12. #13. #14. #15

Now you can see, you have a tile grid of tile grids (where each of those #s above is a txt file)
1.txt, 2.txt, 3.txt etc.
each of these text files can have a csv of all the tiles AND a list of which text files they link to for all of the neighbors.

Now, imagine the player is somewhere in the second textfile, as they walk to the western edge of 2 you know you should start loading #1 in the background and stitch it to the edge of your existing map.

If you walk down to the bottom corner of #2 you may want to load #1, #6, and #7 and stitch all of those to your map
You can create a data structure that will help you understand the bigger picture and what sections you need to load by defining neighbors.

MapTile
textAssetPath:string
size:Vector2
westNeighbor:MapTile
northNeighbor:MapTile
southNeighbor:MapTile
eastNeighbor:MapTile
nwNeighbor.... you get the idea.

Then in your player object you can track your global position on the world map, and simply use some math to figure out what MapTile you are on, and then when you get to the edges ask for the neighbors

Lets pretend you have 10 x 10 in each section to make it easy...
Lets pretend your coordinates are currently 32x15 on the world map.
Well..by simply dividing by 10 on X you can see you are in column 3 (20% of the way down) and dividing 10 on Y you are at row 1 (50% of the way down)

If this percentage gets to 75% or falls to 25% you should ask what the next tile is in that axis and load it, its probably also safe to unload the opposite side cuz its way out of view.

Thats how I would do it anyway.

[dangledorf]

Perhaps you figured this out by now, but you can get the tile data object from a tile at a position like so:

public GameObject GetMoveParticleAtPosition(Vector3 worldPosition, Tilemap tilemap)
{
   // An example of grabbing a tile at a location and pulling data from their ScriptableObject to do nifty things.
    Vector3Int gridPosition = tilemap.WorldToCell(worldPosition);
    var tile = tilemap.GetTile<YourCustomTileClass>(gridPosition);
    if (tile == null)
    {
        return;
    }
    
    // Find the particle to spawn.
    var possibleParticles = tile.MoveParticles;
    if (possibleParticles == null || possibleParticles.Length == 0)
    {
        return;
    }
    
   return possibleParticles[Random.Range(0, possibleParticles.Length)];
}

You could do something similar and parse through all of the Tilemaps in a Grid and store their tile objects and coords in a serialized class. Then use that to split everything out into chunks and generate grids and recreate them.