How does cache --tile-list option work?

[bagage]

While using file cache system, I tried to use the --tile-list option to force cache refresh for a given list.

My file contains:

1/0/1
2/1/2
3/3/4
4/7/9
5/15/19
6/30/39
6/31/39
7/61/79
7/62/79
8/123/159
8/124/159
9/247/318
9/248/318
9/247/319
9/248/319
10/495/637
10/496/637
10/495/638
10/496/638

And I'm reading that file with tegola cache seed --overwrite --tile-list $WORK_FILE. Instead of seeing seeding map (xxx) tile (z/x/y) took: Xms 19 times (there are 19 tiles to refresh), actually tegola refresh all my tiles for all zooms (7416 tiles, because I limited it to zoom 7)???

Did I miss something?

[stvno]

It is broken, the sendTiles function will parse the tilelist-file and for each tile it will test if that tile is within the given zoom range. However the code will return all tiles within that zoomlevel, instead of just the tiles from the list:

https://github.com/go-spatial/tegola/blob/v0.8.x/cmd/tegola/cmd/cache.go#L331

// range
			for _, zoom := range zooms {
				err := tile.RangeFamilyAt(zoom, func(t *slippy.Tile) error {
					if gdcmd.IsCancelled() {
						return fmt.Errorf("cache manipulation interrupted")
					}

					if z1, x1, y1 := t.ZXY(); z != z1 || x != x1 || y != y1 {
						c <- t
					}

					return nil
				})

				// graceful stop if cancelled
				if err != nil {
					return err
				}
			}

[stvno]

I think it should be:

if z1, x1, y1 := t.ZXY(); z == z1 && x == x1 && y == y1 {
     c <- t
}

[stvno]

this is a bad idea: it will check each tile against all possible tiles in the zoomRange :(

[ARolek]

@ear7h can you take a look at this? I’m traveling today so won’t havs a chance.

[stvno]

I've removed the whole tile.RangeFamilyAt function and it will now run just the tile list with no check whatsoever. Which means it ignores min/max-zoom parameters and is also hard to kill:
ctrl-c only kills the running seed, not the channel I believe.

https://github.com/stvno/tegola/blob/seedTileList/cmd/tegola/cmd/cache.go#L312

[ARolek]

Min / max zoom is important. I will have to look at the code but IIRC that’s support for taking in a tile list that has all tiles at a specific zoom but you want to traverse a zoom range for that tile. This is the case when working with imposm tile lists as all z/x/y values are at the same zoom.

[ear7h]

To expand on @ARolek's comment, the tool we targeted to use this feature with (imposm3) generates a list with the same zoom levels. In this case, it is required that we update all tiles up and down the tile quad tree so that all tiles in the cache reflect the changes.

@bagage it seems like your file already gives gives a tile family, starting from the following files.

10/495/637
10/496/637
10/495/638
10/496/638

My recommendation is to remove the other tiles from the list and add --max-zoom=10 to your command. But, keep in mind that the cache at higher zoom levels (if existent) will become outdated if the changes you made to the database affect layers of these higher zoom levels.

edit: to more directly answer the question, the reason it would seed so many tiles is because we look at all the tiles which might be affected by a change to 1/0/1, this means all tiles (at all zooms) in the lower left quarter of the world. But, based on the tile list it seems like you want to update all tiles affected by changes to the 10/*/* tiles, tegola cache seed .... does this heavy lifting which is why you only need the tiles with the smallest zooms.

[bagage]

Oh, thanks for the answer @ear7h, I did not see it that way! I'll do the lowest zoom only then, and see if that is OK for my use-case. Thanks all for the pointers!

[ear7h]

There is still a problem that lower zoom tiles will be redundantly cached, for example running tegola cache seed --max-zoom=10 --tile-list=xxx.txt will cache the zero tile (0/0/0) 4 times, once for each time it walks up the tile tree.
xxx.txt:

10/495/637
10/496/637
10/495/638
10/496/638

[stvno]

ah I see, so the trick is to only send in the highest zoom levels and let tegola decide what tiles to do? I want to be able to reseed the entire Netherlands once there is a data update (unfortunately I do not know which bit is updated - don't get me started...) so I only want to update tiles that are actually containing land and not spend precious resources tiling empty sea.

So I created a list of all the tiles of the Netherlands and wanted to reseed just those. BTW seeding the same tile multiple times is a problem in this scenario since I'm running with --overwrite and it will keep reseeding the same tiles from z=18 upwards..

[ear7h]

@stvno The trick is actually to pick a small amount of tiles with the highest zoom that seeds the area you want. That way you minimize the redundancy while still sticking to a select area. A combination of the tiles from the following screen shot would probably work efficiently (sorry for the poor quality):

edit: it looks like you also need tile 8/133/84 the country lines veer into that tile.

In the meantime you could check if seeding from a bounding box is faster as it doesn't suffer from this redundancy issue. Also, note that seeding empty tiles would not take as much resources as reseeding tiles. I realize this doesn't completely solve the issue, it's just that the current algorithm is only optimized for tiles which are not geographically near each other. With this in mind and if you only have geometries for the Netherlands, the tile 5/16/10 could also work more efficiently than a full list.

To solve this problem entirely, I think there should be command line flag, like --explicit-list which renders strictly the tiles in the given list. cc @ARolek when he gets a chance to look at this.

[stvno]

@ear7h thanks for the pointers. Once the redundancy issue is solved, this sounds like a way more manageable way of reseeding than a 6.5 million long tile list :)

I've already rewritten the --tile-list function to behave like an --explicit-list so I'll clean up the code and send a PR.

[bagage]

you could check if seeding from a bounding box is faster as it doesn't suffer from this redundancy issue.

Wow, actually this is the feature I should have used since the beginning personally! Instead of listing tiles I need to refresh I should provide tegola directly with the bounding box to clear :).

Still --explicit-list would be valuable, but in the mean time @stvno (or other?), I think the configuration option --tile-list should be described more accurately to reflate what has been said here (eg. that all affected tiles will be recomputed, not only the list ones).