Stop allowing bulk downloading of tile.openstreetmap.org tiles

[Firefishy]

I am part of the OpenStreetMap.org sysadmin team.
tile.openstreetmap.org traffic exceeds 1.6TB/hour and 41600 req/second during our daily peak times. The service is run by us, we are all volunteers.

We have a simple usage policy and don't require API keys or anything like that. We intentionally try keep the bar to using our tiles low.

BUT! Our usage policy specifically disallows bulk downloading of tiles because it puts extreme pressure on our infrastructure and strains our limited on-demand rendering capacity. Bulk downloading of tiles disrupts our service and severely degrades our ability to service tiles.

1. We ask that QTiles STOP allowing bulk downloading of tile.openstreetmap.org tiles.
2. We ask that QTiles use a dedicated HTTP user-agent (eg: QTiles/1.0.0) to allow us to differentiate between QGIS (built-in TMS support) and QTiles requests.

It would be a massive loss to the QGIS community if we were forced to block QGIS entirely because of QTiles allowing bulk downloading of OSM tiles.

[simgislab]

1. QTiles for QGIS 3 that is currently available though official QGIS plugins repo is not our development. It was an unsupported (by us) port (by someone) and we don't control it. I.e .you might be at the wrong address.

2. I don't understand the solution yet. QTiles does NOT separately pull OSM tiles. The common use case is: (a) Add OSM to QGIS (b) Render a tileset from project. It is QGIS that renders and pulls data (QgsMapRendererCustomPainterJob).

We can do something on our side (our version of QTiles that we indeed control).

[tomhughes]

The problem is that we on the server side can't tell the difference between somebody using QGIS interactively and somebody "rendering a tileset" as you put it.

I don't know enough about QGIS to know exactly what you mean by that - is that essentially asking it to render and export an image for a given area using it's configured layers? If so then I guess we need to talk to QGIS about using a modified user agent in that case.

[simgislab]

is that essentially asking it to render and export an image for a given area using it's configured layers

yes

need to talk to QGIS

yes

[Zverik]

@Firefishy are there some numbers linking QTiles or QGIS to bulk downloading tiles? I assumed this was too complex for a regular person. Maybe some other tool like SASPlanet is spoofing the user agent?

[Firefishy]

Hmmm ok, so this is likely the bulk tile downloader we are after then: https://bitbucket.org/we-do-it/qgis-latlongo-stage-1/src/master/

[Firefishy]

@Firefishy are there some numbers linking QTiles or QGIS to bulk downloading tiles? I assumed this was too complex for a regular person. Maybe some other tool like SASPlanet is spoofing the user agent?

We primarily track QGIS by its User-Agent. Most QGIS plugins use the QGIS user-agent. QGIS is the single biggest tile consuming app. @pnorman can provide stats breakdown.

Yes, spoofing the User-Agent is possible. Classic tragedy of the commons.

[simgislab]

reopen if necessary

[Firefishy]

@simgislab Please be aware we are going to escalate this issue now. Your tool is abusing OpenStreetMap services. You are degrading our services used by others. I will contact you offline with more details.

[Firefishy]

PS: I cannot re-open this issue.

[simgislab]

are there any numbers showing that this is specifically QTiles problem and not general QGIS or other plugins?

[Zverik]

@Firefishy please be reasonable. NextGIS is not abusing OSM services: they just made the plugin. QTiles is not abusing OSM services: it renders a QGIS project into tiles, not downloads OSM tiles specifically. There cannot be an easy solution, and threats won't help resolve it faster. Discussion is better than ultimatums. We are open-source developers, just like you.