Change to tiles acceptable usage policy

[zerebubuth]

The usage of tile OSM tile infrastructure by the OpenStreetMap website was recently measured at approximately 11% of tile server output, so the vast majority of OSM's tiles are rendered to support 3rd party sites and apps. Although we want to support use of OpenStreetMap data, there are some use cases for which OSM and the surrounding ecosystem receives little or no benefit.

It has been suggested that OWG introduce two new rules to the tiles acceptable usage policy:

1. That only open source apps may use the tile server. (Alternatively, and more strictly; that commercial apps may not use the tile server.)
2. That only publicly-accessible sites may use the tile server. For example, sites requiring a login or accessible only from behind a firewall or VPN would not be public, and therefore would not be allowed to use the tile server.

The technical implementation of this is a detail, so please keep discussion to the policy of whether or not we want to begin to restrict usage in this way.

[iandees]

A couple thoughts:

1. The discussion on this policy should be taken in the context of the recently-announced corporate membership program. There will be claims that this policy is an effort to direct money to potential corporate members. This policy should explicitly not recommend any alternatives, or if it does, have very clear reasons for recommending them.
2. With that context in mind, it might be good to have a discussion about if the operations team and OSMF want to allow use of OSM.org tile infrastructure by commercial entities or in commercial situations (ad-supported or for-pay apps or sites).
3. Although it might be considered a technical implementation detail, the policy should describe in general what the repercussions are for not following the policy. This would help people understand why they are blocked and help alleviate the inevitable questions on IRC.

[lonvia]

Give that Nominatim has similar problems, I would suggest to include the search acceptable usage policy in the discussion. Whatever the outcome, we should apply the same rules for tiles and search.

[gravitystorm]

Although I have views on this issue, given my conflicts of interest I recuse myself from this topic.

[zerebubuth]

@iandees thanks! Those are very good points.

1. The motivation for the proposed changes to the policy have nothing to do with corporate membership and, while I'd encourage any organisation to join OSMF, I'd agree that any such membership should not imply that any recommendation would be given. In general, we would point to switch2osm to provide anyone with further information about alternative sources of tiles - either other providers or hosting their own.

2. Indeed, that's exactly the discussion I was hoping to have! The proposal frames it in terms of "open source apps" and "publicly-accessible sites", but it could potentially be more specific about "ad-supported" or "pay-for" apps or sites.

   I think most people would be happy for as many people as possible to see and use OpenStreetMap tiles, but we aren't able to practically support that. While there are on-going efforts to increase capacity and we welcome donations of tile caches and rendering servers, it's possible that some uses of OSMF tile infrastructure are less beneficial to the community than others.

   For example, one might think that use of OpenStreetMap tiles in a closed-source, pay-for app is less deserving of support than use in open-source, free apps. And therefore, we'd prefer to start restricting use in those sectors first, rather than just an overall restriction on all users as we max out our capacity.

3. I would love to have that discussion as a follow-up to this one. You are right that the method and repercussions are very important. Documenting them and explaining them clearly is very important. However, I worried that this discussion would become hard to follow if we were simultaneously discussing the what at the same time as the how. Since the what would likely inform the how, I felt it was better to have this discussion first.

PS: If anyone has strong feelings about this, please comment - this is an open issue and OWG wants to hear your views.

[systemed]

the vast majority of OSM's tiles are rendered to support 3rd party sites and apps

Do we have any (approximate) breakdown[1] of sites vs apps; and any breakdown of how many sites use what % of capacity?

My rough impression as an interested onlooker is that there are three broad categories of user, other than mapping tools:

1. popular sites with in-browser maps, e.g. Pokemon Go (problematic levels of use)
2. apps (very often problematic levels of use)
3. a long tail of tiny sites with in-browser maps (not problematic levels of use)

I'm not overly bothered about 1 or 2, but to cut off 3 would concern me. Having OSM tiles on (say) a small shop's "Where we are" page is good visibility for OSM yet minimal load on the servers, even in aggregate. OSM also has an wider social role in encouraging people to "use [maps] in creative, productive, or unexpected ways", and to prevent small sites from using our maps as soon as they add their first AdSense embed or affiliate link would damage this role.

On a broader issue, I worry that this might have the effect of making the best-funded service companies into gatekeepers for the OSM ecosystem. If long-tail users are immediately required to sign for with a 'starter plan' with a services company, this may reduce their attachment to OSM per se and to the broader ecosystem.

But if I'm way off on the numbers then this is all moot.

(And thank you for soliciting wider contributions! :) )

[1] I ask this knowing how much @zerebubuth enjoys a bit of numerical analysis

[simonpoole]

I would take issue with the underlying hypothesis that OSM does not benefit from use of the OSMF provided services in "closed" environments, I can see no logical reason why this should be the case and would suggest that this be substantiated before changing rules in a way that might effect a larger number of users short term.

Note: if we do change something we should take #114 in to account

[Komzpa]

I think this is really shortsighted proposal. It for sure will get tile usage levels lower, but it will also move lots of people away from openstreetmap at all.

Sending people to go read switch2osm when they're using osm.org tiles sounds like 'go away'. Switch2osm was started as a guide how to use OSM - not as a way to say "don't use osm, go install your own osm". Google's prices aren't that high compared to self-hosted OSM.

Speaking for myself: I'm lazy nowadays. I know how to set up tiles and did that numerous times. I have two options now:

1. go, buy a server, set up tiles, utilize 0.001% and be happy self-customer;
2. go, buy a server, send credentials to operations@, see it used at 90% by all the osmers (including my country, yay!) and be a happy tile.osm.org consumer.
   I went the second way now. This makes me happy - I use a lot of software with hard coded tile.osm.org without that guilt feeling that is being recurrently planted by "tiles are only for osm.org, go and buy mapbox instead" usage policies. How about we just embrace this behavior more?

"Commercial apps can not use the server" - we're using QGIS with QuickMapServices plugin. If we buy commercial support for QGIS, helping its development, we cannot use osm.org tile cluster anymore?

We're using Carto tiles as background for (hundreds of) GPS traces that had issues with routing, and we fix osm.org roads each time we identify an issue with them. Should this be banned, as it is business usage, or should this be embraced, as it is about improving the map, finding rare and obscure issues?

What we lack is transparency, IMHO:

• which sites are using osm.org tiles? (this allows going to them and saying "hey! you're cool guys, wanna donate?")
• what are users' software? (this allows estimations about "99% have webgl, let's go webgl" or "15% are QGIS, let's ping them to re-implement caching and ad about donating")
• what are their usage patterns? (this allows tech investigations, like 'will prefetching like this increase hit rate?')

We're trying to have social enforcements where there are technical solutions. There were rumors about Varnish migration. We're not now bandwidth limited, we're rendering-power limited - how about implementing logic of:

• if request is from osm.org, go same as now;
• if request is from some other place:
  • send cached tile, if there is;
  • do not rerender if not necessary;
  • show "out of capacity, donate: ..." tile if there is no cached tile and no rendering capacity.

[mtmail]

Thank you komzpa for running a tile server.

I believe identifying top websites using osm.org tiles and asking them to donate will send the wrong message. What if a website refuses for one reason or another? Or donates very little? Or donates once and never again in the following years? Would we then switch off access? That model would be seen as "give us money or else". Same with returning tiles "out of capacity, donate".

https://switch2osm.org/ includes instructions how to setup a tile server but also lists tile provider companies. Pointing users to the website doesn't mean one has to run their own tile server.

Not being bandwidth limited currently is lucky as bandwidth (at least for the hosting in London) is donated.

We're using Carto tiles as background for (hundreds of) GPS traces that had issues with routing, and we fix osm.org roads each time we identify an issue with them. Should this be banned, as it is business usage [...]

Isn't that up to CartoDB's tile policy? But you raise a good point here: editors (people fixing the map) should ideally never be restricted.

[matkoniecz]

That only publicly-accessible sites may use the tile server. For example, sites requiring a login or accessible only from behind a firewall or VPN would not be public, and therefore would not be allowed to use the tile server.

This would also ban for example my local file that makes map of places with missing OSM tags (I make this map to find places for mapping) and all kind of things that are in development, even ones intended to be open for public.

for reference - my file, version showing places with bicycle parkings missing in OSM, present according to data released by city Kraków: https://gist.github.com/matkoniecz/bac244f38693f307b3560e4e71bf8e04

[matkoniecz]

Do we have any (approximate) breakdown[1] of sites vs apps; and any breakdown of how many sites use what % of capacity?

This would be highly useful. I suspect that there are some sites/apps/scrappers like Pokemap with very high usage and long tail of tiny/optimized sites with acceptable levels of usage.

[matkoniecz]

The usage of tile OSM tile infrastructure by the OpenStreetMap website was recently measured at approximately 11% of tile server output

Can you give link to that (I am curious how it was done from technical side).

[zerebubuth]

The usage of tile OSM tile infrastructure by the OpenStreetMap website was recently measured at approximately 11% of tile server output

Can you give link to that (I am curious how it was done from technical side).

I'm afraid it was done in the simplest and least repeatable way possible: Taking the tile render server Apache logs and running awk -F '"' '{total += 1; if ($4 ~ /openstreetmap.org/) {osm+=1;}} END {print osm " " total;}' on it. For example, on yesterday's logs on yevaud, it gives 9112308 78726601, which is 11.6% - slightly higher than last time I ran it.

[dbf256]

I've read that this thread is for policy discussion, not for tech details, but just for better understanding I would like to know is it possible to deploy a configuration that will have 2 separate flows. One is for core activites (like mapping editors, tiles at openstreetmap.org etc) with higher quality of service and not for 3rd party usage; and another flow for all others where service is provided on a best-effort basis (if there is a capacity for it)?

I agree with Komzpa that it is better to have a technical solution (if possible) - rather than a legal once. Because in this case we will be able to block users that are a source of an issue (bandwidth usage etc - it could be a non-profit website like fastpokemonmap), rather than that ones that are harmless but are closed-source/commercial, for example.

[zerebubuth]

Do we have any (approximate) breakdown[1] of sites vs apps; and any breakdown of how many sites use what % of capacity?

My rough impression as an interested onlooker is that there are three broad categories of user, other than mapping tools:

1. popular sites with in-browser maps, e.g. Pokemon Go (problematic levels of use)
2. apps (very often problematic levels of use)
3. a long tail of tiny sites with in-browser maps (not problematic levels of use)

I'm not overly bothered about 1 or 2, but to cut off 3 would concern me.

Here's the breakdown based on renderer requests on yevaud for the preceding 24h period - this is cache misses, so will undercount any sites whose tiles are served entirely from cache. But those aren't the ones which cause us a problem anyway.

Hits	Referer domain	%age
34113602	-	43.81%
9141929	*.openstreetmap.org	11.74%
1672876	not a domain	2.15%
1512934	trackmytaxi.com	1.94%
980502	easymap.land.moi.gov.tw	1.26%
30447314	23,995 sites smaller than 1% usage	39.10%

The - entry is for things which didn't send a referer, and I've put a breakdown of those below. The not a domain entry is for referers which are IP addresses or localhost, or things which are not valid URLs.

Unfortunately, the 23,995 sites in the long tail together make up almost 40% of usage, yet are a very long, smooth tail:

Hits	User agent	%age
5167262	Android	6.64%
2586501	osmdroid	3.32%
2560810	-	3.29%
2211556	OruxMaps	2.84%
2136972	CFNetwork	2.74%
1830649	libwww-perl	2.35%
17619852	others	22.62%

Where - means that no user agent header was sent and others represents the sum of all the remaining requests. The percentage is relative to total requests. Similar to the referers, there's a long tail of smaller user agents. 15% of the usage comes from things which are not individually distinguishable, for example the generic Android user agent, -, "CFNetwork" which is the generic iOS user agent, or "libwww-perl", the generic user agent for scrapers written in Perl.

1. a long tail of tiny sites with in-browser maps (not problematic levels of use)

I'm not overly bothered about 1 or 2, but to cut off 3 would concern me.

The sum of the "long tails" for user agents and referers is 61.72%, which means that almost 2/3rds of our requests are in your 3rd category. When taken individually, they are not problematic levels of use. As a whole, they do cause a problem.

I think everyone would agree that we want to support mapping activity. The requirements for supporting this are clear; we want quick updates and lots of detail.

The problems start to appear when we try to support generic web mapping use-cases, which have a different set of requirements; they are not so bothered about updates, and favour "less cluttered" maps over more detailed ones. In fact, it's better for the generic web mapping use case to update less frequently, as this means better cache hit ratios and faster load times. "Less cluttered" map tiles tend to be smaller, making them faster to load and making better use of edge and device caches.

It is possible support all of this, with more hardware and more donations. However, using discretionary spending (and to a lesser extent, donated hardware) to support generic web mapping means we have fewer resources to support other OSMF-run services, such as the API, planet, Nominatim, etc...