New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve limits for tile speeds #97

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
5 participants
@Komzpa

Komzpa commented Oct 10, 2016

This pull request raises per-second limits for tile usages to comfortable values, and chops off bursts to disallow scraping.

Komzpa added some commits Oct 10, 2016

update squid traffic limits for comfortable tile browsing
300 kb/s per user (~1 full screen update per second) after first 15mb (~ 32 map screens with 32 tile/screen and 15 kb tiles);
raised limit per subnet to 128 times more than per user - should never be hit in practice.
Raise HTTPS limit
on a typical connection, with this change all tiles on screen should be loaded within first 0.25s.
@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Oct 10, 2016

Member

So to summarise this seems to make the per-network buckets 15 times bigger and makes them refill 4800 times faster and makes the per-IP buckets much smaller (about one quarter the size) but makes them refill 75 times faster.

So unless I am misunderstanding something this is going to vastly increase the amount of scraping that people can do and I can't conceive of any way we could accept it.

I'll leave the decision to @Firefishy though as he's the expert.

Member

tomhughes commented Oct 10, 2016

So to summarise this seems to make the per-network buckets 15 times bigger and makes them refill 4800 times faster and makes the per-IP buckets much smaller (about one quarter the size) but makes them refill 75 times faster.

So unless I am misunderstanding something this is going to vastly increase the amount of scraping that people can do and I can't conceive of any way we could accept it.

I'll leave the decision to @Firefishy though as he's the expert.

@Firefishy

This comment has been minimized.

Show comment
Hide comment
@Firefishy

Firefishy Oct 10, 2016

Contributor

I don't think we could handle a traffic jump of this size, but I would be interested to see the effect. Maybe run it for a few hours during the day?

Contributor

Firefishy commented Oct 10, 2016

I don't think we could handle a traffic jump of this size, but I would be interested to see the effect. Maybe run it for a few hours during the day?

@Komzpa

This comment has been minimized.

Show comment
Hide comment
@Komzpa

Komzpa Oct 10, 2016

@tomhughes what are exact concerns about this numbers?

at very least they have math behind them that goes from "a user with browser shouldn't get banned and should have a sane QoS regardless of session time and monitor size".

A single user usually has one IP that they can only change by reconnecting or changing the proxy. This strategy should be rendered useless by lowering bucket sizes, so that reconnecting doesn't bring speed back after using it up, for long at least (15mb isn't very long).

300 kb/s is still slow enough to download the city completely (usual scraping strategy?) but fast enough to browse the map.

These values are what we should aim for. Lower settings will get tiles slow for the legitimate users, we're already at point when public wifi and dense provider networks get throttled because of some neighbours.

Komzpa commented Oct 10, 2016

@tomhughes what are exact concerns about this numbers?

at very least they have math behind them that goes from "a user with browser shouldn't get banned and should have a sane QoS regardless of session time and monitor size".

A single user usually has one IP that they can only change by reconnecting or changing the proxy. This strategy should be rendered useless by lowering bucket sizes, so that reconnecting doesn't bring speed back after using it up, for long at least (15mb isn't very long).

300 kb/s is still slow enough to download the city completely (usual scraping strategy?) but fast enough to browse the map.

These values are what we should aim for. Lower settings will get tiles slow for the legitimate users, we're already at point when public wifi and dense provider networks get throttled because of some neighbours.

@Zverik

This comment has been minimized.

Show comment
Hide comment
@Zverik

Zverik Oct 10, 2016

I doubt it will lead to a significant traffic jump. I propose to merge it, see how much the load is, and if it is too big, rvert the change.

Also, to deal with scrapers, shouldn't we limit the number of renderd requests, not http requests? Can we distinguish requests for cached tiles from not rendered tiles?

Zverik commented Oct 10, 2016

I doubt it will lead to a significant traffic jump. I propose to merge it, see how much the load is, and if it is too big, rvert the change.

Also, to deal with scrapers, shouldn't we limit the number of renderd requests, not http requests? Can we distinguish requests for cached tiles from not rendered tiles?

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Oct 10, 2016

Member

Well I'm not sure of the details of exactly when the two different buckets are used which is why I deferred to @Firefishy as he understands the details. All I did was look at the massive jump in the numbers and think that is smelled fishy.

If you have some calculations that explain how the current limits work and how your proposed new ones compare then please do share them with us.

Member

tomhughes commented Oct 10, 2016

Well I'm not sure of the details of exactly when the two different buckets are used which is why I deferred to @Firefishy as he understands the details. All I did was look at the massive jump in the numbers and think that is smelled fishy.

If you have some calculations that explain how the current limits work and how your proposed new ones compare then please do share them with us.

@Komzpa

This comment has been minimized.

Show comment
Hide comment
@Komzpa

Komzpa Oct 10, 2016

@tomhughes for the record, a quote from my letter to operations@osmfoundation.org from Oct, 2:

On limits:

This time, it looks to me that Squid throttling is configured extremely not the best way.
Looking at code here:
https://github.com/openstreetmap/chef/blob/master/cookbooks/tilecache/templates/default/squid.conf.erb#L159 

and reading through Squid documentation suggests there are the following limits:
1) per first N1=7 bits of address (loop-generated);
2) per third N2=8 bits of address (the class 3 buckets);
3) per last N3=8 bits (host) within third 8 bits of address (the class 3 buckets).

The issue I see is with network bucket sizes. 

If we imagine a fully-saturated system, with tiles being downloaded from every subnet, we'll 
see outgoing bandwidth of 
2**(N1+N2)* net_bucket_refill = 2**(7+8)*8192 = 268435456 = 256 mb/s, or 2 Gbit/s.
So, for that case (over-saturated single host system) this setup is probably okay.

Every client then is limited, if the whole subnet is saturated, to net_bucket_refill / 256 = 8 bytes per 
second. In reality it could be better, but in any case not more than up to 4 kb/s, which reminds me of 
dial-up times, and is roughly 1 tile per second. There are ~30 tiles on single map screen on my monitor 
- this means 30 seconds to redraw it after zoom-in.

Issues are:
 - OSM's tilecache cluster bandwidth is not over-saturated;
 - we have geographical load balancing based on country. This means all clients from the same ISP 
usually go to the same server, and they also traditionally have same /24 subnet.

What I see as more appropriate settings:
 - raise ip_bucket_refill to [tile_size]*[tiles_on_sceen] = 15kb*30 = 307200;
 - lower ip_bucket_size to something that encourages only online usage, without preloading - 
say 5mb = 5242880;
 - raise net_bucket_size and net_bucket_refill to the skies, to at least 30 x ip_* - users are already
 segregated by geodns, I don't see the actual need to duplicate this effort with another restriction.

This way we allow continuous workflows and disallow tilescrapers in the current reality.

Komzpa commented Oct 10, 2016

@tomhughes for the record, a quote from my letter to operations@osmfoundation.org from Oct, 2:

On limits:

This time, it looks to me that Squid throttling is configured extremely not the best way.
Looking at code here:
https://github.com/openstreetmap/chef/blob/master/cookbooks/tilecache/templates/default/squid.conf.erb#L159 

and reading through Squid documentation suggests there are the following limits:
1) per first N1=7 bits of address (loop-generated);
2) per third N2=8 bits of address (the class 3 buckets);
3) per last N3=8 bits (host) within third 8 bits of address (the class 3 buckets).

The issue I see is with network bucket sizes. 

If we imagine a fully-saturated system, with tiles being downloaded from every subnet, we'll 
see outgoing bandwidth of 
2**(N1+N2)* net_bucket_refill = 2**(7+8)*8192 = 268435456 = 256 mb/s, or 2 Gbit/s.
So, for that case (over-saturated single host system) this setup is probably okay.

Every client then is limited, if the whole subnet is saturated, to net_bucket_refill / 256 = 8 bytes per 
second. In reality it could be better, but in any case not more than up to 4 kb/s, which reminds me of 
dial-up times, and is roughly 1 tile per second. There are ~30 tiles on single map screen on my monitor 
- this means 30 seconds to redraw it after zoom-in.

Issues are:
 - OSM's tilecache cluster bandwidth is not over-saturated;
 - we have geographical load balancing based on country. This means all clients from the same ISP 
usually go to the same server, and they also traditionally have same /24 subnet.

What I see as more appropriate settings:
 - raise ip_bucket_refill to [tile_size]*[tiles_on_sceen] = 15kb*30 = 307200;
 - lower ip_bucket_size to something that encourages only online usage, without preloading - 
say 5mb = 5242880;
 - raise net_bucket_size and net_bucket_refill to the skies, to at least 30 x ip_* - users are already
 segregated by geodns, I don't see the actual need to duplicate this effort with another restriction.

This way we allow continuous workflows and disallow tilescrapers in the current reality.
@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Oct 10, 2016

Member

I note that you comment that "OSM's tilecache cluster bandwidth is not over-saturated" which is probably true, but these rate limits are designed to protect the system as a whole not just to control the outgoing bandwidth from the caches.

In fact the biggest problem, especially with scrapers, is the load on the upstream renderers, and controlling access to the caches acts as a proxy for controlling access to the render nodes.

Several times this summer we had to act on an emergency basis to control abusive use of the tile serving system so there is very real issue here and the controls we have in place were not put in place just to annoy people.

Member

tomhughes commented Oct 10, 2016

I note that you comment that "OSM's tilecache cluster bandwidth is not over-saturated" which is probably true, but these rate limits are designed to protect the system as a whole not just to control the outgoing bandwidth from the caches.

In fact the biggest problem, especially with scrapers, is the load on the upstream renderers, and controlling access to the caches acts as a proxy for controlling access to the render nodes.

Several times this summer we had to act on an emergency basis to control abusive use of the tile serving system so there is very real issue here and the controls we have in place were not put in place just to annoy people.

@Komzpa

This comment has been minimized.

Show comment
Hide comment
@Komzpa

Komzpa Oct 10, 2016

@tomhughes these settings weren't touched since Jul 2014.
Change commit has no comment on why it was changed - pool was set 4x times more, and bucket refill was halved:
Komzpa@bd08729

Commit contains no explanation on why it was done this way. It looks like it was done as a response to slow loading tiles after some time, but was resolved in wrong way - adding more unlimited traffic and chopping off traffic right after it was used up.

Komzpa commented Oct 10, 2016

@tomhughes these settings weren't touched since Jul 2014.
Change commit has no comment on why it was changed - pool was set 4x times more, and bucket refill was halved:
Komzpa@bd08729

Commit contains no explanation on why it was done this way. It looks like it was done as a response to slow loading tiles after some time, but was resolved in wrong way - adding more unlimited traffic and chopping off traffic right after it was used up.

@zerebubuth

This comment has been minimized.

Show comment
Hide comment
@zerebubuth

zerebubuth Oct 11, 2016

Contributor

The squid delay pools are applied only to cache misses. Items served from cache do not count towards the delay pool and are always served at full speed, unless completely blocked for some other reason.

If we aim to supply uncached tiles at a rate of 30 per second per client then we would need to size the rendering cluster appropriately as well. To estimate this, we need an approximate measure of the number of concurrent clients, which is difficult if we assume that many of them are being delayed. Looking at the peak number of [established connections to the tile caches](http://munin.osm.org/network-day.html#Connections through firewall) suggests that 177,360 concurrent connections may occur, across 3 hostnames and 6 connections per browser per host gives an estimate of 9,853 concurrent users. Of those, 72% are hitting the cache, so about 2,760 are missing.

Assuming we want those missing the cache to be able to do 30 tiles per second, we'd need to be able to serve 82,768 tiles per second from the render machines (cache miss doesn't always mean render). We currently serve 2,300 per second fresh from disk, so that would need to increase by a factor of 36x.

If we assume that we want people to be able to see freshly rendered tiles, then even if they perfectly fit within metatiles - the best case scenario, we'd need to be able to render 1,293 metatiles per second, where we currently peak at 30. So we'd need to increase capacity by 43x.

It should be noted that these are increases in capacity, which doesn't necessarily mean increases in the number of machines. Indeed, acquiring another 129 render machines might be more difficult than achieving the same increase in capacity by improving the efficiency of the processes, software and hardware that we already have available. Something that might improve efficiency could be to move rendering to the edge by using vector tiles. It would be fantastic to have a PR so that we could start checking what the size of that improvement could be. There's a lot of work to prepare not just the software stack but also the style, but I know a lot of people are excited about the possibilities.

Until that point, requests for non-cached and non-rendered tiles will be limited - whether we control that by our delay pool policy or let it happen "naturally" by renderd dropping requests when it gets overloaded.

One data point that we have is that the recent Fastpokemap issues #78 is that a +13% increase in traffic is enough to seriously degrade service to a level where there were many complaints, and a large proportion of contributors' edits were not being rendered in a timely manner. Best case, we could assume we were at 78% capacity and we've since had a very generous donation by @Komzpa of a +50% capacity increase (assuming perfect work split, which isn't totally true). Which would mean that we should initially consider raising limits by a factor of 1.9 at the most to avoid a repeat of such overload.

Contributor

zerebubuth commented Oct 11, 2016

The squid delay pools are applied only to cache misses. Items served from cache do not count towards the delay pool and are always served at full speed, unless completely blocked for some other reason.

If we aim to supply uncached tiles at a rate of 30 per second per client then we would need to size the rendering cluster appropriately as well. To estimate this, we need an approximate measure of the number of concurrent clients, which is difficult if we assume that many of them are being delayed. Looking at the peak number of [established connections to the tile caches](http://munin.osm.org/network-day.html#Connections through firewall) suggests that 177,360 concurrent connections may occur, across 3 hostnames and 6 connections per browser per host gives an estimate of 9,853 concurrent users. Of those, 72% are hitting the cache, so about 2,760 are missing.

Assuming we want those missing the cache to be able to do 30 tiles per second, we'd need to be able to serve 82,768 tiles per second from the render machines (cache miss doesn't always mean render). We currently serve 2,300 per second fresh from disk, so that would need to increase by a factor of 36x.

If we assume that we want people to be able to see freshly rendered tiles, then even if they perfectly fit within metatiles - the best case scenario, we'd need to be able to render 1,293 metatiles per second, where we currently peak at 30. So we'd need to increase capacity by 43x.

It should be noted that these are increases in capacity, which doesn't necessarily mean increases in the number of machines. Indeed, acquiring another 129 render machines might be more difficult than achieving the same increase in capacity by improving the efficiency of the processes, software and hardware that we already have available. Something that might improve efficiency could be to move rendering to the edge by using vector tiles. It would be fantastic to have a PR so that we could start checking what the size of that improvement could be. There's a lot of work to prepare not just the software stack but also the style, but I know a lot of people are excited about the possibilities.

Until that point, requests for non-cached and non-rendered tiles will be limited - whether we control that by our delay pool policy or let it happen "naturally" by renderd dropping requests when it gets overloaded.

One data point that we have is that the recent Fastpokemap issues #78 is that a +13% increase in traffic is enough to seriously degrade service to a level where there were many complaints, and a large proportion of contributors' edits were not being rendered in a timely manner. Best case, we could assume we were at 78% capacity and we've since had a very generous donation by @Komzpa of a +50% capacity increase (assuming perfect work split, which isn't totally true). Which would mean that we should initially consider raising limits by a factor of 1.9 at the most to avoid a repeat of such overload.

@Komzpa

This comment has been minimized.

Show comment
Hide comment
@Komzpa

Komzpa Oct 25, 2016

@zerebubuth
To get 129 rendering nodes - updated https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN to contain information on fact that rendering nodes are also required.

Vector tiles are bright future, but there should also be big shortcuts before we switch to totally different architecture.

Can we make sure that a hit on tile of metatile gets whole metatile prefetched to delivery node? This way limits are exhausted at rate of 1/64th, with same aount of render calls and disk seeks.

Komzpa commented Oct 25, 2016

@zerebubuth
To get 129 rendering nodes - updated https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN to contain information on fact that rendering nodes are also required.

Vector tiles are bright future, but there should also be big shortcuts before we switch to totally different architecture.

Can we make sure that a hit on tile of metatile gets whole metatile prefetched to delivery node? This way limits are exhausted at rate of 1/64th, with same aount of render calls and disk seeks.

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Oct 25, 2016

Member

If you have some magic way of achieving that then I'm sure we could consider it, though I don't see how it factors into the rate limiting which is applied when delivering tiles from the cache, not when receiving them from the render node.

Member

tomhughes commented Oct 25, 2016

If you have some magic way of achieving that then I'm sure we could consider it, though I don't see how it factors into the rate limiting which is applied when delivering tiles from the cache, not when receiving them from the render node.

@zerebubuth

This comment has been minimized.

Show comment
Hide comment
@zerebubuth

zerebubuth Oct 25, 2016

Contributor

I don't see how it factors into the rate limiting which is applied when delivering tiles from the cache, not when receiving them from the render node.

Rate limiting is applied when delivering tiles from upstream, but not when delivering them directly from the node's cache.

If it were possible to pre-fetch the metatile into cache then it would potentially mean additional future requests wouldn't be rate limited. Although that's obviously open to a race condition where someone opens several connections, and is rate limited on all of them, whereas doing the requests in serial over a single connection would potentially avoid rate limiting on all but the first request.

I'm not aware of any way to do this using squid. I know that an effort is underway to use varnish instead, which is more configurable using its VCL language. I'm not sure if that would extend to pre-fetching.

To get 129 rendering nodes - updated https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN to contain information on fact that rendering nodes are also required.

The information on that wiki page says 32GB minimum RAM, which is less than half of any current tile rendering machine. I based the 129 extra estimate on the current rendering rate, which is using 3 machines with an "average RAM" each of 98GB. Scaling isn't linear, but such a huge difference in RAM would suggest we'd need significantly more than 129 machines if many of them have <64GB.

Contributor

zerebubuth commented Oct 25, 2016

I don't see how it factors into the rate limiting which is applied when delivering tiles from the cache, not when receiving them from the render node.

Rate limiting is applied when delivering tiles from upstream, but not when delivering them directly from the node's cache.

If it were possible to pre-fetch the metatile into cache then it would potentially mean additional future requests wouldn't be rate limited. Although that's obviously open to a race condition where someone opens several connections, and is rate limited on all of them, whereas doing the requests in serial over a single connection would potentially avoid rate limiting on all but the first request.

I'm not aware of any way to do this using squid. I know that an effort is underway to use varnish instead, which is more configurable using its VCL language. I'm not sure if that would extend to pre-fetching.

To get 129 rendering nodes - updated https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN to contain information on fact that rendering nodes are also required.

The information on that wiki page says 32GB minimum RAM, which is less than half of any current tile rendering machine. I based the 129 extra estimate on the current rendering rate, which is using 3 machines with an "average RAM" each of 98GB. Scaling isn't linear, but such a huge difference in RAM would suggest we'd need significantly more than 129 machines if many of them have <64GB.

@Komzpa

This comment has been minimized.

Show comment
Hide comment
@Komzpa

Komzpa Oct 26, 2016

@zerebubuth @Firefishy where can these Varnish efforts be found?

Komzpa commented Oct 26, 2016

@zerebubuth @Firefishy where can these Varnish efforts be found?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment