Kiezer struggling

[gravitystorm]

Similar to #284, tiles service here has been very slow today. It seems to be a problem with keizer

The network graph shows drops in the expected diurnal cycle of traffic:

The squid times shows median service times measured in multiple seconds

The CPU chart shows the machine is pretty busy, and that something is stealing from the (presumably virtualised) CPUs.

[tomhughes]

Sadly we don't have the resources to manage these issues - we have far more caches that we are reasonably able to monitor or manage and much of their performance is outside our control anyway.

[tomhughes]

Also because keizer is just one quarter of the German cache we have no way to control the load going to it on an individual basis. I believe kalessin is also getting too much load but the same problem applies.

It's not really something we can solve without a complete redesign of the cache system...

[tomhughes]

Sorry @gravitystorm I wasn't necessarily trying to say we should close this... I'm just a bit jaded trying to keep the cache show on the road and finding that every time you fix one machine it moves the problem somewhere else :-(

I'm a bit surprised Poland is on the German cache as I thought it pretty much only did Germany so we could probably move Poland off it.

Germany is always a problem due to the massive load and fact that the "cache" is actually four machines of rather different capabilities being round robined via DNS.

[tomhughes]

Hopefully openstreetmap/dns@cef3ef9 will help.

[gravitystorm]

Sorry @gravitystorm I wasn't necessarily trying to say we should close this... I'm just a bit jaded trying to keep the cache show on the road and finding that every time you fix one machine it moves the problem somewhere else :-(

Oh, I can understand the jadedness, I just felt that having this specific issue open was therefore not going to help solve any of the wider problems. If it was a specific problem with the specific server, then that would be different, but since it's just part of the wider tile serving situation then I was happy just to close it.

Thanks for looking into the situation though!

[AndreSchwarzer]

@tomhughes && @gravitystorm Would another server on the same or different provider help?

BTW I'm the sponsor of keizer 😄 👨‍💻

[tomhughes]

Tricky... Germany is a bit of a nightmare, being split across four servers of differing capacities :-(

[AndreSchwarzer]

So a "clone" (same capacities) of keizer on cloud.hetzner.de too ?

So it'll be 5 servers of 4 settings :P

[Firefishy]

@AndreSchwarzer yes, another server would be greatly appreciated. Best is via email operations AT osmfoundation. Org

[AndreSchwarzer]

I am resending my mail from August 08 2019

[AndreSchwarzer]

You should have mail @Firefishy