"This website is under heavy load (queue full)"

[matkoniecz]

We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.

appearing on openstreetmap.org (also reported by others)

[tomhughes]

Fixed.

[Firefishy]

Yes, we have been having issues over the last 2 days. We have been trying to get the planetdump process running again, but app is leaking memory.

[matkoniecz]

So planet dump become buggy and drags down everything with it? (I am quite curious, if you have more interesting things to do then feel free to ignore that question)

EDIT: yes, I want to know. I am interested in such things. And given current resources (including sysadmin time) it is wonderful that OSM site and servers are usually working.

[tomhughes]

You really don't want to know. And yes we know it's bad and we have a plan to fix it but there are things we need first.

[Firefishy]

@matkoniecz Yes, the planet dumps currently run on the storage server ironbelly. Ironbelly is also the NFS server for web servers too. The planetdump-ng v1.1.7 was meant to fix a memory leak issue, but unfortunately didn't. NFS ran slow, web rails processes locked while waiting for NFS, which eventually starved web server of works and caused the queue issue.

What we doing to fix?

1. We'll get planetdump-ng fixed.
2. Ops team have discussed splitting up ironbelly.
3. User images now use S3 as storage backend, once Rail 6.1 is available we plan to add code to move GPX data to S3, and remove the requirement for the NFS server.