ysera load is jumping up/down

[pnorman]

I considered that this might be a sampling artifact where conversion to a rate results in oddities, but the same is present on the CPU graphs

[pnorman]

It's paired with culebre, which is showing normal load.

Relevant VCL

director european_servers_0 random {
  { .backend = F_Odin___Amsterdam; .weight = 10; }
  { .backend = F_Nidhogg___Umea; .weight = 30; }
}
director european_servers_1 random {
  { .backend = F_Ysera___Slough; .weight = 10; }
  { .backend = F_Culebre___Dublin; .weight = 30; }
}

declare local var.x_mod INTEGER;

if (server.region == "APAC" || server.region == "Asia" || server.region ~ "^Asia-") {
  set req.backend = australia;
} else if ((server.region == "North-America" || server.region ~ "^(US|SA)-")
&& server.datacenter != "IAD" && server.datacenter != "EWR" && server.datacenter != "LGA" && server.datacenter != "YYZ" && server.datacenter != "MIA" && server.datacenter != "DFW" ) {
  set req.backend = america;
} else {
  if (req.url.path ~ "^/([1-9]?[0-9]+)/([0-9]+)/([0-9]+)\.png") {
    // Compute the X metatile and split into two groups
    set var.x_mod = std.atoi(re.group.2);
    set var.x_mod /= 8;
    set var.x_mod %= 2;
    
    // Assign to different servers based on groups
    if (var.x_mod == 0 ) {
      set req.backend = europe_0;
    } else {
      set req.backend = europe_1;
    }
  } else {
    set req.backend = europe_0;
  }
}

[pnorman]

Turned off ysera at the CDN. No major impact on traffic. odin and nidhogg are spiking. Suspecting that there's someone using the tile server and bypassing the CDN.

[pnorman]

Restored the CDN to config it was over the weekend, but it's not changed anything.

Odin is badly off. Started at about 7 AM UTC

[pnorman]

Contacted Fastly support. Unsure what's going on.

[pnorman]

Graph of

SELECT COUNT(*), time FROM logs.fastly_logs_v18
WHERE year = 2022 AND month = 07 AND day = 18 AND hour = 08 AND render='odin.openstreetmap.org' and cachehit='MISS'
GROUP BY time ORDER BY time;

[pnorman]

Removed odin from all_servers and european_servers_all groups to see if it has any impact.

[pnorman]

Removed odin from all_servers and european_servers_all groups to see if it has any impact.

No impact, as expected. Now removing odin completely from the CDN

[pnorman]

Removing it from the CDN removes all load, as expected. Bringing it back in with minimal weight.

[pnorman]

Bringing it back with a weight of 1 has brought the traffic up to the same level it had when it had a weight of 10

[pnorman]

I'm wondering if it's healthcheck related somehow. if nidhogg failed its healthcheck it would divert all of its traffic to odin, regardless of weighting.

Trying an adjustment by making the healthcheck only want 1/10 of checks to pass, with checks ever 5s.

[pnorman]

I changed the health check to 1/10 threshold & window, 1 initial, 5000s interval and it's changed the behavior.

The behavior is still not correct, there should be a 1:30 ratio between odin traffic and nidhogg traffic, but the fact that it changed indicates some connection with health checks.

[pnorman]

On Fastly's suggestion, set up https://support.fastly.com/hc/en-us/community/posts/360040444872-Monitoring-health-status-from-fastly-health-checks#:~:text=Regarding%20this%20topic%2C%20yes%2C%20we%27re%20aware%20of%20your%20needs%20as%20this%20is%20one%20of%20the%20frequently%20asked%20questions.

[pnorman]

Confirmed that servers were failing health checks - at least pyrene, nidhogg, and ysera.

A cluster-wide apache restart has been completed, and that appears to have fixed it.

[pnorman]

A few things to investigate

What was wrong that an Apache restart fixed?

The error [Mon Jul 18 09:38:37.894876 2022] [mpm_event:error] [pid 2239:tid 140189126043520] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. was observed

What happened to Pyrene?

Pyrene stopped responding to all traffic at 2022-07-17T00:03:15Z. This is suspiciously close to midnight UTC.

What monitoring do we need?

We had no monitoring that indicated that health checks were failing, and when I loaded tiles manually, everything worked, so it wasn't immediately obvious what went wrong.

What alarms do we need?

Probably some apache ones, but perhaps also on monitoring of health checks

[pnorman]

Failover from US to Europe doesn't work as intended. america is defined as a fallback from pyrene to all_servers, which will put US traffic on the wrong even/odd servers. The same is true for Australia.

Each region needs its own even/odd split defined, even if the even/odd servers are the same. They can then fall back to the right servers in Europe

[pnorman]

The tile service experienced degraded service from 2022-07-18T07:30:00Z to approximately 2022-07-18T10:00:00Z, impacting an average of 4% of traffic during the event.

See graph of sum(rate(fastly_rt_status_code_total{service_name=~"OSM Tiles CDN",status_code=~"404|5.."}[$__rate_interval]))/sum(rate(fastly_rt_requests_total{service_name=~"OSM Tiles CDN"}[$__rate_interval]))

[pnorman]

Published in openstreetmap/owg-website@6fd72e7