Edge Gateway CF Worker system errors

[vasco-santos]

We have been seeing CF errors in the wild Network Connection Lost, together with some sporadic KV GET failed: 403 Forbidden + KV GET failed: 501 Not Implemented (when we use the KV get for every request).

Some details of one of the requests: worker_id_string _5IO2WMTFx + cf_ray_string 6ff44daab9ee5a55

Per convos in Cloudflare Developers Discord, we are trying a temporary solution recommended at https://discord.com/channels/595317990191398933/779390076219686943/967047749881192468 to work around these errors #50.

We should look into the root issue with CF Workers team

[olizilla]

@vasco-santos can you quote the recommended solution here in the issue please.

[vasco-santos]

@vasco-santos can you quote the recommended solution here in the issue please.

Not really a fancy solution:

We had a single get on KV namespace for each request. We hve millions of requests each day worldwide . So we had to implement a recursive fail-safe pattern

We did this, looks like it helped but with ~15M req/24H we start hitting these issues in CF. We need to talk with them and try go get this fixed at the root, in the meantime a temporary solution could be replicating the KV and have like 3 different KV with the same content. Then, randomly choose one of them per request to check Denylist. Cron Job would need to update content on the 3 of them.

[vasco-santos]

There is another angle we can go here. We do this check before going into cache as we can be caching a bad response.

Considering cache hit is really high, we can reduce a lot the number of requests to the KV if we turn things around and go to cache first. We really just need to remove things from cache when they are added to KV https://developers.cloudflare.com/workers/runtime-apis/cache/#delete.

For this, we would need a protected (JWT token) route in the worker gateway (or API) /cache/delete and automate this call when we update KV.

Thoughts @olizilla ?

[olizilla]

Yes that seems reasonable. If it's in the cache, let's respond ASAP! If it's bad, clear it out of the cache!

[vasco-santos]

An update here: Cloudflare KV seems to have scaled and we have not seen these errors for a few weeks now, even with larger scale on our side. Keeping the retry for now does not hurt though.

Even though, I think we should track work for the cache optimization described above. I created an issue for this in the new appropriate repo web3-storage/reads#44