Logs: let us filter, sample or turn off API gateway (edge_logs) ingest

[abdatta]

We run a small restaurant point-of-sale app on the Free plan: a few tablets in one shop. Under the new Logs Ingest quota we're on course for about 2 GB a month against 1 GB included. Everything else we use is far inside the free limits.

We measured where it comes from: about 96% is edge_logs, one ~2.8 KB line per API request (about 30,000 requests a day). Our Postgres settings are already at their quietest defaults, so the levers in the Manage Logs Ingest guide don't apply. Support confirmed there is no setting today that lowers API-gateway log volume. We've already cut our own requests by about 30%, and we're still over.

We read these logs only to investigate an incident, and would happily trade detail for volume. Any of these would solve it:

1. A per-project setting to turn API-gateway logs off.
2. Or to filter them: keep errors (4xx/5xx) and slow requests, sample 1 in N successful ones, and skip CORS preflight (OPTIONS) requests, which are ~9% of ours.
3. On the Free plan, stop logging once the quota is reached instead of applying Fair Use restrictions to the project. Losing logs for the rest of the month is fine; a read-only database or 402s on a point-of-sale is not.

Related: #50925 (the pricing announcement, where another team reports Postgres is under 1% of their volume) and #51306 (pooler health-check logs filling a Free plan's quota).

[kemaldotearth]

Hi @abdatta, thanks for posting your experience with logs. Appreciate the work you've done to reduce your ingest by 30% and hopefully we're on the right path to deliver something that reduces this problem, with less effort in the future.

We've put together a mock of what this could look like in the future. The data in the video is dummy data for demo purposes. Similar to your first idea, you would get a per-project setting to manage your logs ingest. In this mock, we do not keep historical context of your ingest if you are switched "off" or set to "min".

While we address feasibility, it would be great to get your thoughts on:

• As you're reading logs particularly during incidents, does "raise the level when needed, no history" suffice? or would you expect historical logs when bumping up the level of debug mode?
• Would an "off", "min" and "max" suffice? or do you require more granular filters by source?
• For your third point regarding free plan, would it work for you to stop ingest while the project runs as usual?
• Anything we might have missed in this mock?

Look forward to hearing your thoughts!

debug-mode-public-demo-converted.mp4

[abdatta]

Hi @kemaldotearth, thanks. The mock is very close to what we'd use. Answers:

1. Raise when needed, no history is enough for us, as long as min keeps errors. The one real outage we diagnosed from these logs (a polling bug that froze our counter) we found hours later from gateway 5xx, latency and Postgres statement timeouts. Those are errors, so min would have kept them. A slow-request threshold at min would have caught the rest.
2. Off / min / max is enough if min means 4xx/5xx plus slow requests, and CORS preflight (OPTIONS) is dropped below max. Per-source levels (say, gateway at min, Postgres and Auth at default) would be nice, not essential.
3. Yes. Stopping ingest at the quota while the project keeps running normally is exactly what we want on Free, so long as going over never pauses the project, makes it read-only or returns 402s.
4. Two small additions: ingest per day per source on the usage page, so we can see what a change did; and the setting on Free before enforcement starts.

Thanks for picking this up.

[Fab-Ver]

Hi @kemaldotearth, thanks for the mock, and +1 to @abdatta 's answers above. Some feedback from a different workload: a live-event backend, not an always-on app.

Our situation. The bulk of our log ingest is edge_logs, roughly 2 KB per request. Our own backend calls PostgREST with service_role many times per event, so every successful read is logged and we can't make the clients send fewer requests. Traffic comes in bursts: a few days of heavy use, then near-silence for weeks.

On your questions:

1. No history is fine, provided "min" keeps 4xx/5xx and slow requests. Please document exactly what each level keeps. We need to know what we'd still see during an incident.
2. Off/min/max is enough for us. Per-source levels are not essential. Sampling successful 2xx requests (1 in N) would be a useful middle level between "min" and "max".
3. Stopping ingest on Free is fine, as long as the project keeps running normally: no read-only mode, no 402s, no pause. Please also send a warning before a project reaches the quota.

[kemaldotearth]

Thanks for the feedback @abdatta and @Fab-Ver! This is really helpful for us. Some common takeaways I'm summarising:

• min needs to keep 4xx/5xx and slow reqs with exact behaviour of each debug level documented.
• CORS preflight dropping below max.
• Sampling of successful 2xx requests as a possible middle level between min and max.
• On free plans, stopping ingest at quota must never pause the project, make it read-only or return 402's.
  • Ideally a warning before quota is reached.
• Per-day/source ingest on the usage page.
• If I understand right, something like a debug mode would be appreciated before free plan quota enforcement kicks in.

We don't currently have a timeline of when a feature like this will drop, but this gives us plenty to mull over. Keep the feedback coming here if you think of anything else!

[Support-SDL]

This demo looks good, and agree with the suggestions made. I think that would solve our usage case.

@kemaldotearth Will the grace period be extended until these fixes are in place so we aren't charged a huge bill?

As it stands, in a billing period, we would incur about a $50 per month charge extra for log ingestion on top of our paid plan.

[smeubank]

We are still discussing the exact date of enforcement, and we are exactly considering how to make this the best experience possible for folks.

There is world even with controls, that people will still have increase bills. And we want to make sure the grace period is long enough that people have time to react. THis is also why we have release the meters in the UI before we enforced actual billing.

We understand your concerns and will release more info when we have it! :)