We're exceeding our papertrail plan's log message limit.

[eddierubeiz]

Something has changed in the past few weeks that has caused us to frequently reach the max number of log messages per day.

This manifests in lines like:
Error L10 (output buffer overflow): 4 messages dropped since 2023-12-26T00:16:42.862947+00:00.
https://my.papertrailapp.com/groups/22231012/events?q=Error+L10

Figure out what the problem is, and fix it.

[jrochkind]

Sorry @eddierubeiz , but I believe that's circular and the outcome rather than the cause -- I think those lines ARE what happens when we exceed our papertrial plan, they are the EFFECT of exceeding our papertrail plan, they are what gets logged when papertrail is full.

[jrochkind]

We can exclude those lines from papertrail though, so papertrail filters them out and does not include them in it's logs.

That could keep the problem from making itself worse in a circular fashion, and making it harder to identify what's really causing it. But would not eliminate the actual original problem of our papertrail plan being exceeded.

[eddierubeiz]

@jrochkind notes:

Note a graphic of recent size of logs is available in papertrail, and ALL our past logs are avail (via papertrail) on S3. So the question I have is, why have our logs gotten bigger recently than historical trends (compare to a past log file that was smaller a couple weeks ago or whatever)?

Is it just more traffic than we used to get? Or was some kind of log line showing up recently that didn’t used to show up?

And I guess the next question: is there anything we can exclude from logs to keep size reliably down , or do we just need a bigger papertrail plan?

[eddierubeiz]

L10 messages indicate Heroku's log drains can't handle incoming logs. I believe this can be due to a) a burst of incoming log messages (thousands per second) that momentarily overwhelms the log drain or b) Papertrail simply refusing to accept any more logs, because we've reached our daily limit. Otherwise stated, L10s often co-occur with us running out of log space a the end of the day, but don't always.
See a more structured discussion in the wiki.

[eddierubeiz]

We've decided to pay for more space (550MB) for a few weeks, to see whether that's any help in practice. See issue 2490 (linked above) which covers evaluating this decision based on a couple months' data. I do not believe this will cure those of our L10 errors that are due to a burst of bot activity, but on days where we get e.g. 400MB of logs, we should be able to keep and examine those logs.

[eddierubeiz]

As of April, switching to a more expensive plan did help.