Skip to content

Reduce SQLite contention from OpenTelemetry SDK debug logs#26396

Merged
zanie-oai merged 4 commits into
mainfrom
zb/reduce-sqlite-log-contention
Jun 4, 2026
Merged

Reduce SQLite contention from OpenTelemetry SDK debug logs#26396
zanie-oai merged 4 commits into
mainfrom
zb/reduce-sqlite-log-contention

Conversation

@zanie-oai

@zanie-oai zanie-oai commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • skip opentelemetry_sdk DEBUG and TRACE events before formatting or queueing them for the SQLite log sink
  • preserve INFO, WARN, and ERROR events from the SDK, along with TRACE events from application targets
  • add a persistence-level regression test for the target and level policy

Why

OpenTelemetry's batch log processor emits internal BatchLogProcessor.ExportingDueToTimer meta-events every second per Codex process. In measured high-fanout logs_2.sqlite databases, low-level opentelemetry_sdk events accounted for over 30% of retained rows (30-60% on the machines of people I asked to check).

Persisting this SDK bookkeeping across many processes adds substantial write volume and contention without representing application activity.

Validation

  • just test -p codex-state (132/132 tests passed, plus bench smoke)
  • just fix -p codex-state
  • just fmt

These are just noise and can make up a huge proportion of the log volume and increase database contention
@zanie-oai zanie-oai marked this pull request as ready for review June 4, 2026 13:59
@zanie-oai

Copy link
Copy Markdown
Contributor Author

I'm hoping to reduce the number of concurrent app-servers separately, e.g., with changes to the Codex application, but these logs don't seem worthwhile anyway.

@zanie-oai

Copy link
Copy Markdown
Contributor Author

I think the number reported is a bit high — I'll see if I can get some more samples.

@zanie-oai

Copy link
Copy Markdown
Contributor Author

I had two other people check volume and they had 35% and 55% respectively.

@zanie-oai zanie-oai merged commit d81fcdf into main Jun 4, 2026
31 checks passed
@zanie-oai zanie-oai deleted the zb/reduce-sqlite-log-contention branch June 4, 2026 15:12
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants