Skip to content

Limit OG image generation concurrency#47

Merged
retlehs merged 2 commits intomainfrom
fix/og-generation-concurrency
Mar 19, 2026
Merged

Limit OG image generation concurrency#47
retlehs merged 2 commits intomainfrom
fix/og-generation-concurrency

Conversation

@retlehs
Copy link
Member

@retlehs retlehs commented Mar 19, 2026

Summary

  • Under heavy crawler traffic, unbounded go generatePackageOG() goroutines
    were constantly contesting the SQLite write lock, causing the pipeline to
    fail with SQLITE_BUSY
  • Adds a non-blocking semaphore (capacity 2) — excess requests skip generation
    instead of queuing, so the image gets generated on the next page visit
  • Combined with the busy_timeout increase (b71d4f5), this should eliminate
    pipeline failures from OG lock contention

Test plan

  • make test passes
  • Deploy and verify pipeline completes during heavy traffic
  • Verify OG images still generate (just throttled)

🤖 Generated with Claude Code

retlehs and others added 2 commits March 19, 2026 18:32
Under heavy crawler traffic, unbounded OG generation goroutines were
constantly contesting the SQLite write lock, starving the pipeline.
Non-blocking semaphore skips generation when at capacity — the image
will be generated on the next visit instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Acquiring the semaphore before spawning the goroutine ensures excess
requests under crawler spikes don't create unbounded goroutines that
immediately exit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@retlehs retlehs merged commit 5599d2a into main Mar 19, 2026
7 checks passed
@retlehs retlehs deleted the fix/og-generation-concurrency branch March 19, 2026 23:37
@tangrufus
Copy link
Member

Wrap handleDetail in a rate limiting middleware seems a better choice

@tangrufus
Copy link
Member

@retlehs
Copy link
Member Author

retlehs commented Mar 20, 2026

Thank you, will take a peek tomorrow or sometime soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants